Use custom models

Bring your own LLM as the evaluator, from a provider or a custom endpoint

Evaluations need a model to act as the evaluator: to read each response and decide whether it passes, fails, or scores in a range. A custom model lets you bring your own LLM as that evaluator instead of a Future AGI model, when a model of yours knows your domain better, when inference has to stay in a specific cloud or region, or when you want eval costs tracked against a model you already pay for. Once added, it appears in the model dropdown wherever you configure an eval.

Two ways to connect:

  • From a provider: a direct integration with Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, or Azure
  • Custom endpoint: any model behind an HTTP API, including self-hosted, fine-tuned, or proxy deployments

Add a model

Go to Settings → AI Providers, open the Custom model tab, and click Create custom model.

AI Providers page on the Custom model tab, listing existing custom models with masked credentials and edit, delete, and copy icons, with a callout pointing at the Create custom model button top right

The Custom model tab lists everything you’ve added so far; edit, delete, or copy any entry from here

An Add Model drawer opens with two options: From model Provider or Configure Custom Model.

From a provider

With From model Provider selected, pick a provider from the Model Provider dropdown: Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, or Azure.

Model Provider dropdown open, listing Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, and Azure, with a callout pointing at Open AI

Five supported providers, each with its own credential form

Fill in the provider’s form: a Model Name to recognize it later, Input and Output Token Cost Per Million Tokens for cost tracking, and the provider’s credentials, an API key for Open AI, region and access keys for Bedrock or Sagemaker, a service account for Vertex AI, endpoint and key for Azure. A Form and JSON toggle lets you fill the fields individually or paste a raw config.

Add Model drawer with Open AI selected as the Model Provider, showing Model Name, Input and Output Token Cost Per Million Tokens, a Form/JSON toggle, API Key, and an optional Base URL field, with a callout pointing at the API Key field

Open AI’s form: model name, token costs, API key, and an optional base URL

Custom endpoint

Select Configure Custom Model instead to connect any model behind an HTTP API. Fill in the Model Name, the token costs, and the API Base URL, the endpoint Future AGI calls. Anything else the endpoint needs, an auth header, a tenant ID, a routing parameter, goes under Custom Configuration as key/value pairs, and Add more configuration adds another pair.

Add Model drawer with Configure Custom Model selected, showing required Model Name, Input and Output Token Cost Per Million Tokens, and API Base URL fields, plus an Add Custom Configuration section with Custom Key and Custom Value inputs and an Add more configuration button, with a callout pointing at the API Base URL field

A custom endpoint needs an API base URL; everything else it needs goes in Custom Configuration

Save the model

Click Add Custom model. The model joins the Custom model list, and it now shows up in the model dropdown wherever you pick an evaluator, like when you create a custom eval.

Keep exploring

Running Evaluations

Run an eval with your model on any surface

Create a custom eval

Write an eval rule your model applies

Evaluator models

The built-in models you can use instead

Evaluate in CI/CD

Turn eval scores into a merge gate
Was this page helpful?

Questions & Discussion