Use decision models as evaluators

Pick Jev, GPT-6 Luna Decisions, Perplexity Decider or another decision model as an eval's judge, test it, run it on a dataset and call it from code.

About

A decision model answers a typed question about your data instead of writing free text. You give it a Pass/fail question, a set of choices or an ordered rubric, and it returns the answer with its probabilities. The evaluator model picker lists 35 decision models next to the Future AGI models: TypeSafe Jev, OpenAI GPT-6 Luna Decisions, Perplexity Decider, Microsoft Decision 1, Solar Decide, Clef, Reflex and more. Each one runs on your own provider key.

This guide follows one support team. Their assistant answers customers of an outdoor-gear store, and they want three checks on every reply:

EvalQuestionOutput typeDecision model
resolves_requestDoes the reply resolve the request or give a specific next step?Pass/failGPT-6 Luna Decisions via OpenRouter
ticket_topicWhich topic is the ticket about?ChoicesDecider 1.2 via Perplexity
reply_toneHow warm and helpful is the reply?ScoringJev 1.14 via TypeSafe

Picking GPT-6 Luna Decisions on its OpenRouter connection and testing the eval on two replies

Decision models return typed decisions, not reasoning. Use an LLM judge such as a Turing model when you need a written explanation for each score.

Before you start

Prerequisites
  • A Future AGI workspace where you can create evals
  • A key for at least one connection: OpenRouter (every decision model), TypeSafe (Jev), OpenAI (GPT-6 Luna Decisions) or Perplexity (Decider)

The model picker never borrows a key or switches to another model. If a key is missing or rejected, the eval stops with an error until you fix the key.

Create a decision-model eval

Start an LLM-as-a-Judge eval

Open Evals, select Create evaluation, then choose the LLM-As-A-Judge tab. Enter a name and write the question in the prompt. Reference your data with mustache variables:

Customer message: {{customer_message}}

Assistant reply: {{assistant_reply}}

Does the assistant reply resolve the customer's request or give a specific next step?

Write the question only. The output type you pick below tells the model how to answer.

Open the model picker

Select Select model under the prompt. Scroll past the Future AGI models to Decision models, or type part of a name, author or connection in Search models….

Each row shows the model, its author, the output types it supports and the key state for the selected connection. A Moving alias badge marks a latest model whose version changes upstream.

The Decision models section of the model picker: 35 models, with Jev 1.14, Jev 1.13 and Jev latest each showing TypeSafe, its output types, its key state and a Connection selector

Choose the model and its connection

Typing Luna in Search models, choosing the OpenRouter connection for GPT-6 Luna Decisions, and the model appearing under the prompt as GPT-6 Luna Decisions · OpenRouter

Use the row’s Connection selector to choose how the model is reached. For example, GPT-6 Luna Decisions can run on OpenAI directly or through OpenRouter. Every model is available through OpenRouter; Jev, GPT-6 Luna Decisions and Decider also have a direct connection.

If the connection has no key, choosing it opens Configure API keys in place. Search for the provider, paste the key and select Add. The model is selected once the key is saved. You can also add keys in advance in Settings → AI Providers.

Configure API keys opened from the model picker and filtered to OpenRouter, with an API Key field and Add button

The selected model appears under the prompt as Model · Connection, for example GPT-6 Luna Decisions · OpenRouter.

Configure the output type

Pick Pass/fail, Scoring or Choices under Output Type. Each decision model lists the types it supports; a model that does not support the selected type is locked in the picker with the reason.

Output typeWhat you configureWhat the model returns
Pass/failA Pass threshold (50% by default) and optional Describe a pass (true) and Describe a fail (false)The probability that the pass description is true, compared with the threshold to give Passed or Failed
ChoicesChoice names, each shown as Pass, Neutral or FailOne choice, with the probability of every choice and a confidence
Scoring2 to 10 Ordered levels, lowest first, and a Pass thresholdThe expected level and a normalized score: the level divided by level count − 1

For resolves_request, set the descriptions:

  • Describe a pass (true): The reply resolves the request or gives a concrete next step, such as a tracking number, a refund or return instructions.
  • Describe a fail (false): The reply deflects, stays vague or gives the customer nothing to act on.

resolves_request with GPT-6 Luna Decisions · OpenRouter, Pass/fail output, a 50% pass threshold and the pass and fail descriptions

Test it on real examples

In Test Evaluation, open the Custom tab and paste a JSON object with one key per variable:

{
  "customer_message": "Where is my order #48213? It was supposed to arrive Tuesday.",
  "assistant_reply": "Sorry for the wait. Order #48213 left our Denver warehouse on Monday with UPS, tracking number 1Z84A9E70312345678. UPS now expects to deliver it Thursday by 8 pm."
}

Select Test Evaluation. The result shows the verdict and the decision details: the probability, the model you requested, the model the provider returned, the connection and the token usage.

Test a reply that should fail before you save. "We'll look into it." as the answer to "My package says delivered but it's not here." returns Failed.

Test Evaluation run twice: the tracking-number reply returns Pass with probability 1.00, then the reply that only says it will look into it returns Fail with probability 0.00

Test Evaluation result for the tracking-number reply: Pass, probability 1.00 against a 0.50 threshold, requested and returned model, OpenRouter connection and token usage

Save the eval

Select Save Evaluation. The eval keeps its model and connection, and every place that runs it uses that model unless you choose another one for a run.

Choices: route tickets by topic

ticket_topic asks only about the customer message:

Customer message: {{customer_message}}

Which topic is this support ticket about?

Choose Choices and add four choices: Billing, Shipping, Returns and exchanges and Product question. Topics are neither good nor bad, so show each one as Neutral. Then pick Decider 1.2 with the Perplexity connection.

Adding four choices shown as Neutral, picking Decider 1.2 on its Perplexity connection, and testing the duplicate-charge ticket, which returns Billing

For "I was charged twice for the Ridgeline 2 tent.", Decider 1.2 chooses Billing with a probability of 1.00. The result lists the probability of every choice.

Choices result: Chosen label Billing with the distribution across the four topics, confidence, requested and returned model, and usage

Scoring: rate the tone

reply_tone uses an ordered rubric:

Customer message: {{customer_message}}

Assistant reply: {{assistant_reply}}

How warm and helpful is the tone of the assistant reply?

Pick Jev 1.14 with the TypeSafe connection; it shows under the prompt as Jev 1.14 (pinned). Choose Scoring, then add three levels with Add level: Dismissive, Neutral and Warm and helpful.

Picking Jev 1.14, choosing Scoring, adding the levels Dismissive, Neutral and Warm and helpful, and testing a dismissive reply

Jev places "That's not something I can help with." at the Dismissive level, and the duplicate-charge refund at Warm and helpful.

Scoring result for the dismissive reply: expected level, normalized score and the probability of each level

Run the evals on a dataset

The team keeps a sample of real conversations in a dataset with customer_message and assistant_reply columns.

customer_messageassistant_reply
Where is my order #48213? It was supposed to arrive Tuesday.Sorry for the wait. Order #48213 left our Denver warehouse on Monday with UPS, tracking number 1Z84A9E70312345678. UPS now expects to deliver it Thursday by 8 pm.
I was charged twice for the Ridgeline 2 tent.Thanks for flagging this. I can see two charges of $289.00 on 14 November. I’ve refunded the duplicate charge, and it will show on your card in 3 to 5 business days.
Is the Summit 45 backpack waterproof?Please check the product page for details.
My package says delivered but it’s not here.We’ll look into it.

Add the evals

Open the dataset and select Evaluate, then Add Evaluations. Search for resolves_request and select Add.

The drawer opens on the eval’s saved decision model. Map each variable to a column under Variable Mapping, and give the result column a short Name, such as resolved. Step through Test on row 1 of 10 to check a few rows, then select Add Evaluation. Repeat for reply_tone (named tone) and ticket_topic (named topic_decider).

Opening Evaluate on the dataset, adding resolves_request, naming the column resolved, then adding reply_tone and ticket_topic, and ticket_topic again with Solar Decide (sped up)

The evaluation drawer for resolves_request on the dataset, with its decision model, the result column name and the variable mapping

Compare two decision models on the same eval

Add ticket_topic a second time, name it topic_solar and choose a different model in the drawer’s model picker, such as Solar Decide · OpenRouter. The saved eval does not change; only this result column uses Solar Decide.

ticket_topic added again with Solar Decide · OpenRouter chosen for this run

Run them

Select Run All. Each eval fills its own column as rows finish.

Run All on the dataset: the resolved, tone, topic_decider and topic_solar columns fill in row by row, then the grid scrolls right to the topic columns (sped up)

The dataset grid: each conversation with its resolved verdict, Passed in green or Failed in red, and the start of the tone column

Scroll right to compare the models. In this run, Decider 1.2 and Solar Decide chose the same topic for all 10 tickets, and the four replies that only deflect, such as "We'll look into it." and "Refunds take 5 to 10 business days.", failed resolved and scored lowest on tone.

The tone, topic_decider and topic_solar columns side by side, with the same topic from both models on every row

Run a saved eval from code

Call a saved decision-model eval by name. It runs on the model and connection saved with the eval. Pass a decision model ID as the model to use a different one for that call.

# python -m pip install agent-learning-kit==0.3.7
# export FI_API_KEY=... FI_SECRET_KEY=...
from fi.alk.evals import Evaluator

evaluator = Evaluator()

result = evaluator.evaluate(
    eval_templates="resolves_request",
    inputs={
        "customer_message": "My package says delivered but it's not here.",
        "assistant_reply": "We'll look into it.",
    },
)
print(result.eval_results[0].output)  # Failed
print(result.eval_results[0].reason)  # GPT-6 Luna Decisions probability 0.00 vs threshold 0.50 → Fail

# Same eval inputs, another decision model for this call only
topic = evaluator.evaluate(
    eval_templates="ticket_topic",
    inputs={"customer_message": "My package says delivered but it's not here."},
    model_name="decision/openrouter/upstage/solar-decide",
)
print(topic.eval_results[0].output)  # {'score': 0.5, 'choice': 'Shipping'}
curl -X POST "https://api.futureagi.com/sdk/api/v1/new-eval/" \
  -H "X-Api-Key: $FI_API_KEY" \
  -H "X-Secret-Key: $FI_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "eval_name": "ticket_topic",
    "inputs": {"customer_message": "My package says delivered but it'"'"'s not here."},
    "model": "decision/openrouter/upstage/solar-decide"
  }'

Leave out "model" to use the model saved with the eval. The response lists each result with its output and reason:

{
  "status": true,
  "result": [
    {
      "evaluations": [
        {
          "name": "ticket_topic",
          "reason": "Solar Decide chose \"Shipping\" (probability 1.00)",
          "runtime": 562,
          "output": {"score": 0.5, "choice": "Shipping"},
          "output_type": "choices",
          "eval_id": "a6519263-1b35-4dc1-8c19-b08e41154901"
        }
      ]
    }
  ]
}

Find a model ID

List the decision models with model_type=decision. Add search to filter by name, author or connection, and limit to get more than 10 per page. is_available is true when your workspace has a key for that connection.

curl -G "https://api.futureagi.com/model-hub/api/models_list/" \
  --data-urlencode "model_type=decision" \
  --data-urlencode "search=luna" \
  -H "X-Api-Key: $FI_API_KEY" \
  -H "X-Secret-Key: $FI_SECRET_KEY"
{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "model_name": "openai/gpt-6-luna-decisions",
      "label": "GPT-6 Luna Decisions",
      "author": "OpenAI",
      "version": "versioned",
      "question_types": ["noul", "choice", "score"],
      "min_choices": 1,
      "max_choices": 255,
      "max_state_bytes": 120000,
      "notice": "Text and JSON inputs only.",
      "unavailable_reason": null,
      "connections": [
        { "model_name": "decision/openai/gpt-6-luna", "provider": "openai", "label": "OpenAI", "is_available": false },
        { "model_name": "decision/openrouter/openai/gpt-6-luna-decisions", "provider": "openrouter", "label": "OpenRouter", "is_available": true }
      ],
      "type": "decision"
    }
  ],
  "total_pages": 1,
  "current_page": 1
}

Use a connection’s model_name as the model ID. question_types uses the decision API names: noul is Pass/fail, choice is Choices and score is Scoring.

Available decision models

AuthorModelsConnections
TypeSafeJev 1.14, Jev 1.13, Jev latestTypeSafe, OpenRouter
OpenAIGPT-6 Luna DecisionsOpenAI, OpenRouter
PerplexityDecider 1.2, Decider Flash 1.2, Decider latest, Decider Flash latestPerplexity, OpenRouter
MicrosoftMicrosoft Decision 1OpenRouter
CloudflareClef, Clef Flash, Clef OmniOpenRouter
Liquid AId1.1, d1, d1-ultra, d1 latest, d1-ultra latestOpenRouter
BasetenJoey 1, Elephant 1, Mammoth 1OpenRouter
Nace.AIDrex Max, Drex 2.0, Drex 1.5OpenRouter
RespanSpan-02, Span-01, Span-01 Lite, Span-01 Lite (free)OpenRouter
UpstageSolar Decide, Solar Decide FlashOpenRouter
InceptionMercury DecideOpenRouter
MorphReflexOpenRouter
ScaleDownClassify-1OpenRouter
TogetherTev1 4B ExperimentalOpenRouter
Jared PalmerKev 4BOpenRouter
ParallelDecisions Fast (free)OpenRouter

Model IDs follow three patterns:

ConnectionID patternExample
OpenRouterdecision/openrouter/<author>/<model>decision/openrouter/upstage/solar-decide
OpenAI, Perplexitydecision/<connection>/<model>decision/perplexity/pplx-decider-v1.2
TypeSafejev-<version>jev-1.14.0, jev-latest

Inputs and limits

Decision models read text and JSON only. The mapped input can be up to 120,000 UTF-8 bytes, except for these models:

ModelLimit or behavior
Clef, Clef Flash2,000 UTF-8 bytes of input. A larger input is refused rather than truncated.
Joey 1, Elephant 1, Mammoth 148,000 UTF-8 bytes of input
Kev 4B24,000 UTF-8 bytes of input
Span-01, Span-01 Lite, Span-01 Lite (free)Pass/fail only
Tev1 4B Experimental2 to 24 choices
Decisions Fast (free)Researches the live web, so your inputs may be used in web queries
Mercury DecideListed but not selectable: OpenRouter has no active endpoint for it

Decision models do not use few-shot examples, tools, knowledge bases or internet checks. If the eval already has few-shot datasets selected, the editor keeps them, and the decision model does not use them. See Jev evaluators for the controls the editor disables.

A Moving alias (Jev latest, Decider latest, d1 latest) follows the provider’s newest version of that model, never a different model. Pick a versioned model when you need results that stay comparable over time.

Costs

Each call is billed by the provider to the key you saved. For the free models, the picker notes: “Provider inference is free; FutureAGI evaluation charges still apply.” Jev’s Cloud fee is described in Evaluator models.

Troubleshooting

A test or run fails

MessageWhat to do
OpenAI rejected the request with the saved API key. Check the key and its model permissions in Settings → AI Providers.The provider refused the key for this model. Update the key in Settings → AI Providers, or choose another connection, such as OpenRouter, in the model picker.
Decision evaluators are not available in this build.The deployment was built without Future AGI’s ee code, which runs decision models. Use a standard Future AGI build.

A model is locked in the picker

A locked row shows the reason in place of its key state:

ReasonWhat to do
Span-01 does not support Choices. Choose another model or output type.The model cannot answer the selected output type. Change the output type or the model.
Tev1 4B Experimental requires 2 to 24 choices.Add or remove choices to fit the model’s range.
No active OpenRouter endpoint. Choose another decision model.The model has no working endpoint. Choose another decision model.

Next steps

Was this page helpful?

Questions & Discussion