Evaluation FAQ & fixes

In this page

The questions people ask most about evaluation, and the errors they run into, with a direct fix for each. Hit an error? Jump straight to Common errors and fixes. If your answer isn’t here, reach out via support.

Common errors and fixes

Symptom	Cause	Fix
model_name required	A cloud eval was run without an evaluator model	Pass model_name="turing_flash" (or another evaluator model)
A mapped variable is flagged red, or comes up empty at run time	The trace attribute key doesn’t exactly match the variable name; the eval still runs, but that variable stays unpopulated	Send the attribute key with the exact variable name on every call; an empty value is fine, a missing key isn’t
”Not in row” errors on a trace-level eval	The variables live on different spans, so no single row carries them all	In the eval’s config, open the + menu, choose Data Injection, and enable Trace Context (Agent Evaluator only); sparse values pass as long as the key exists
Results show under Evals but not on your traces	The eval ran in the playground, which is standalone testing	Attach the eval to the project under Evals & Tasks, so results land inline on your traces
Eval errors on a missing field	A required input for the template wasn’t provided	Check the template’s required inputs in the Built-in evals reference
Results don’t appear after a run	Credentials not set, or the evaluator model is unavailable	Verify FI_API_KEY and FI_SECRET_KEY are set, and check the eval’s model against Evaluator models
The error localization block is empty	The eval passed, or it’s a code-type or composite eval	Error localization runs only on a failed result
Can’t find your API keys	API keys are visible to the Owner role only	Ask an Owner to generate or share keys from Settings → API Keys; see API Keys

Getting started

What can I evaluate?

Future AGI has 156 built-in evaluation templates covering quality, safety, factuality, RAG retrieval, format, bias, audio, and image, plus custom evals you define yourself. See Built-in evals for the full list.

How do I run my first eval?

Run it from the platform UI or the Python SDK. See Running Evaluations for every surface and the step-by-step flow.

Do I need to write code?

No. You can add evals to a dataset or a project and score them entirely from the UI. The SDK & API is there when you want to run the same evals from your own pipeline.

Choosing an eval

Which evals should I use for RAG?

Use retrieval-specific evals like context adherence, chunk attribution, and recall@k. See the RAG Evaluation cookbook for a walkthrough.

Can evals score audio directly?

Yes. Pass the audio URL or base64 as an attribute and map it in the eval’s inputs. Audio evals need turing_large or protect; Turing Small and Flash are text and image only. See Evaluator models.

Which evals need an expected answer?

Match and statistical evals compare your output against a reference value, so they need one: Ground Truth Match, Fuzzy Match, BLEU, ROUGE, and the similarity metrics all take an expected_response (or expected_value). See Ground truth for how to supply it.

Do all evals call an LLM?

No. Every eval is one of three eval types: Agent Evaluator and LLM-as-Judge use an evaluator model, while a Code Eval runs deterministic code, calls no model, and consumes no evaluator-model credits.

Can I use the Future AGI knowledge base as my agent’s knowledge store?

No. A knowledge base gives the evaluator reference context while it judges, like your policies or SOPs. It doesn’t serve retrieval to your application at runtime, and it isn’t a replacement for your own RAG store.

Results and scoring

What do a verdict, a score, and a label mean?

Pass/Fail templates return a Passed or Failed verdict. Score templates return a number shown as a percentage, and choice templates return a category label. See Output types & scoring.

How do I see why an eval failed?

Model-driven evals return a reason explaining the verdict (Code Evals don’t). For a failed result, error localization also pinpoints which input field, the prompt, context, or query, drove the failure.

Can I evaluate a whole conversation, not just one span?

Yes. When you attach an eval to a project you pick the level it runs at: span, trace, or session. A trace-level eval reads the whole request; a session-level eval reads the whole conversation.

Does a trace-level eval cost more than a span-level one?

No. Cost is per execution of the evaluator model, whatever level it runs at, and the length of the reasoning doesn’t change it either.

Why doesn’t Optimization improve on my eval?

Optimization consumes numeric results only: scores and pass/fail verdicts can drive it, a plain category label can’t. Pick a Score or Pass/Fail template for evals that feed an optimization run.

SDK

Which API keys do I need?

FI_API_KEY and FI_SECRET_KEY, found under Settings → API Keys.

Can I run several evals in one call?

Yes. evaluate() runs a single case or a batch, and each result comes back as an entry in eval_results.

How do I get results from an async run?

The SDK returns an eval_id immediately. Retrieve the result with evaluator.get_eval_result(eval_id) once the run finishes.

Can Future AGI trigger my CI pipeline?

No, the direction is always your pipeline calling Future AGI: run evals as a build step and gate the merge on the score. See Evaluate in CI/CD.

Keep exploring