SDK & API

Evals from code

Everything you can do with evals in the platform, you can also do from code. The ai-evaluation package gives you a single evaluate() function that runs a metric locally, in the cloud, or as an LLM judge, and returns a result you can assert on.

This page is the reference for reaching evals programmatically. For the full SDK docs, see the SDK evals section.

Install and authenticate

pip install ai-evaluation

from fi.evals import evaluate

The 76+ local metrics run without an API key. Cloud evals and LLM-as-Judge need your Future AGI credentials:

export FI_API_KEY="your-api-key"
export FI_SECRET_KEY="your-secret-key"

Find both under Settings → API Keys in the platform.

The LLM-as-Judge engine doesn’t use these keys: it goes through LiteLLM, which reads your model provider’s standard environment variable (OPENAI_API_KEY, GEMINI_API_KEY, and so on).

One function, three engines

evaluate() picks the engine from what you pass. You never call a different function to switch engines.

You pass	Engine	Speed	Key needed
Metric name only	Local heuristic	under 1ms	No
Metric + model="turing_flash"	Cloud (Turing)	~1–3s	FI_API_KEY + FI_SECRET_KEY
prompt= + engine="llm" + model=	LLM-as-Judge	~2–5s	Model provider key

Force an engine with engine="local", engine="turing", or engine="llm".

There’s also a combined mode: a metric plus model= and augment=True runs the local heuristic first, then hands its score and reasoning to the LLM for refinement.

# Local metric: instant, no key
result = evaluate("contains", output="Hello world", keyword="Hello")

# Cloud metric: needs credentials + model
result = evaluate("toxicity", output="You're doing great!", model="turing_flash")

# LLM-as-Judge: custom criteria
result = evaluate(
    prompt="Rate helpfulness from 0 to 1",
    output="Here are 3 steps to fix that...",
    engine="llm",
    model="gemini/gemini-2.5-flash",
)

The Turing engine calls the same evaluation API the platform uses, so a template scored from code returns the same score you’d see in the platform.

What a result contains

A single eval returns an EvalResult. A list of metrics returns one result per metric.

Field	Description
eval_name	The metric that produced this result
score	The numeric result: 1.0 / 0.0 for pass/fail metrics, or a 0–1 value
passed	The boolean verdict
reason	Why the evaluator scored it that way
latency_ms	How long the eval took
status	”completed”, or “error” when the run failed
error	The error message when status is “error”
metadata	Extra engine-specific detail about the run

When a metric returns only a score, passed derives from score >= 0.5.

print(result.score)    # 1.0
print(result.passed)   # True
print(result.reason)   # "Keyword 'Hello' found"

# Batch: several metrics in one call
results = evaluate(["contains", "is_json"], output='{"greeting": "Hello"}', keyword="Hello")
for r in results:
    print(r.eval_name, r.passed)

Function signature

def evaluate(
    eval_name: str | list[str] | None = None,
    *,
    prompt: str | None = None,
    engine: str | None = None,
    model: str | None = None,
    augment: bool | None = None,
    config: dict | None = None,
    feedback_store: Any | None = None,
    generate_prompt: bool = False,
    fi_api_key: str | None = None,
    fi_secret_key: str | None = None,
    fi_base_url: str | None = None,
    **inputs,
) -> EvalResult | BatchResult

Inputs the metric needs (output, context, expected_response, and so on) are passed as keyword arguments. The Built-in evals reference lists the required inputs for each template.

generate_prompt=True turns a short plain-English prompt into full grading criteria before the run; it needs a model=.

Keep exploring

The SDK reference in depth:

Page	What it covers
evaluate()	The full evaluate() reference: parameters, return types, batch runs
Metrics Reference	Every local metric by category: string, JSON, similarity, hallucination, RAG, agents, guardrails
Cloud Evals	100+ pre-built Turing templates
LLM-as-Judge	Custom criteria with any LiteLLM model
Streaming	Score output token-by-token with early stopping
Distributed Evaluator	Run at scale with ThreadPool, Celery, Ray, or Temporal