Running Evaluations

The evaluate() function is the main entry point for running evaluations. It accepts a metric name (or list), your inputs as keyword arguments, and optionally a model. The engine is selected automatically based on what you pass.

Quick Examples

Local metric (no API key needed)

from fi.evals import evaluate

result = evaluate("contains", output="Hello world", keyword="Hello")

print(result.eval_name)   # "contains"
print(result.score)       # 1.0
print(result.passed)      # True
print(result.reason)      # "Keyword 'Hello' found"
print(result.latency_ms)  # 0.73

Cloud eval (needs API key + model)

from fi.evals import evaluate

result = evaluate(
    "toxicity",
    output="You're doing a great job!",
    model="turing_flash",
)

print(result.score)   # 1.0
print(result.passed)  # True
print(result.reason)  # "This evaluation is given as the content fully follows..."

LLM-as-Judge (custom criteria)

from fi.evals import evaluate

result = evaluate(
    prompt="Rate how helpful this response is from 0 to 1. A helpful response directly answers the question with actionable steps.",
    output="Here are 3 steps to fix the issue: 1. Check your config...",
    query="How do I fix the login error?",
    engine="llm",
    model="gemini/gemini-2.5-flash",
)

print(result.score)   # 0.9
print(result.reason)  # '{"score": 0.9, "reason": "Provides structured steps..."}'

Batch evaluation

from fi.evals import evaluate

results = evaluate(
    ["contains", "one_line", "is_json"],
    output="Hello world",
    keyword="Hello",
)

for r in results:
    print(f"{r.eval_name}: score={r.score}, passed={r.passed}")
# contains: score=1.0, passed=True
# one_line: score=1.0, passed=True
# is_json: score=0.0, passed=False

Function Signature

def evaluate(
    eval_name: str | list[str] | None = None,
    *,
    prompt: str | None = None,
    engine: str | None = None,
    model: str | None = None,
    augment: bool | None = None,
    config: dict | None = None,
    generate_prompt: bool = False,
    feedback_store: Any | None = None,
    fi_api_key: str | None = None,
    fi_secret_key: str | None = None,
    fi_base_url: str | None = None,
    **inputs,
) -> EvalResult | BatchResult

Parameters

Return Types

EvalResult (single eval)

BatchResult (multiple evals)

Returned when eval_name is a list. Iterable collection of EvalResult objects.

results = evaluate(["toxicity", "faithfulness"], output="...", model="turing_flash")

# Iterate
for r in results:
    print(r.eval_name, r.score)

# Access by name
toxicity = results.get("toxicity")

# Check overall pass rate
print(results.success_rate)  # 0.0 to 1.0

# Count
print(len(results))  # 2

Engine Routing

If you don’t set engine explicitly, the function picks one:

1. No model passed → local engine (heuristic metrics, <1ms)
2. model="turing_flash" / "turing_small" / "turing_large" → Turing cloud engine
3. Any other model string → LLM-as-Judge engine
4. augment=True → local first, then LLM refinement

You can check which engine ran via result.metadata["engine"].

Common Patterns

Error handling

result = evaluate("toxicity", output="test", model="turing_flash")

if result.status == "error":
    print(f"Eval failed: {result.error}")
else:
    print(f"Score: {result.score}")

Augmented evaluation (local + LLM)

result = evaluate(
    "faithfulness",
    output="Paris is the capital of France.",
    context="France is a European country with Paris as its capital.",
    model="gemini/gemini-2.5-flash",
    augment=True,
)

print(result.metadata["engine"])  # "local+llm"

Auto-generated grading criteria

result = evaluate(
    prompt="Check if the response is empathetic and acknowledges the customer's frustration",
    output="I understand this is frustrating. Let me help fix that right away.",
    engine="llm",
    model="gpt-4o",
    generate_prompt=True,
)
# The prompt is expanded into a detailed rubric automatically

Environment Variables

Variable	Required for	Default
FI_API_KEY	Cloud (Turing) evals	—
FI_SECRET_KEY	Cloud (Turing) evals	—
FI_BASE_URL	Custom API endpoint	https://api.futureagi.com
GOOGLE_API_KEY	Gemini models (LLM judge)	—
OPENAI_API_KEY	OpenAI models (LLM judge)	—
ANTHROPIC_API_KEY	Claude models (LLM judge)	—

Related