Evaluations SDK

Evaluate LLM outputs with the ai-evaluation package: 76+ local metrics for tone, hallucination, bias, and factual accuracy, plus guardrails (toxicity, PII, prompt injection) that run in under 10ms. Available in Python and TypeScript.

pip install ai-evaluation

from fi.evals import evaluate

# Local metric — no API key needed
result = evaluate("contains", output="Hello world", keyword="Hello")
print(result.score)    # 1.0
print(result.passed)   # True

# Cloud metric — needs FI_API_KEY and FI_SECRET_KEY
result = evaluate("toxicity", output="Hello world", model="turing_flash")
print(result.score)    # 1.0
print(result.passed)   # True

Optional extras (Python)

Extra	Install	What it adds
NLI models	pip install ai-evaluation[nli]	DeBERTa for faithfulness and hallucination detection
Embeddings	pip install ai-evaluation[embeddings]	Sentence-transformers for semantic similarity
Feedback	pip install ai-evaluation[feedback]	ChromaDB-backed feedback collection
Distributed	pip install ai-evaluation[celery]	Celery + Redis for distributed eval runs
Everything	pip install ai-evaluation[all]	All optional dependencies

Full reference