Built-in evals

Built-in evals are pre-configured evaluation templates you can attach to dataset runs, prompt runs, simulations, and live traces in Observe. Pick the evals you need, add them to your run, and the platform scores results automatically; Running Evaluations shows every surface you can attach them to.

Each row lists the inputs a template needs, where it fits, and how it scores, whether by LLM-as-Judge, an LLM-based ranker, a deterministic rule, or a statistical metric. For what the resulting scores mean, see Output types & scoring.

Required inputs are the fields you map when you attach an eval: input is the user query, output is your AI’s response, context is the retrieved or reference material, and expected_response is the known-good answer. Eval templates covers how mapping works.

RAG & retrieval

Whether what you retrieved was right, and whether the response stayed inside it

Eval	Description	Required Inputs	Use Cases	Evaluation Method
Context Adherence	Measures how well responses stay within the provided context.	output, context	Text, Audio, Image, Chat, RAG & Retrieval, Hallucination	LLM as Judge
Context Relevance	Evaluates the relevancy of the context to the user query.	input, context	Text, Audio, Image, Chat, RAG & Retrieval	LLM as Judge
Completeness	Evaluates if the response completely answers the query.	input, output	Text, Audio, Chat, RAG & Retrieval	LLM as Judge
Chunk Attribution	Tracks if the context chunk is used in generating the response.	output, context	RAG & Retrieval	LLM as Judge
Chunk Utilization	Measures how effectively context chunks are used in responses.	output, context	RAG & Retrieval	LLM as Judge
Groundedness	Ensures response strictly adheres to the provided context without external information.	output, context	Text, Audio, Chat, RAG & Retrieval, Hallucination	LLM as Judge
Detect Hallucination	Identifies fabricated facts not present in the input or reference.	input, output	Text, Audio, Image, Chat, RAG & Retrieval, Hallucination	LLM as Judge
Eval Ranking	Provides a ranking score for each context based on specified criteria.	input, context	RAG & Retrieval, Custom	LLM as Ranker
Recall@K	Evaluates recall at K for retrieval-based systems.	output, context	RAG & Retrieval	Statistical Metric
Precision@K	Evaluates precision at K for retrieval-based systems.	output, context	RAG & Retrieval	Statistical Metric
NDCG@K	Calculates normalized discounted cumulative gain at K.	output, context	RAG & Retrieval	Statistical Metric
MRR	Calculates mean reciprocal rank for retrieval results.	output, context	RAG & Retrieval	Statistical Metric
Hit Rate	Measures the fraction of queries where the correct item appears in top-K results.	output, context	RAG & Retrieval	Statistical Metric
Non-LLM Context Precision	Measures the fraction of retrieved contexts that exact-match a reference context.	output, expected	RAG & Retrieval	Statistical Metric
Non-LLM Context Recall	Measures the fraction of reference contexts that were successfully retrieved.	output, expected	RAG & Retrieval	Statistical Metric
Mean Average Precision	Averages precision at each relevant rank position, rewarding relevant items retrieved earlier.	reference, hypothesis	RAG & Retrieval	Statistical Metric

Safety & compliance

Anything that must never reach your user

Eval	Description	Required Inputs	Use Cases	Evaluation Method
PII Detection	Detects personally identifiable information (PII) in text.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
Toxicity	Evaluates content for toxic or harmful language.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
Sexist	Detects sexist content and gender bias.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
Prompt Injection	Evaluates text for potential prompt injection attempts.	input, output	Text, Audio, Image, Chat, Safety	LLM as Judge
Data Privacy Compliance	Checks output for GDPR, HIPAA, and other privacy regulation compliance.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
Cultural Sensitivity	Analyzes output for cultural appropriateness and inclusive language.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
Bias Detection	Identifies gender, racial, cultural, or ideological bias in output.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
No Racial Bias	Ensures output does not contain or imply racial bias.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
No Gender Bias	Checks the response does not reinforce gender stereotypes.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
No Age Bias	Evaluates if content is free from age-based stereotypes.	output	Text, Audio, Image, Chat, Safety	LLM as Judge
Answer Refusal	Checks if the model correctly refuses harmful or restricted queries.	input, output	Text, Audio, Chat, Safety	LLM as Judge
No Harmful Therapeutic Guidance	Ensures the model does not provide potentially harmful psychological advice.	output	Text, Audio, Chat, Safety	LLM as Judge
Clinically Inappropriate Tone	Evaluates whether tone is unsuitable for clinical or mental health contexts.	output	Text, Audio, Chat, Safety	LLM as Judge
Is Harmful Advice	Detects advice that could be physically, emotionally, legally, or financially harmful.	output	Text, Audio, Chat, Safety	LLM as Judge
Is Refusal	Detects whether a model output is a refusal, using pattern matching against common refusal phrasing.	text	Text, Safety	Deterministic / Rule-based

Conversation & agents

Multi-turn behavior: coherence, goal completion, and how a customer-facing agent handles the hard moments

Eval	Description	Required Inputs	Use Cases	Evaluation Method
Conversation Coherence	Evaluates if a conversation flows logically and maintains context throughout.	conversation	Conversation, Chat, Audio	LLM as Judge
Conversation Resolution	Checks if the conversation reaches a satisfactory conclusion.	conversation	Conversation, Chat, Audio	LLM as Judge
Evaluate Function Calling	Assesses accuracy and effectiveness of LLM function calls.	output	Text	LLM as Judge
Task Completion	Measures whether the model fulfilled the user’s request accurately.	input, output	Text, Audio, Chat	LLM as Judge
Customer Agent: Loop Detection	Detects if a customer agent is stuck in a loop during a conversation.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Context Retention	Evaluates if the agent correctly retains context across conversation turns.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Query Handling	Assesses how effectively the agent handles customer queries.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Termination Handling	Evaluates how the agent handles conversation termination.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Interruption Handling	Checks how the agent responds to interruptions during a conversation.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Conversation Quality	Evaluates the overall quality of a customer agent conversation.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Objection Handling	Assesses how the agent handles objections raised by the customer.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Language Handling	Evaluates language consistency and appropriateness in agent responses.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Human Escalation	Checks if the agent correctly identifies when to escalate to a human.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Clarification Seeking	Evaluates if the agent appropriately seeks clarification when needed.	conversation	Conversation, Chat, Audio	LLM as Judge
Customer Agent: Prompt Conformance	Checks if agent responses conform to the defined prompt and guidelines.	system_prompt, conversation	Conversation, Chat, Audio	LLM as Judge
Conversation Hallucination	Checks whether an agent fabricated facts, user attributions, or self-contradictions across a conversation.	conversation, context	Conversation, Chat, Hallucination	LLM as Judge
Customer Agent: Task Completion	Checks whether an agent fully resolved the customer’s request, including valid policy-based refusals.	agent_prompt, conversation	Conversation, Chat	LLM as Judge
Tool Call Accuracy	Compares an agent’s actual tool calls against expected calls, scoring matches on function name and arguments.	output, expected	Agents, Tool Use	Code
Trajectory Match	Compares an agent’s actual action sequence against an expected trajectory using configurable matching modes.	output, expected	Agents, Tool Use	Code
Step Count	Validates the number of steps in an agent trajectory against an exact count or a min/max range.	output	Agents	Code

Output quality & format

Whether a single response is well-made: tone, brevity, structure, and format checks

Eval	Description	Required Inputs	Use Cases	Evaluation Method
Tone	Analyzes the tone and sentiment of content.	output	Text, Audio, Chat, Safety	LLM as Judge
Instruction Adherence	Assesses how closely the output follows prompt instructions.	input, output	Text, Audio, Chat, Hallucination	LLM as Judge
Summary Quality	Evaluates if a summary captures main points and achieves appropriate length.	input, output	Text, Audio, Image, RAG & Retrieval	LLM as Judge
Translation Accuracy	Evaluates translation quality, accuracy, and cultural appropriateness.	output, expected_response	Text, Audio, RAG & Retrieval	LLM as Judge
No LLM Reference	Ensures output does not reference being an LLM or OpenAI model.	output	Text, Audio, Chat, Safety	LLM as Judge
No Apologies	Checks if the model unnecessarily apologizes.	output	Text, Audio, Chat	LLM as Judge
Is Polite	Ensures output maintains a respectful and non-aggressive tone.	output	Text, Audio, Chat	LLM as Judge
Is Concise	Measures whether the answer is brief and avoids redundancy.	output	Text, Audio, Chat	LLM as Judge
Is Helpful	Evaluates whether the response answers the user’s question effectively.	input, output	Text, Audio, Chat	LLM as Judge
Is Good Summary	Evaluates if a summary is clear, well-structured, and captures key points.	input, output	Text, Audio, RAG & Retrieval	LLM as Judge
Is Informal Tone	Detects whether the tone is casual (slang, contractions, emoji).	output	Text, Audio, Chat	LLM as Judge
Contains Code	Checks whether the output is valid code or contains expected code snippets.	output	Text	LLM as Judge
Text to SQL	Evaluates the quality and correctness of text-to-SQL generation.	input, output	Text	LLM as Judge
Is JSON	Validates if content is proper JSON format.	output	Text	Deterministic / Rule-based
One Line	Checks if the text is a single line.	output	Text	Deterministic / Rule-based
Contains Valid Link	Checks for presence of valid URLs in the output.	output	Text	Deterministic / Rule-based
Is Email	Validates email address format.	output	Text	Deterministic / Rule-based
No Invalid Links	Checks if the text contains no invalid URLs.	output	Text	Deterministic / Rule-based
Is HTML	Checks that text contains well-formed HTML with all tags properly closed and matched.	text	Text, Format	Code
Is SQL	Checks that text looks like syntactically valid SQL.	text	Text, Format	Code
Is URL	Checks that text is a properly formatted URL with a valid scheme and host.	text	Text, Format	Code
Is XML	Checks that text parses as well-formed XML, rejecting unsafe DOCTYPE/ENTITY declarations.	text	Text, Format	Code
JSON Diff	Compares structural and value-level similarity between two JSON documents.	output, expected	Text, Format	Code
Syntax Validation	Checks code syntax without executing it, for Python, JSON, and JavaScript.	text	Text, Format	Code
Latency Check	Checks whether a latency value is within an acceptable bound.	text	Text, Format	Code
Regex PII Detection	Checks text against regex patterns for SSN, credit card, phone, email, and IP address.	text	Text, Safety	Code
Word Count In Range	Checks whether a text’s word count falls within a configured min/max range.	text	Text, Format	Code

Reference & similarity

Compare a response against a known-good answer, from fuzzy matching to statistical scores

Eval	Description	Required Inputs	Use Cases	Evaluation Method
Fuzzy Match	Compares output with expected answer using approximate matching.	output, expected_response	Text, Audio, RAG & Retrieval	LLM as Judge
Ground Truth Match	Checks if the output matches a provided ground truth answer.	generated_value, expected_value	Text, Audio	LLM as Judge
BLEU Score	Computes BLEU score between expected answer and model output.	output, expected_response	Text	Statistical Metric
ROUGE Score	Calculates ROUGE score between generated and reference text.	output, expected_response	Text	Statistical Metric
Levenshtein Similarity	Calculates edit distance between generated and reference text.	output, expected_response	Text	Statistical Metric
Numeric Similarity	Calculates numerical difference between generated and reference value.	output, expected_response	Text	Statistical Metric
Embedding Similarity	Calculates semantic similarity between generated and reference text.	output, expected_response	Text	Statistical Metric
Semantic List Contains	Checks if text contains phrases semantically similar to reference phrases.	output, expected_response	Text	Statistical Metric
Jaccard Similarity	Measures token-set overlap between output and expected text.	output, expected	Text	Statistical Metric
Jaro-Winkler Similarity	Measures character-matching string similarity boosted by a common-prefix bonus.	output, expected	Text	Statistical Metric
Hamming Similarity	Measures matching character positions between two equal-length strings.	output, expected	Text	Statistical Metric

Audio & voice

Score speech directly: transcription accuracy, audio quality, and synthesized speech

Eval	Description	Required Inputs	Use Cases	Evaluation Method
Audio Transcription (ASR/STT)	Checks accuracy of a speech-to-text transcription against the audio source.	audio, transcription	Audio	LLM as Judge
Audio Quality	Evaluates the quality of audio (clarity, noise, distortion).	audio	Audio	LLM as Judge
TTS Accuracy	Evaluates the accuracy and naturalness of text-to-speech output.	text, generated_audio	Audio, Conversation	LLM as Judge
Dead Air Detection	Detects excessive silence in conversation audio using RMS energy analysis against configurable thresholds.	input_audio	Audio	Code
Character Error Rate	Measures character-level edit distance between reference and hypothesis transcripts.	reference, hypothesis	Audio	Statistical Metric
Match Error Rate	Measures edit operations relative to hits plus edits at the word level.	reference, hypothesis	Audio	Statistical Metric
Word Error Rate	Measures word-level edit distance between reference and hypothesis transcripts.	reference, hypothesis	Audio	Statistical Metric
Word Info Lost	Measures word information lost, derived from hits relative to reference and hypothesis length.	reference, hypothesis	Audio	Statistical Metric
Word Info Preserved	Measures word information preserved, hits relative to reference and hypothesis length.	reference, hypothesis	Audio	Statistical Metric

Image & document

Generated images and document extraction

Eval	Description	Required Inputs	Use Cases	Evaluation Method
Caption Hallucination	Detects hallucinated or fabricated details in image captions.	instruction, output	Image, RAG & Retrieval, Hallucination	LLM as Judge
Synthetic Image Evaluator	Evaluates synthetic or AI-generated images against criteria.	image, instruction	Image	LLM as Judge
OCR Evaluation	Evaluates the accuracy of optical character recognition (OCR) output.	input_pdf, json_content	Text, PDF / Document	LLM as Judge
FID Score	Computes the Fréchet Inception Distance between two sets of images; lower scores indicate more similar image distributions.	real_images, fake_images	Image	Statistical Metric
CLIP Score	Measures how well images match their text descriptions; higher scores indicate better image-text alignment (range: 0–100).	images, text	Image	Statistical Metric
Image Instruction Adherence	Measures how well generated images adhere to a given text instruction across subject, style, and composition.	instruction, images	Image	LLM as Judge
Image Properties	Checks image dimensions, format, and file size against configured constraints.	text	Image	Code
SSIM	Measures Structural Similarity Index between two images across luminance, contrast, and structure.	output, expected	Image	Statistical Metric
PSNR	Measures Peak Signal-to-Noise Ratio between two images from mean squared error over RGB pixels.	output, expected	Image	Statistical Metric

Statistical & NLP metrics

Code-based scores for classification, regression, and text quality, no LLM judge involved

Eval	Description	Required Inputs	Use Cases	Evaluation Method
Accuracy	Measures the fraction of predicted labels that exactly match expected labels.	output, expected	Text, Classification	Statistical Metric
Balanced Accuracy	Averages per-class recall, correcting for class imbalance that skews plain accuracy.	output, expected	Text, Classification	Statistical Metric
F1 Score	Measures token-level overlap between output and expected text as the harmonic mean of precision and recall.	output, expected	Text, Classification	Statistical Metric
F-Beta Score	Measures precision/recall on a chosen positive label, weighted toward precision or recall.	output, expected	Text, Classification	Statistical Metric
Precision Score	Measures the fraction of predicted-positive labels that are actually positive.	output, expected	Text, Classification	Statistical Metric
Cohen’s Kappa	Measures inter-rater agreement between predicted and expected labels, adjusted for chance agreement.	output, expected	Text, Classification	Statistical Metric
Matthews Correlation	Measures balanced classification quality across all four confusion matrix categories.	output, expected	Text, Classification	Statistical Metric
Fleiss’ Kappa	Measures multi-rater agreement from a rating count matrix.	output	Text, Classification	Statistical Metric
Log Loss	Measures cross-entropy between predicted probabilities and true labels.	output, expected	Text, Classification	Statistical Metric
RMSE	Measures root mean squared error between predicted and actual numeric values.	output, expected	Text, Regression	Statistical Metric
R2 Score	Measures the proportion of variance in actual values explained by predicted values.	output, expected	Text, Regression	Statistical Metric
Pearson Correlation	Measures the strength of the linear relationship between two numeric arrays.	output, expected	Text, Regression	Statistical Metric
Spearman Correlation	Measures the strength of the monotonic relationship between two numeric arrays.	output, expected	Text, Regression	Statistical Metric
METEOR Score	Measures unigram precision/recall with stemming and a fragmentation penalty.	reference, hypothesis	Text, NLP Metrics	Statistical Metric
chrF Score	Measures character n-gram F-score, robust for morphologically rich languages and short texts.	reference, hypothesis	Text, NLP Metrics	Statistical Metric
GLEU Score	Measures a sentence-level BLEU variant taking the min of precision and recall per n-gram order.	reference, hypothesis	Text, NLP Metrics	Statistical Metric
CodeBLEU	Measures n-gram BLEU blended with code-keyword matching.	reference, hypothesis	Text, NLP Metrics	Statistical Metric
Code Complexity	Measures cyclomatic complexity of Python code via AST, lower complexity scores higher.	text	Text, NLP Metrics	Statistical Metric
Type-Token Ratio	Measures lexical diversity as unique tokens divided by total tokens.	text	Text, NLP Metrics	Statistical Metric
Distinct-N	Measures vocabulary diversity as unique n-grams divided by total n-grams.	text	Text, NLP Metrics	Statistical Metric
Repetition Rate	Measures repeated n-gram rate to flag degenerate or looping output.	text	Text, NLP Metrics	Statistical Metric
Readability Score	Measures Flesch Reading Ease, normalized to a 0-1 score.	text	Text, NLP Metrics	Statistical Metric
Sentence Count	Checks sentence count against a configured min/max range.	text	Text, NLP Metrics	Statistical Metric
Translation Edit Rate	Measures word-level edit distance to transform hypothesis into reference.	reference, hypothesis	Text, NLP Metrics	Statistical Metric
SQuAD Score	Measures SQuAD-style QA scoring as the average of exact match and token F1.	output, expected	Text, NLP Metrics	Statistical Metric

Keep exploring