Audio & ASR Metrics

Code-based metrics that score ASR/STT transcription accuracy against a ground-truth reference

Code-based metrics that score transcription accuracy by comparing an ASR/STT hypothesis against a ground-truth reference, at the character or word level.

Metrics

Metric	What it measures	Required inputs	Output
`character_error_rate`	Character-level edit distance between reference and hypothesis	`reference`, `hypothesis`	score (0-1), 1 - CER, higher = better
`match_error_rate`	Edit operations relative to hits plus edits at the word level	`reference`, `hypothesis`	score (0-1), 1 - MER, higher = better
`word_error_rate`	Word-level edit distance (insertions, deletions, substitutions) between reference and hypothesis	`reference`, `hypothesis`	score (0-1), 1 - WER, higher = better
`word_info_lost`	Word information lost, derived from hits relative to both reference and hypothesis length	`reference`, `hypothesis`	score (0-1), 1 - WIL, higher = better
`word_info_preserved`	Word information preserved, hits relative to both reference and hypothesis length	`reference`, `hypothesis`	score (0-1), higher = better

Run a metric from code

Call evaluate() with the template id and the metric’s required inputs. Swap the template id to run any metric in this table.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "word_error_rate",
    reference="the quick brown fox jumps over the lazy dog",
    hypothesis="the quick brown fox jump over the lazy dog",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "word_error_rate",
  {
    reference: "the quick brown fox jumps over the lazy dog",
    hypothesis: "the quick brown fox jump over the lazy dog",
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

These metrics fit any pipeline where an ASR/STT system produces a transcript that needs to be checked against a known-correct reference.

Evaluating ASR/STT pipelines against ground-truth transcripts
Comparing transcription models or configurations on the same audio set
Tracking recognition quality on noisy or difficult audio over time
Choosing character-level (CER) checks for languages or domains where word boundaries are unreliable, versus word-level checks (WER, MER, WIL, WIP) elsewhere

Was this page helpful?

Questions & Discussion