Audio Transcription
Checks the accuracy of a speech-to-text transcription against the audio source, flagging omissions, additions, and misrepresentations.
Audio Transcription checks whether a generated transcript accurately reflects what was said in an audio file. Run it to catch omissions, additions, and misrepresentations in speech-to-text output.
What it does
Audio Transcription is an LLM-as-Judge eval. It listens to the audio and reads the generated transcript, then scores how accurately the transcript represents the speech.
Input
| Required Input | Type | Description |
|---|---|---|
audio | string | The file path or URL to the audio file containing the speech |
generated_transcript | string | The text transcription to be evaluated for accuracy |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher scores indicate a more accurate transcription |
| Reason | string | A plain-language explanation of the transcription assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"ASR/STT_accuracy",
audio="https://datasets-server.huggingface.co/assets/MLCommons/peoples_speech/--/f10597c5d3d3a63f8b6827701297c3afdf178272/--/clean/train/0/audio/audio.wav",
generated_transcript="i wanted this to share a few things but i'm going to not share as much as i wanted to share because we are starting late i'd like to get this thing going so we all get home at a decent hour this this election is very important to",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"ASR/STT_accuracy",
{
audio: "https://datasets-server.huggingface.co/assets/MLCommons/peoples_speech/--/f10597c5d3d3a63f8b6827701297c3afdf178272/--/clean/train/0/audio/audio.wav",
generated_transcript: "i wanted this to share a few things but i'm going to not share as much as i wanted to share because we are starting late i'd like to get this thing going so we all get home at a decent hour this this election is very important to"
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Audio Transcription wherever an audio source is converted to text and you need to confirm the transcript is faithful to what was actually said.
- Speech-to-text (ASR/STT) pipelines, to verify transcripts before they feed downstream steps
- Call transcription and meeting notes, where missed or added words change meaning
- Any audio-based workflow where the transcript is later used for search, summarization, or compliance review
What to do when Audio Transcription fails
If the transcription accuracy score is lower than expected:
- Ensure the audio is clear with minimal background noise
- Check for proper capitalization and punctuation in the transcription
- Include all filler words (um, uh, etc.) for verbatim accuracy if required
- Verify correct spelling of technical terms, names, or specialized vocabulary
- Review for word substitution errors where similar-sounding words are confused
- Consider using professional transcription services for important content
- For non-native speakers, ensure the transcriber is familiar with the accent
- Use timestamps for longer audio to help identify where errors might occur
Questions & Discussion