NLP & Text Metrics
Code-based metrics for reference overlap, lexical diversity, readability, and code structure.
These are code-based metrics, not LLM judges. Some compare generated text against a reference (meteor, chrf, gleu, code_bleu, translation_edit_rate, squad), some measure a text’s own diversity or structure (type_token_ratio, distinct_n, repetition_rate, sentence_count, readability_score), and one measures code structure directly (code_complexity).
Metrics
| Metric | What it measures | Required inputs | Output |
|---|---|---|---|
meteor_score | Unigram precision/recall with stemming and a fragmentation penalty, correlates with human judgment better than BLEU on many tasks | reference, hypothesis | score (0-1) |
chrf_score | Character n-gram F-score (order up to 6, recall-weighted), robust for morphologically rich languages and short texts | reference, hypothesis | score (0-1) |
gleu_score | Sentence-level BLEU variant taking the min of precision and recall per n-gram order | reference, hypothesis | score (0-1) |
code_bleu | N-gram BLEU blended with code-keyword matching (def, if, return, etc), weighted 0.7 BLEU / 0.3 keyword overlap | reference, hypothesis | score (0-1) |
code_complexity | Cyclomatic complexity of Python code via AST (branch points from if, for, while, except, boolean ops); lower complexity scores higher | text | score (0-1), derived from a decision-point count |
type_token_ratio | Lexical diversity: unique tokens divided by total tokens | text | ratio (0-1) |
distinct_n | Vocabulary diversity: unique n-grams divided by total n-grams (n configurable, default 1) | text | ratio (0-1) |
repetition_rate | Repeated n-gram rate, returned as 1 minus the rate so higher means less repetitive; flags degenerate/looping output | text | score (0-1) |
readability_score | Flesch Reading Ease, normalized to 0-1 (Flesch-Kincaid grade level also reported in the reason) | text | score (0-1) |
sentence_count | Sentence count checked against a min_sentences/max_sentences range you configure | text | Pass/Fail |
translation_edit_rate | Word-level edit distance to transform hypothesis into reference, normalized by reference length, returned as 1-TER | reference, hypothesis | score (0-1) |
squad_score | SQuAD-style QA scoring: average of exact match and token F1 after normalizing case, articles, and punctuation | output, expected | score (0-1) |
Run a metric from code
Call evaluate() with the template id and the metric’s required inputs. Swap the template id to run any metric in this table.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"meteor_score",
reference="The cat sat quietly on the warm windowsill.",
hypothesis="A cat sat quietly on the warm windowsill.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"meteor_score",
{
reference: "The cat sat quietly on the warm windowsill.",
hypothesis: "A cat sat quietly on the warm windowsill.",
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Reach for these when you need a fast, deterministic score instead of an LLM judge.
- Translation or generation tasks where you have a reference text to score overlap against (meteor, chrf, gleu, translation_edit_rate)
- Code generation, where code_bleu rewards matching structure and keywords, not just exact tokens
- QA pipelines with a gold answer, using squad_score for exact match plus F1
- Diversity and repetition checks on generated text (type_token_ratio, distinct_n, repetition_rate), useful for catching degenerate or looping output
- Enforcing structural constraints like sentence count, or checking readability level for a target audience
- Flagging overly complex generated code with code_complexity before it ships
Questions & Discussion