Statistical & Classification Metrics
Code-based metrics that score predicted labels or values against expected ones for classification and regression
These are code-based (CustomCodeEval) metrics: each one runs a fixed formula over the output and expected values you supply and returns a normalized 0-1 score, no LLM judge involved.
Metrics
| Metric | What it measures | Required inputs | Output |
|---|---|---|---|
accuracy | Fraction of predicted labels that exactly match expected labels | output, expected | score (0-1) |
balanced_accuracy | Average per-class recall, corrects for class imbalance that skews plain accuracy | output, expected | score (0-1) |
f1_score | Token-level overlap between output and expected text, harmonic mean of precision and recall | output, expected | score (0-1) |
f_beta_score | Precision/recall on a chosen positive label, weighted by beta (below 1 favors precision, above 1 favors recall) | output, expected, positive_label | score (0-1) |
precision_score | Fraction of predicted-positive labels that are actually positive (TP / (TP + FP)) | output, expected, positive_label | score (0-1) |
cohen_kappa | Inter-rater agreement between predicted and expected labels, adjusted for chance agreement | output, expected | score (0-1), higher = better |
matthews_correlation | Balanced classification quality across all four confusion matrix categories | output, expected | score (0-1), higher = better |
fleiss_kappa | Multi-rater agreement from a rating count matrix (rows = subjects, columns = categories) | output (rating matrix) | score (0-1), higher = better |
log_loss | Cross-entropy between predicted probabilities and true 0/1 labels, lower loss scores higher | output, expected | score (0-1), higher = better |
rmse | Root mean squared error between predicted and actual numeric values, lower error scores higher | output, expected | score (0-1), higher = better |
r2_score | Proportion of variance in the actual values explained by the predicted values | output, expected | score (0-1), higher = better |
pearson_correlation | Strength of the linear relationship between two numeric arrays | output, expected | score (0-1), higher = better |
spearman_correlation | Strength of the monotonic relationship between two numeric arrays, based on rank differences | output, expected | score (0-1), higher = better |
Run a metric from code
Call evaluate() with the template id and the metric’s required inputs. Swap the template id to run any metric in this table.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"f1_score",
output="The capital of France is Paris",
expected="Paris is the capital of France",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"f1_score",
{
output: "The capital of France is Paris",
expected: "Paris is the capital of France",
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
These metrics fit tasks where you already have a ground-truth label or value to compare against, not open-ended generation.
- Classification evals: accuracy, balanced_accuracy, precision_score, f1_score, f_beta_score, cohen_kappa, and matthews_correlation for predicted vs. expected labels
- Regression evals: rmse, r2_score, and log_loss for predicted vs. actual numeric values or probabilities
- Inter-rater or inter-model agreement: cohen_kappa for two raters, fleiss_kappa for three or more
- Correlation checks: pearson_correlation for linear relationships, spearman_correlation for monotonic (rank-based) relationships between two numeric series
Questions & Discussion