Code & Output Validation Checks

Reference for code-based checks that validate output format, structure, latency, PII, and content constraints.

Code-based checks that validate the shape or safety of an output: format detectors (is_html/is_sql/is_url/is_xml), structural diffs and syntax checks (json_diff, syntax_validation), operational and safety guards (latency_check, regex_pii_detection), and content constraints (one_line, word_count_in_range, image_properties).

Metrics

Check	What it validates	Required inputs	Output
`is_html`	Text contains well-formed HTML: tags present, all non-void tags closed, no mismatched or orphan tags	`text`	Pass/Fail
`is_sql`	Text looks like syntactically valid SQL: starts with a recognized keyword, required clauses present, balanced parens and quotes	`text`	Pass/Fail
`is_url`	Text is a properly formatted URL with a valid scheme and a well-formed host (or a valid body for schemes like `mailto`)	`text`	Pass/Fail
`is_xml`	Text parses as well-formed XML; rejects `DOCTYPE`/`ENTITY` declarations to avoid XXE and billion-laughs risk	`text`	Pass/Fail
`json_diff`	Structural and value-level similarity between two JSON documents, compared recursively key by key	`output`, `expected`	Score 0-1 (fraction of matching nodes)
`syntax_validation`	Code syntax without executing it: Python via `ast.parse`, JSON via `json.loads`, JavaScript via bracket balancing	`text`	Pass/Fail
`latency_check`	Whether a latency value is within an acceptable bound (`latency <= max_latency_ms`)	`text`, `max_latency_ms`	Pass/Fail
`regex_pii_detection`	Text against regex patterns for SSN, credit card, phone, email, and IP address	`text`	Pass/Fail
`one_line`	Text is a single line: no newline breaks, at most 2 sentences, at most 50 words	`text`	Pass/Fail
`word_count_in_range`	Word count of text falls within a configured `min_words`/`max_words` range	`text`	Pass/Fail
`image_properties`	Image dimensions, format, and file size against configured constraints (width, height, format, max size)	`text` (image)	Pass/Fail

Run a check from code

Call evaluate() with the template id and the check’s required inputs. Swap the template id to run any check in this table.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "is_sql",
    text="SELECT id, name FROM users WHERE active = 1",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "is_sql",
  {
    text: "SELECT id, name FROM users WHERE active = 1",
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Gate structured outputs before they ship: is_html, is_sql, is_url, is_xml, syntax_validation, and json_diff catch malformed or drifted output from code- or query-generation pipelines
Catch PII before returning a response, with regex_pii_detection scanning for SSNs, card numbers, phone numbers, emails, and IPs
Enforce latency budgets on generation or tool calls with latency_check
Constrain response shape with one_line and word_count_in_range, for titles, tweets, summaries, and other length-sensitive outputs
Verify generated or uploaded images meet size, dimension, and format requirements with image_properties

Was this page helpful?

Questions & Discussion