Contains Code

Checks whether output is valid code or contains expected code snippets, validating structure and syntax.

Contains Code checks whether a response contains valid code, catching cases where a model was expected to produce code but returned prose instead. Run it wherever code generation is part of the task.

What it does

Contains Code is an LLM-as-Judge eval. It reads the generated output and checks whether it contains valid code.

Input

Required Input	Type	Description
`output`	`string`	The model output to be checked for valid code content

Output

Field	Type	Description
Result	`Pass / Fail`	Pass means the output contains valid code; Fail means it does not
Reason	`string`	A plain-language explanation of the code detection assessment

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "contains_code",
    output="def fibonacci(n):\n    a, b = 0, 1\n    for _ in range(n):\n        print(a)\n        a, b = b, a + b",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "contains_code",
  {
    output: "def fibonacci(n):\n    a, b = 0, 1\n    for _ in range(n):\n        print(a)\n        a, b = b, a + b"
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Contains Code wherever a task expects a code snippet rather than a natural-language answer.

Text outputs from code generation or coding-assistant tasks
Pipelines that parse or execute the model’s output downstream, where prose instead of code would break things
Quality gates for coding agents, before handing output to a compiler or interpreter

What to do when Contains Code fails

Ensure the code is properly formatted with appropriate indentation and syntax for its language. This evaluation can identify code across common programming languages like Python, JavaScript, and Java.

Mixed content (code with extensive natural language explanations) might yield uncertain results, and code snippets with syntax errors might still be identified as code, since the evaluation focuses on structural patterns rather than correctness.

Was this page helpful?

Questions & Discussion