Instruction Adherence
Assesses how closely a response follows prompt instructions, checking task completion and constraint compliance.
Instruction Adherence checks whether a response actually does what the prompt asked, including any constraints on format, length, or content. Run it to catch outputs that miss part of the instruction.
What it does
Instruction Adherence is an LLM-as-Judge eval. It reads the prompt and the generated output, then scores how closely the output follows the prompt’s instructions.
Input
| Required Input | Type | Description |
|---|---|---|
prompt | string | The input prompt provided to the model |
output | string | The output generated by the model |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher scores indicate better adherence to the prompt instructions |
| Reason | string | A plain-language explanation of the instruction adherence assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"prompt_instruction_adherence",
prompt="Write a short poem about nature that has exactly 4 lines and includes the word 'sunshine'.",
output="Morning rays filter through leaves,\nBirds sing in harmony with sunshine's glow,\nGreen meadows dance in the gentle breeze,\nNature's symphony in perfect flow.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"prompt_instruction_adherence",
{
prompt: "Write a short poem about nature that has exactly 4 lines and includes the word 'sunshine'.",
output: "Morning rays filter through leaves,\nBirds sing in harmony with sunshine's glow,\nGreen meadows dance in the gentle breeze,\nNature's symphony in perfect flow."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Instruction Adherence wherever a prompt carries specific constraints the response must satisfy.
- Text, audio, and chat outputs generated against a detailed prompt
- Format or length-constrained tasks, where the prompt sets a specific structure to follow
- Hallucination checks, to catch responses that drift from what was actually asked
What to do when Instruction Adherence fails
Identify specific areas where the output deviates from the given instructions. Providing targeted feedback helps refine the content to better align with the prompt.
Reviewing the prompt for clarity and completeness is essential, as ambiguous or vague instructions may contribute to poor adherence. If necessary, adjusting the prompt to offer clearer guidance can improve response accuracy.
Enhancing the model’s ability to interpret and follow instructions through fine-tuning or prompt engineering can further strengthen adherence.
Questions & Discussion