Task Completion
Evaluates whether a response successfully completes the task or request described in the user's input.
Task Completion checks whether a response actually accomplishes what the user asked for, not just whether it’s relevant. Run it to catch answers that address the topic but stop short of finishing the job.
What it does
Task Completion is an LLM-as-Judge eval. It reads the input request and the output, then scores whether the response fulfilled the user’s request.
Input
| Required Input | Type | Description |
|---|---|---|
input | string | User request or question to the model |
output | string | Response of the model based on the input |
Output
| Field | Type | Description |
|---|---|---|
| Result | Pass / Fail | Pass means the response successfully completes the requested task; Fail means it doesn’t |
| Reason | string | A plain-language explanation of why the response was classified as completing the task or not |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"task_completion",
input="Why doesn't honey go bad?",
output="Honey doesn't spoil because its low moisture and high acidity prevent the growth of bacteria and other microbes.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"task_completion",
{
input: "Why doesn't honey go bad?",
output: "Honey doesn't spoil because its low moisture and high acidity prevent the growth of bacteria and other microbes."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Task Completion wherever a response is supposed to fulfill a specific request, not just discuss it.
- Text, audio, and chat outputs answering how-to requests, questions, or multi-part asks
- Agent responses that need to be checked for whether they actually finished the job the user requested
- Regression checks after a prompt change, to confirm the model still completes the same tasks it used to
What to do when Task Completion fails
If a response is evaluated as not completing the task, make sure it directly addresses the specific task or question asked and that every part of a multi-part request is covered. Provide complete information without assuming prior knowledge.
For how-to requests, include clear, actionable steps; for questions seeking explanations, provide the reasoning behind the answer. Check whether the task needs specific formatting, calculations, or output types, and verify the response is accurate and relevant to the task.
Questions & Discussion