Step Count
Validates the number of steps in an agent trajectory against an exact count or a min/max range.
Step Count checks whether an agent used a reasonable number of steps to complete a task. Run it to catch agents that loop, take shortcuts, or otherwise drift from an expected execution length.
What it does
Step Count is a code-based check. It parses the agent trajectory into a list of steps and counts them, then validates that count against the configured tuning params: expected_steps for an exact count, or min_steps and max_steps for a range. At least one of these must be set, otherwise the eval fails outright.
Input
| Required Input | Type | Description |
|---|---|---|
output | string | Agent trajectory (JSON array of steps or comma-separated string) |
Output
| Field | Type | Description |
|---|---|---|
| Result | Pass / Fail | Pass means the step count satisfies the configured constraint (expected_steps, or the min_steps/max_steps range); Fail means it falls outside it |
| Reason | string | A plain-language explanation of the step count assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"step_count",
output='["plan", "search", "answer"]',
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"step_count",
{
output: '["plan", "search", "answer"]'
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Step Count wherever an agent’s execution length itself is a signal of quality, not just its final answer.
- Agent pipelines expected to complete within a fixed or bounded number of tool calls
- Efficiency checks, to catch agents padding out trajectories with redundant steps
- Regression checks after a planner or prompt change, to confirm step counts stay within budget
What to do when Step Count fails
If it’s failing high, the agent is taking too many steps, look for loops or redundant tool calls it could skip. If it’s failing low, the agent is short-circuiting, likely answering before gathering enough information. Either adjust the agent’s planning logic or revisit the configured expected_steps, min_steps, and max_steps bounds if they don’t match realistic task lengths.
Questions & Discussion