Tool Call Accuracy
Compares an agent's actual tool calls against expected calls, scoring matches on function name and arguments.
Tool Call Accuracy checks whether an agent picked the right tools and passed the right arguments, by comparing its actual calls against an expected set. Run it wherever an agent’s correctness depends on invoking tools correctly, not just producing a reasonable-looking final answer.
What it does
Tool Call Accuracy is a code-based check. It parses the actual and expected tool calls (accepting both the plain {"name": ..., "arguments": ...} format and the OpenAI {"function": {...}} format), then greedily matches each actual call to the best unused expected call.
An exact match on both name and arguments scores 1.0 for that call; a match on name only (arguments differ) scores 0.5. The per-call scores are summed and divided by max(len(expected), len(actual)), so extra or missing calls also pull the score down. A higher score means the agent’s tool selection and argument construction line up more closely with what was expected.
Input
| Required Input | Type | Description |
|---|---|---|
output | string | Actual tool calls (JSON string or list). Format: [{"name": "func", "arguments": {...}}] or [{"function": {"name": "func", "arguments": {...}}}] |
expected | string | Expected tool calls (same format as output) |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher scores indicate the agent’s tool calls matched expected calls more closely on name and arguments |
| Reason | string | A plain-language explanation of the score |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"tool_call_accuracy",
output='[{"name": "get_weather", "arguments": {"city": "Paris"}}]',
expected='[{"name": "get_weather", "arguments": {"city": "Paris", "units": "celsius"}}]',
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"tool_call_accuracy",
{
output: '[{"name": "get_weather", "arguments": {"city": "Paris"}}]',
expected: '[{"name": "get_weather", "arguments": {"city": "Paris", "units": "celsius"}}]',
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Tool Call Accuracy wherever an agent’s job includes calling functions or tools correctly, not just generating text.
- Agent pipelines that call external APIs, databases, or functions as part of completing a task
- Regression testing against a fixed set of expected tool calls when a prompt or tool schema changes
- Debugging agents that produce a plausible final answer but arrive at it through the wrong tool calls
What to do when Tool Call Accuracy is low
Read the reason string first: it reports exact matches, name-only matches, and the actual versus expected call counts, which tells you whether the agent is missing calls entirely or getting names right but arguments wrong.
If calls match on name but not arguments, check argument formatting first (types, units, key names, JSON structure) before assuming the agent reasoned incorrectly. If names themselves are wrong, tighten the tool/function descriptions and schema so the agent can disambiguate between similar tools. Missing or extra calls relative to expected usually point to the agent under- or over-triggering tool use, which is worth checking against the prompt’s instructions for when to call a tool at all.
Questions & Discussion