Evaluate Function Calling

Assesses whether an LLM correctly identifies when to call a function, picks the right tool, and extracts the right parameters.

Evaluate Function Calling checks whether a model correctly recognized that a function or tool call was needed and produced it with the right structure. Run it wherever your model’s output includes tool calls.

What it does

Evaluate Function Calling is an LLM-as-Judge eval. It reads the input that should trigger a function call and the model’s output, then scores whether the function call was correctly identified and formatted.

Input

Required Input	Type	Description
`input`	`string`	Input provided to the LLM that triggers the function call
`output`	`string`	LLM’s output that has the resulting function call or response

Output

Field	Type	Description
Result	`Pass / Fail`	Pass means the LLM correctly identified that a function/tool call was necessary; Fail means it did not correctly handle the function call requirement
Reason	`string`	A plain-language explanation of the function calling evaluation

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "evaluate_function_calling",
    input="Get the weather for London",
    output='{"function": "get_weather", "parameters": {"city": "London", "country": "UK"}}',
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "evaluate_function_calling",
  {
    input: "Get the weather for London",
    output: '{"function": "get_weather", "parameters": {"city": "London", "country": "UK"}}'
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Evaluate Function Calling wherever your model’s output is expected to trigger a tool or function call.

Agentic pipelines where the model chooses between multiple tools
Requests that should always resolve to a specific function, to catch missed or unnecessary calls
Parameter extraction checks, to confirm arguments passed to the function match the input

What to do when Evaluate Function Calling fails

Examine the output to determine whether the failure was missed function call identification or incorrect parameter extraction. If the output didn’t recognize the need for a function call, review the input to make sure the function’s necessity was clearly communicated. If parameters were incorrect or incomplete, check how the model maps input fields to function arguments.

Refining the model’s output or adjusting the function call handling process can help improve accuracy in future evaluations.

Was this page helpful?

Questions & Discussion