Trajectory Match

Compares an agent's actual action sequence against an expected trajectory using configurable matching modes.

Trajectory Match checks whether an agent executed the right sequence of actions, not just whether it reached the right answer. Run it wherever the path an agent takes matters as much as the outcome.

What it does

Trajectory Match is a code-based check. It parses the actual and expected action sequences, then scores how closely they match according to the configured mode: strict compares the two sequences in order and scores the length of the matching prefix, unordered treats both sequences as sets and scores their Jaccard overlap, subset checks that every expected action appears in the actual trajectory, and superset checks that every actual action was expected. The default mode is strict.

Input

Required Input	Type	Description
`output`	`string`	Actual trajectory (JSON array of action names or objects with a “name” field)
`expected`	`string`	Expected trajectory (same format)

Output

Field	Type	Description
Result	`score`	Higher scores indicate the actual action sequence matches the expected trajectory more closely, per the configured mode
Reason	`string`	A plain-language explanation of the score

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "trajectory_match",
    output='["search", "read", "answer"]',
    expected='["search", "read", "answer"]',
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "trajectory_match",
  {
    output: '["search", "read", "answer"]',
    expected: '["search", "read", "answer"]'
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Trajectory Match wherever an agent’s tool-call order or action plan is part of the spec, not just its final output.

Agent pipelines with a known-correct sequence of tool calls, like search then read then answer
Regression checks after a prompt or planner change, to confirm the agent still follows the expected path
Debugging agents that reach a correct final answer through an inefficient or unexpected route

What to do when Trajectory Match fails

Start by picking the mode that matches your use case: strict if order matters, unordered or subset if it doesn’t. Then inspect where the actual sequence diverged from the expected one, the reason field reports the matching prefix length in strict mode or the missing and extra actions in subset and superset modes.

If the agent is consistently taking a different but valid route, switch from strict to unordered or subset rather than treating every reordering as a failure.

Was this page helpful?

Questions & Discussion