Advanced usage

An Agent Evaluator reasons over the text you hand it, but a connectors menu in its model bar lets it do more: reach the web, call your tools, read your knowledge bases, pull in extra context, and shape how it writes up its verdict. This guide walks every option in that menu, and how what you turn on gets saved.

Open the connectors menu

Open an Agent Evaluator, like the hateful_speech custom eval from Create a custom eval. In the model bar under the instructions, next to the eval type and model, click the + button to open the connectors menu.

The + button sits in the model bar, under the instructions

The connectors menu lists five capabilities. Each one you turn on shows up as a chip in the model bar, so you can see at a glance what the evaluator has.

Use Internet

A toggle that lets the evaluator search the web while it judges, for verdicts that depend on public facts the response doesn’t carry. Switch Use Internet on, and an Internet chip appears in the model bar.

The five capabilities in the connectors menu

Connectors

Connectors let the evaluator call your own tools mid-judgment, the same way it uses web search, so it can check a claim against your database, confirm an ID exists, or verify a rule in an internal service. A connector is a tool you expose over the Model Context Protocol (MCP) and register once on the MCP Connectors page; this section assumes you have one registered.

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart LR
  R["Response to judge"] --> AE["Agent Evaluator"]
  AE -->|"needs an outside fact"| T["Tool<br/>web search or your connector"]
  T -->|"result"| AE
  AE --> V["Verdict + reason"]
  style T fill:#2f2f2f,stroke:#ffffff,stroke-width:2px

In the connectors menu, open Connectors and pick the ones this eval should call. Registered connectors belong to your whole organization and are shared across evals (and with Falcon AI). If yours isn’t listed yet, add it on the MCP Connectors page, or open Manage connectors to jump straight there.

Pick a registered connector, or open Manage connectors to add one

Knowledge Base

Attach one or more knowledge bases and the evaluator can look up reference context while it judges, like your policies, SOPs, or style guides. In the connectors menu, choose Knowledge Base and select from the searchable list, or Create in Knowledge Base if you don’t have one yet.

Attach a knowledge base, or create one if you have none yet

Data Injection

By default an eval sees only the {{variables}} you mapped to your data when you set the eval up. Data Injection widens that, so the evaluator also reads surrounding context. In the connectors menu, choose Data Injection and toggle the context to include:

Option	What the evaluator also receives
Template variables	Only the mapped {{variables}} (the default)
Dataset row context	All columns from the current dataset row
Call context	The call transcript, recording, and scenario
Full span context	The complete span, including its metadata
Trace context	The full trace tree, every span in the request
Session context	The full conversation history

Pick the context the evaluator receives beyond the mapped variables

For example, when you attach an eval as a task on your traces, its Data Injection level is the level it runs on: trace, span, or session. The screenshot below shows a different eval, the built-in toxicity one, with Trace context turned on so it scores across the whole trace instead of a single span, which is what you need when an eval’s variables live on different spans. See the errors table for the “not in row” case it solves.

In a task, the context level you pick is the level the eval runs on

Summary

Summary controls how the evaluator writes up its result, from a raw verdict to a full explanation. In the connectors menu, choose Summary and pick a preset:

• None: return the raw evaluation output with no summary
• Short: a brief summary with the key points only
• Long: a detailed summary with full context and explanations
• Concise: a compact summary focusing on essential insights

Pick how detailed the write-up should be

For anything the presets don’t cover, choose Create custom template, name it, and write your own summary criteria to reuse later.

Write and save your own summary criteria as a reusable template

Save what you turned on

Everything you enable in the connectors menu is part of the eval’s configuration, shown as chips in the model bar. Whether those choices are saved depends on the template.

Enabled capabilities show as chips; Save Version records them

On a custom eval

A custom eval you own can carry these choices in the template itself. Click Save Version to record them into a new version.

Saving records your choices into a new version, here V2

The latest version becomes the default, and when you attach the eval as a task you can pick which version runs.

The latest version runs by default; switch to an earlier one from the version dropdown

On a built-in eval

A built-in eval is read-only, so you can’t save a version. Set these capabilities when you configure a run, like attaching it as a task on your traces, and they apply to that run only, never saved back into the template. That’s what the built-in toxicity eval did earlier: its Trace context was set for that one task, not stored on the template.

Keep exploring