Customer Agent: Prompt Conformance

Measures how well the agent adheres to system prompt constraints, including persona, language, and conversation guidelines.

Customer Agent Prompt Conformance checks whether an agent stays within the rules set by its system prompt: persona, tone, language, and topics it should avoid. Run it to catch agents that drift from their configured behavior.

What it does

Customer Agent Prompt Conformance is an LLM-as-Judge eval. It reads the system prompt and the conversation, then scores how well the agent’s responses adhere to the system prompt.

Input

Required Input	Type	Description
`system_prompt`	`string`	The system prompt defining the agent’s persona, constraints, and behavior guidelines
`conversation`	`string`	The full conversation history between the customer and agent

Output

Field	Type	Description
Result	`score`	Higher values indicate stronger adherence to the system prompt
Reason	`string`	A plain-language explanation of the prompt conformance assessment

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "customer_agent_prompt_conformance",
    system_prompt="You are Aria, a friendly support agent for TechCorp. Always respond in English, maintain a professional tone, and never discuss competitors.",
    conversation="User: Can you compare your product to CompetitorX?\nAgent: I'm not able to make comparisons with other products, but I'd love to tell you about what makes TechCorp's solution great!",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "customer_agent_prompt_conformance",
  {
    system_prompt: "You are Aria, a friendly support agent for TechCorp. Always respond in English, maintain a professional tone, and never discuss competitors.",
    conversation: "User: Can you compare your product to CompetitorX?\nAgent: I'm not able to make comparisons with other products, but I'd love to tell you about what makes TechCorp's solution great!"
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Customer Agent Prompt Conformance wherever an agent operates under a defined persona or system-level constraints.

Branded agents with a defined persona, tone, or set of topics to avoid
Checking whether an agent holds its guardrails under adversarial or off-topic questioning
Regression testing after a system prompt change, to confirm behavior still matches the new rules

What to do when Customer Agent Prompt Conformance fails

Review cases where the agent broke persona or violated a stated constraint, and strengthen the system prompt with explicit rules and examples covering those cases. Add guardrails for topics the agent should never discuss.

Test with adversarial prompts that try to break the agent out of its persona, so gaps surface before real users find them.

Was this page helpful?

Questions & Discussion