Custom Eval Metrics

Custom evals are created in the platform, then called by name from the SDK. Go to app.futureagi.com → Evals (left sidebar under BUILD) and click the Create your own evals card.

You should see the Custom Evaluations drawer open with Add Details, Configure Parameters, and Choose Output Type.

Fill in the form:

• Name: support_quality (lowercase, underscores only)
• Evaluation type: select Use Future AGI Agents
• Language Model: TURING_SMALL
• Output Type: Pass/Fail

Write the Rule Prompt using {{variable_name}} for the values you pass in at run time:

You are evaluating a customer support response.

The customer asked: {{user_query}}
The agent responded: {{agent_response}}

Mark PASS only if all of these are true:
- It acknowledges the customer's specific issue
- It gives a concrete next step or resolution
- It maintains a professional and empathetic tone

Mark FAIL if any required condition is missing, or if the response is dismissive, vague, or off-topic.

Return a clear PASS/FAIL decision with a short reason.

Click Create Evaluation. You should see a support_quality card in the Evaluators grid, tagged USER_BUILT (filter Eval Categories to User Built to isolate it), ready to select in Dataset and Simulation evaluation flows.

Evaluator from ai-evaluation calls a custom eval by its registered name. Pass the same variable names used in the Rule Prompt. By default the run is scored by the model you set in the dashboard. Pass model_name to evaluate() only when you want to override it.

import os
from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

result = evaluator.evaluate(
    eval_templates="support_quality",
    inputs={
        "user_query": "My order arrived damaged. What do I do?",
        "agent_response": "Please contact our returns department.",
    },
)

eval_result = result.eval_results[0]
print(eval_result.output)
print(eval_result.reason)

You should see a FAIL-style output, with the reason naming the missing resolution step. This is the illustrative shape of the output, not a guaranteed score: the judge model can vary its wording between runs.

Rerun with a response that acknowledges the issue and adds a concrete next step.

result = evaluator.evaluate(
    eval_templates="support_quality",
    inputs={
        "user_query": "My order arrived damaged. What do I do?",
        "agent_response": "I'm sorry to hear that. I've filed a replacement request and you'll receive a shipping confirmation within 24 hours.",
    },
)

eval_result = result.eval_results[0]
print(eval_result.output)
print(eval_result.reason)

You should see the output flip to a PASS-style result, with the reason citing the acknowledgment, resolution, and tone. What changed between the two runs: the second response adds a concrete next step (the replacement request and shipping confirmation) instead of just naming a department to contact.

Repeat step 2 with a weighted rubric instead of a binary one. Use Percentage when you need a continuous score rather than Pass/Fail.

• Name: code_review_quality
• Output Type: Percentage (returned by the SDK as 0.0 to 1.0)
• Rule Prompt:

You are evaluating a code review comment.

The code change: {{code_diff}}
The review comment: {{review_comment}}

Score using these weights:
- 40 points: Does it clearly explain what's wrong?
- 30 points: Does it suggest a concrete fix or improvement?
- 30 points: Is it constructive and respectful?

Return a normalized score from 0.0 to 1.0 (for example, 0.91 for 91/100).

Click Create Evaluation. You should see a code_review_quality card alongside support_quality in the Evaluators grid, tagged USER_BUILT (filter Eval Categories to User Built to isolate it).

Call it the same way, with the variable names from its Rule Prompt.

result = evaluator.evaluate(
    eval_templates="code_review_quality",
    inputs={
        "code_diff": "- return user.name\n+ return user.name.strip()",
        "review_comment": "Good catch: whitespace in names can cause login failures. Consider adding a test case for this.",
    },
)

eval_result = result.eval_results[0]
print(f"Score: {eval_result.output}")
print(f"Reason: {eval_result.reason}")

You should see a score close to 1.0 and a reason breaking it down against the three weighted criteria. This is illustrative, not a fixed value: rerun it and the exact number can shift slightly.