Create Your Own Evaluations

​

Why Create Custom Evaluations?

While Future AGI offers comprehensive evaluation templates, custom evaluations are essential when you need:

• Domain-Specific Validation: Assess content against industry-specific standards or regulations
• Business Rule Compliance: Ensure outputs meet your organization’s unique guidelines
• Complex Scoring Logic: Implement multi-criteria assessments with weighted scoring
• Custom Output Formats: Validate specific response structures or formats unique to your application

​

Creating Custom Evaluations

​

Using Web Interface

Step 1: Access Evaluation Creation

• Navigate to your dataset in the Future AGI platform
• Click on the Evaluate button in the top-right menu
• Click on Add Evaluation button
• Select Create your own eval

Step 2: Configure Basic Settings

Start by setting up the fundamental properties of your evaluation:

• Name: Enter a unique evaluation name (lowercase letters, numbers, and underscores only)

• Model Selection: Choose the appropriate model for your evaluation complexity:

  • Future AGI Models: Proprietary models optimized for evaluations
  • TURING_LARGE turing_flash: Flagship evaluation model that delivers best-in-class accuracy across multimodal inputs (text, images, audio). Recommended when maximal precision outweighs latency constraints.

  • TURING_SMALL turing_small: Compact variant that preserves high evaluation fidelity while lowering computational cost. Supports text and image evaluations.

  • TURING_FLASH turing_flash: Latency-optimised version of TURING, providing high-accuracy assessments for text and image inputs with fast response times.

  • PROTECT protect: Real-time guardrailing model for safety, policy compliance, and content-risk detection. Offers very low latency on text and audio streams and permits user-defined rule sets.

  • PROTECT_FLASH protect_flash: Ultra-fast binary guardrail for text content. Designed for first-pass filtering where millisecond-level turnaround is critical.

  • Other LLMs: Use external language models from providers like OpenAI, Anthropic, or your own custom models.

Step 3: Define Evaluation Rules

• Rule Prompt: Write the evaluation criteria and instructions
• Use {{variable_name}} syntax to create dynamic variables that will be mapped to dataset columns
• Be specific about what constitutes a pass/fail or scoring criteria

Step 4: Configure Output Type

• Pass/Fail: Binary evaluation (1.0 for pass, 0.0 for fail)
• Percentage: Numerical score between 0 and 100
• Deterministic Choices: Select from predefined categorical options

Step 5: Additional Settings

• Tags: Add relevant tags for organization and filtering
• Description: Provide a clear description of the evaluation’s purpose
• Check Internet: Enable web access for real-time information validation

​

Example: Creating a Chatbot Evaluation

Let’s walk through creating a custom evaluation for a customer service chatbot. This example will show how to ensure the chatbot’s responses are both polite and effectively address user queries.

​

Step 1: Basic Configuration

• Name: chatbot_politeness_and_relevance
• Model Selection: TURING_SMALL (ideal for straightforward evaluations like this)
• Description: “Evaluates if the chatbot’s response is polite and relevant to the user’s query.”

​

Step 2: Define Evaluation Rules

Create a rule prompt that clearly specifies the evaluation criteria:

Evaluate the chatbot's response based on two criteria:
1.  **Politeness**: Is the language used courteous and respectful?
2.  **Relevance**: Does the response directly address the user's query: '{{user_query}}'?

The user's query was: {{user_query}}
The chatbot's response was: {{chatbot_response}}

Provide a pass/fail score. The response passes if it is both polite and relevant.

​

Step 3: Configure Output

• Output Type: Pass/Fail (1.0 for pass, 0.0 for fail)
• Tags: customer-service, politeness, relevance

​

Step 4: Map Variables

In your dataset, map the variables to their corresponding columns:

• {{user_query}} → Column containing user questions
• {{chatbot_response}} → Column containing chatbot responses

This evaluation will help ensure your chatbot maintains high standards of interaction by checking both the tone and relevance of responses. The pass/fail output makes it easy to quickly identify responses that need improvement.

​

Running the Evaluation

You can either run the evaluation through the web interface or using the SDK.

​

Using Web Interface

• Navigate to your dataset in the Future AGI platform
• Click on the Evaluate button in the top-right menu
• Click on the evaluation you just created
• Configure the columns that you want to use for the evaluation
• Click on the Add & Run button

​

Using SDK

After creating the evaluation, you can run it using the FutureAGI SDK.

pip install futureagi
pip install ai-evaluation

from fi.evals import Evaluator

evaluator = Evaluator(fi_api_key="your_api_key", fi_secret_key="your_secret_key")

eval_result = evaluator.evaluate(eval_templates = "your_eval_template_name", # Name of the evaluation template to use
                                inputs = {
                                    "input": "your_input_text",
                                    "output": "your_output_text"
                                }, # Data can be sent as a dictionary or using TestCase class
                                timeout = 10, #Optional Timeout in seconds
                                model_name = "turing_small" #  Model name to use for Future AGI Built Evals
)

print(eval_result)