Benchmarking LLMs Across Providers with Portkey and Future AGI

Import the instrumentation and gateway libraries, then define two dataclasses: ModelConfig for each provider under test, and TestResult for what a single run produces.

import time
from dataclasses import dataclass

from portkey_ai import Portkey
from traceai_portkey import PortkeyInstrumentor
from fi_instrumentation import register
from fi_instrumentation.fi_types import (
    ProjectType,
    EvalTag,
    EvalTagType,
    EvalSpanKind,
    EvalName,
    ModelChoices,
)
from dotenv import load_dotenv

load_dotenv()

# 1024 tokens comfortably covers every scenario's response, including the
# SQL query with its explanatory clauses, without truncating any of them
MAX_RESPONSE_TOKENS = 1024
# A fixed, moderate temperature keeps sampling variance low so response
# differences across models reflect capability, not randomness
COMPARISON_TEMPERATURE = 0.5


@dataclass
class ModelConfig:
    name: str
    provider: str
    virtual_key: str
    model_id: str


@dataclass
class TestResult:
    model_name: str
    prompt_name: str
    response_text: str
    response_time: float

You should see: no output yet, just a clean import.

setup_tracing registers a Future AGI project and attaches three EvalTag objects, one per built-in eval. Each mapping tells the evaluator which span attribute holds the prompt and which holds the response. Call it once for the whole run: register() raises a ValidationError if you call it again with the same eval_tags under the same project, so this cannot live inside a per-model loop.

def setup_tracing(project_version_name: str):
    """Register a Future AGI project and instrument the Portkey client."""
    tracer_provider = register(
        project_name="Model-Benchmarking",
        project_type=ProjectType.EXPERIMENT,
        project_version_name=project_version_name,
        eval_tags=[
            EvalTag(
                type=EvalTagType.OBSERVATION_SPAN,
                value=EvalSpanKind.LLM,
                eval_name=EvalName.IS_CONCISE,
                custom_eval_name="Is_Concise",
                mapping={"input": "llm.output_messages.0.message.content"},
                model=ModelChoices.TURING_LARGE,
            ),
            EvalTag(
                type=EvalTagType.OBSERVATION_SPAN,
                value=EvalSpanKind.LLM,
                eval_name=EvalName.CONTEXT_ADHERENCE,
                custom_eval_name="Response_Quality",
                mapping={
                    "context": "llm.input_messages.0.message.content",
                    "output": "llm.output_messages.0.message.content",
                },
                model=ModelChoices.TURING_LARGE,
            ),
            EvalTag(
                type=EvalTagType.OBSERVATION_SPAN,
                value=EvalSpanKind.LLM,
                eval_name=EvalName.TASK_COMPLETION,
                custom_eval_name="Task_Completion",
                mapping={
                    "input": "llm.input_messages.0.message.content",
                    "output": "llm.output_messages.0.message.content",
                },
                model=ModelChoices.TURING_LARGE,
            ),
        ],
    )
    PortkeyInstrumentor().instrument(tracer_provider=tracer_provider)
    return tracer_provider

You should see: nothing printed yet, this only wires up tracing. register() must run, and PortkeyInstrumentor().instrument() must fire, before you create any Portkey client, or that client’s calls won’t be traced.

Before building out the full benchmark, prove the pipeline end to end with a single call: one model, one prompt, one traced response with eval scores attached.

tracer_provider = setup_tracing(project_version_name="Benchmark-Run")

client = Portkey(virtual_key="openai-virtual-key-id")

refund_policy_prompt = (
    "Policy: Refunds are issued in full within 30 days of purchase if the item is "
    "unopened. After 30 days, only store credit is issued, and opened items are not "
    "eligible for any refund. A customer opened their order 35 days ago and wants a "
    "refund. What can they receive, and why?"
)

completion = client.chat.completions.create(
    messages=[{"role": "user", "content": refund_policy_prompt}],
    model="gpt-4o",
    max_tokens=MAX_RESPONSE_TOKENS,
    temperature=COMPARISON_TEMPERATURE,
)
print(completion.choices[0].message.content)

You should see: a printed answer citing store credit, since the order was opened and is past the 30-day window. Now open the Prototype tab in your Future AGI dashboard, find the Model-Benchmarking project, and open this single trace.

Every EvalTag from step 2 becomes a scored attribute on the LLM span

That’s the whole pipeline working: a real call, traced, and scored. The rest of this cookbook widens it to every model and every scenario.

List the providers you want to compare and the prompts you’ll send to each. Replace the placeholder virtual_key values with the real IDs from your Portkey dashboard.

def get_models() -> list[ModelConfig]:
    """Model configs, keyed to their Portkey virtual keys."""
    return [
        ModelConfig("GPT-4o", "OpenAI", "openai-virtual-key-id", "gpt-4o"),
        ModelConfig("Claude-3.7-Sonnet", "Anthropic", "anthropic-virtual-key-id", "claude-3-7-sonnet-latest"),
        ModelConfig("Llama-3-70b", "Groq", "groq-virtual-key-id", "llama3-70b-8192"),
    ]


def get_test_scenarios() -> dict[str, str]:
    """Prompts to run against every model."""
    return {
        "refund_policy_qa": (
            "Policy: Refunds are issued in full within 30 days of purchase if the item "
            "is unopened. After 30 days, only store credit is issued, and opened items "
            "are not eligible for any refund. A customer opened their order 35 days ago "
            "and wants a refund. What can they receive, and why?"
        ),
        "ticket_summary": (
            "Summarize this support ticket in two sentences for a handoff to billing: "
            "'Customer says their invoice for March shows two charges for the Pro plan. "
            "They were only supposed to be on Pro since March 15th, after upgrading from "
            "Basic. They want the duplicate charge removed and a corrected invoice sent.'"
        ),
        "sql_query": (
            "Given a table `orders(order_id, customer_id, status, created_at)`, write a "
            "SQL query that returns the count of orders with status = 'refunded' per "
            "customer_id, for orders created in the last 90 days."
        ),
    }

You should see: a get_models() call returns 3 ModelConfig objects, one per provider. Add more entries here to widen the benchmark.

test_model is the same call you made in step 3, generalized to take any model and any prompt. Because PortkeyInstrumentor is already active, every call it makes is traced automatically.

def test_model(model_config: ModelConfig, prompt_name: str, prompt: str) -> TestResult:
    """Send one prompt to one model and capture the timed response."""
    client = Portkey(virtual_key=model_config.virtual_key)
    start_time = time.time()

    completion = client.chat.completions.create(
        messages=[{"role": "user", "content": prompt}],
        model=model_config.model_id,
        max_tokens=MAX_RESPONSE_TOKENS,
        temperature=COMPARISON_TEMPERATURE,
    )
    response_time = time.time() - start_time
    response_text = completion.choices[0].message.content or ""

    return TestResult(
        model_name=model_config.name,
        prompt_name=prompt_name,
        response_text=response_text,
        response_time=response_time,
    )

You should see: calling test_model(get_models()[0], "refund_policy_qa", get_test_scenarios()["refund_policy_qa"]) reproduces the same kind of result you already saw traced in step 3, now returned as a TestResult instead of just printed.

main loops every scenario across every model. Tracing is already set up from step 3, so nothing here needs to call setup_tracing again.

def main():
    """Run every model against every scenario and print the results."""
    models_to_test = get_models()
    scenarios = get_test_scenarios()

    for model_config in models_to_test:
        for prompt_name, prompt in scenarios.items():
            result = test_model(model_config, prompt_name, prompt)
            print(f"{result.model_name} / {result.prompt_name}: {result.response_time:.2f}s")


if __name__ == "__main__":
    main()

You should see: one printed line per model/scenario pair, 9 lines total for 3 models and 3 scenarios, each ending in a response time.

Open the Model-Benchmarking project’s Benchmark-Run version again. It now holds every trace from step 6 alongside the single trace from step 3, each carrying the Is_Concise, Response_Quality, and Task_Completion scores from the eval tags you defined in step 2.

The Prototype dashboard lists every trace under one project version, one row per model and scenario

You should see: three eval scores per traced LLM call. A low Response_Quality or Task_Completion score on an otherwise fast model is the signal that latency and quality don’t move together, which is the point of running both tools. The next step walks through one of those low scores.

Open the Llama-3-70b / sql_query trace from step 6. Its Task_Completion score comes back low because the response leaves out the 90-day filter, returning a count per customer_id with no WHERE clause on created_at at all. The prompt asked for a filtered count; the model gave an unfiltered one.

Tighten the prompt to name the exact clauses the answer must include, then rerun only that case.

def rerun_with_tighter_prompt(model_config: ModelConfig) -> TestResult:
    """Rerun the sql_query scenario with the filter spelled out explicitly."""
    tightened_prompt = (
        "Given a table `orders(order_id, customer_id, status, created_at)`, write a SQL "
        "query that returns the count of orders with status = 'refunded' per "
        "customer_id. Filter to orders where created_at is within the last 90 days, and "
        "include the WHERE and GROUP BY clauses explicitly in your answer."
    )
    return test_model(model_config, "sql_query_tightened", tightened_prompt)


llama_config = get_models()[2]
result = rerun_with_tighter_prompt(llama_config)
print(result.response_text)

You should see: the rerun’s response now includes both a WHERE created_at >= ... filter and a GROUP BY customer_id clause. Open its trace in the Future AGI dashboard and compare Task_Completion against the first attempt: naming the required clauses in the prompt is what moves the score, not the model choice.

Open your Portkey dashboard to see the operational side: a unified log of every call across OpenAI, Anthropic, and Groq, with cost and latency tracked per request.

Portkey’s log view is where you compare $/request and p95 latency across providers

You should see: one row per call, matching the 9 calls main() made plus the two single calls from steps 3 and 8. Put this next to the Future AGI eval scores from the previous steps to pick a model on cost, speed, and quality together, not on cost alone.