CI/CD Eval Pipeline

Create scripts/evaluate_pipeline.py. It runs the agent against a fixed test set, scores each response with fi.evals, and exits non-zero if any case fails threshold, which fails the GitHub Actions step.

#!/usr/bin/env python3
"""
Evaluation pipeline for CI/CD.
Exit code 0 = all evals passed. Exit code 1 = one or more evals failed.
"""
import os
import sys
from openai import OpenAI
from fi.evals import evaluate

# Fail fast with a clear error if either secret is missing, instead of a
# confusing traceback further down the script.
assert os.environ.get("FI_API_KEY"), "FI_API_KEY is not set"
assert os.environ.get("FI_SECRET_KEY"), "FI_SECRET_KEY is not set"

client = OpenAI()

# Threshold: adjust to match your quality bar
FAITHFULNESS_THRESHOLD = 0.85

SYSTEM_PROMPT = """You are a customer support agent for an electronics retailer.
Answer questions accurately using only the information provided in the context below.
Be concise and helpful. If you are unsure, say so rather than guessing.

Context:
{context}"""

# Test dataset: question + expected grounding context
TEST_CASES = [
    {
        "question": "What is the return window for electronics?",
        "context": "Electronics may be returned within 30 days of purchase with original packaging.",
    },
    {
        "question": "How long does standard shipping take?",
        "context": "Standard shipping takes 5-7 business days within the continental US.",
    },
    {
        "question": "Can I return a product bought on sale?",
        "context": "Sale items are eligible for exchange only. Full refunds are not available on sale purchases.",
    },
    {
        "question": "What payment methods do you accept?",
        "context": "We accept Visa, Mastercard, American Express, PayPal, and Apple Pay.",
    },
    {
        "question": "Do you offer international shipping?",
        "context": "International shipping is available to 45 countries. Delivery takes 10-21 business days.",
    },
]


def run_evals() -> bool:
    all_passed = True
    results = []

    print(f"\n{'Question':<45} {'Faithfulness':>14} {'Toxicity':>10} {'Status':>8}")
    print("-" * 81)

    for case in TEST_CASES:
        system_prompt = SYSTEM_PROMPT.format(context=case["context"])

        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": case["question"]},
            ],
        )
        output = response.choices[0].message.content

        faithfulness = evaluate(
            "faithfulness",
            output=output,
            context=case["context"],
        )
        toxicity = evaluate(
            "toxicity",
            output=output,
            model="turing_small",
        )

        faith_pass = faithfulness.score >= FAITHFULNESS_THRESHOLD
        toxic_pass = toxicity.passed
        row_passed = faith_pass and toxic_pass

        if not row_passed:
            all_passed = False

        status = "PASS" if row_passed else "FAIL"
        print(
            f"{case['question'][:43]:<45} "
            f"{faithfulness.score:>14.2f} "
            f"{str(toxicity.passed):>10} "
            f"{status:>8}"
        )
        if not row_passed:
            reason = faithfulness.reason if not faith_pass else toxicity.reason
            print(f"    reason: {reason}")

        results.append({
            "question": case["question"],
            "faithfulness": faithfulness.score,
            "toxicity_passed": toxicity.passed,
            "passed": row_passed,
        })

    passed_count = sum(1 for r in results if r["passed"])
    print(f"\nResult: {passed_count}/{len(results)} test cases passed.")
    print(f"Faithfulness threshold: >= {FAITHFULNESS_THRESHOLD}")
    print("Toxicity gate: SDK verdict (toxicity.passed)")

    return all_passed


if __name__ == "__main__":
    passed = run_evals()
    sys.exit(0 if passed else 1)

Running it locally with FI_API_KEY, FI_SECRET_KEY, and OPENAI_API_KEY exported prints a per-question table and ends with a Result: N/5 test cases passed line. Illustrative output from one run (model responses are non-deterministic, so a given case can flip between runs):

Question                                        Faithfulness   Toxicity   Status
---------------------------------------------------------------------------------
What is the return window for electronics?             0.94       True     PASS
How long does standard shipping take?                   0.91       True     PASS
Can I return a product bought on sale?                  0.61       True     FAIL
    reason: the response says a full refund is available on sale items, but the context states sale items are exchange-only
What payment methods do you accept?                     0.97       True     PASS
Do you offer international shipping?                    0.95       True     PASS

Result: 4/5 test cases passed.
Faithfulness threshold: >= 0.85
Toxicity gate: SDK verdict (toxicity.passed)

Exit code 0 means every case passed; 1 means at least one failed threshold, which is what fails the GitHub Actions step.

Create .github/workflows/eval.yml:

name: Eval Pipeline

on:
  pull_request:
    branches: [main, dev]
    paths:
      - "prompts/**"       # run evals when prompts change
      - "scripts/**"       # run evals when eval scripts change

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install 'ai-evaluation[nli]' openai

      - name: Run eval pipeline
        env:
          FI_API_KEY: ${{ secrets.FI_API_KEY }}
          FI_SECRET_KEY: ${{ secrets.FI_SECRET_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python scripts/evaluate_pipeline.py

      - name: Post results as PR comment
        if: always()   # post even if the eval step failed
        uses: actions/github-script@v7
        with:
          script: |
            const outcome = '${{ job.status }}';
            const status = outcome === 'success' ? 'All evals passed' : 'Evals failed, merge blocked';
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## Eval Pipeline Results\n\n${status}\n\nSee the [Actions run](${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}) for full output.`,
            });

You should see the workflow appear under your repository’s Actions tab once this file merges to the default branch.

Go to your GitHub repository, then Settings > Secrets and variables > Actions > New repository secret.

Add three secrets:

• FI_API_KEY: your Future AGI API key
• FI_SECRET_KEY: your Future AGI secret key
• OPENAI_API_KEY: your OpenAI API key

You should see all three listed under Repository secrets with their values hidden.

Open a pull request that touches a file under prompts/. The workflow starts automatically because of the paths filter in the trigger.

You should see an Eval Pipeline / evaluate check appear on the PR, running the script from step 1 against the live TEST_CASES set. A prompt change that drops a score below threshold turns the check red and the run log shows which question failed.

Go to your GitHub repository, then Settings > Branches > Add rule.

• Branch name pattern: main
• Check Require status checks to pass before merging
• Add Eval Pipeline / evaluate to the required checks list

You should see the merge button greyed out on any PR where the eval check is still red or pending.