Compare AI SDR Opener Prompts

The dataset holds one row per prospect: a value_proposition, the prospect’s combined_posts (their recent LinkedIn posts), and two candidate openers already generated by two different prompts.

import pandas as pd

dataset = pd.DataFrame([
    {
        "value_proposition": "Get location information of your social media following to place better ads and sponsorships",
        "combined_posts": "Post 1: In the past 12 months, my LinkedIn following went from 36k to 58k... Post 2: Pro-tip that booked me 4-5 meetings from my top accounts per quarter...",
        "opener_1": "I recently saw your post about leveraging LinkedIn for building a pipeline; location insights could enhance your ad strategies even further!",
        "opener_2": "I recently saw your post about leveraging your LinkedIn presence to build a pipeline, which aligns perfectly with optimizing audience targeting",
    },
    {
        "value_proposition": "Benchmark your support team's response time against industry peers",
        "combined_posts": "Post 1: We cut our median first-response time to 4 minutes this quarter... Post 2: Sharing our support playbook at next month's CX meetup...",
        "opener_1": "Congrats on the 4-minute response time, curious how that benchmarks against the rest of your support org",
        "opener_2": "I saw your post about the CX meetup and wanted to reach out because our platform helps teams like yours track support benchmarks",
    },
])
pd.set_option("display.max_colwidth", None)

Replace this with your own prospect rows, or pd.read_csv("your_file.csv") once you have a real export with the same four columns.

Each row looks like this (posts are shortened here for readability, yours will run longer):

opener_1 came from a short, direct generation prompt. opener_2 came from a longer prompt with explicit style instructions. You should see one row per prospect with all four columns populated.

import os

from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

You should see no output.

Run one call against a single row before scoring the full dataset, so you catch a bad API key or a misnamed eval early:

smoke_test = evaluator.evaluate(
    eval_templates="custom_deterministic_eval",
    inputs={
        "opener": dataset.iloc[0]["opener_1"],
        "combined_posts": dataset.iloc[0]["combined_posts"],
        "value_proposition": dataset.iloc[0]["value_proposition"],
        "description": "Evaluate whether the opener captures attention and encourages interaction or further thought. Choose Good if the opener is engaging, sparks curiosity, or creates a sense of interest. Choose Poor if it feels generic or uninspiring.",
    },
    model_name="turing_flash",
)
print(smoke_test.eval_results[0].output)

You should see Good or Poor printed, this is the same eval_results[0].output field step 5 reads at scale, and it confirms the eval and credentials both work before you loop over the full dataset.

Create a deterministic eval in your Future AGI project with this config, so evaluate() in step 5 can call it by name.

{description} is filled per criterion in step 4, so the same eval definition is reused for all five judging criteria.

You should see the eval listed by name in your project’s eval list, with Good and Poor as its deterministic choices once it saves.

Each criterion is a description string that gets substituted into the eval’s {description} placeholder. The eval returns Good or Poor for each one.

JUDGING_CRITERIA = {
    "Engagement": "Evaluate whether the opener captures attention and encourages interaction or further thought. Choose Good if the opener is engaging, sparks curiosity, or creates a sense of interest, making the reader want to engage further. Choose Poor if the opener feels generic, uninspiring, or fails to prompt any interaction or interest.",
    "Tone": "Evaluate whether the tone of the opener is respectful, professional, and avoids being patronizing or condescending. Choose Good if the tone matches the context, feels approachable, and conveys professionalism without being overly casual or rigid. Choose Poor if the tone is overly formal, dismissive, condescending, or inappropriate for the intended audience.",
    "Relevance": "Evaluate whether the opener is relevant to the combined posts. Choose Good if the opener aligns closely with the topic, addresses the subject matter accurately, and stays on-point. Choose Poor if the opener feels disconnected, includes irrelevant information, or strays from the primary focus of the combined posts.",
    "Appropriateness": "Evaluate whether the correct post from the combined posts was selected to create the opener. Choose Good if the selected post clearly supports the value proposition and fits well with the purpose of the opener. Choose Poor if the selection feels irrelevant, random, or poorly suited to the context or value proposition.",
    "Impact": "Evaluate how compelling and effective the opener is in delivering its message. Choose Good if the opener leaves a strong impression, effectively conveys its value proposition, and makes the reader want to engage further. Choose Poor if the opener feels weak, ineffective, or fails to make a memorable or persuasive impact.",
}

Add your own criteria the same way, as long as each description tells the model exactly how to choose between Good and Poor.

You should see no output. JUDGING_CRITERIA now holds five entries, one per criterion, ready to substitute into the eval’s {description} placeholder in step 5.

For each criterion and each row, run the eval once on opener_1 and once on opener_2, then read the tag off eval_results[0].output.

complete_result = {}
for criterion, description in JUDGING_CRITERIA.items():
    results_1 = []
    for _, row in dataset.iterrows():
        result_1 = evaluator.evaluate(
            eval_templates="custom_deterministic_eval",
            inputs={
                "opener": row["opener_1"],
                "combined_posts": row["combined_posts"],
                "value_proposition": row["value_proposition"],
            },
            model_name="turing_flash",
        )
        results_1.append(result_1.eval_results[0].output)

    results_2 = []
    for _, row in dataset.iterrows():
        result_2 = evaluator.evaluate(
            eval_templates="custom_deterministic_eval",
            inputs={
                "opener": row["opener_2"],
                "combined_posts": row["combined_posts"],
                "value_proposition": row["value_proposition"],
            },
            model_name="turing_flash",
        )
        results_2.append(result_2.eval_results[0].output)

    complete_result[f"{criterion} Eval Rating 1"] = results_1
    complete_result[f"{criterion} Eval Rating 2"] = results_2

complete_result_df = pd.DataFrame(complete_result)

EvalResult has no metrics field: the deterministic tag comes back on .output. You should see complete_result_df with ten columns, two (Rating 1 and Rating 2) per criterion, each cell either Good or Poor.

Split the interleaved columns back into one table per prompt and print them.

from tabulate import tabulate

complete_result_prompt1 = complete_result_df.iloc[:, ::2].copy()
complete_result_prompt1.columns = [c.replace(" Eval Rating 1", "") for c in complete_result_prompt1.columns]

complete_result_prompt2 = complete_result_df.iloc[:, 1::2].copy()
complete_result_prompt2.columns = [c.replace(" Eval Rating 2", "") for c in complete_result_prompt2.columns]

print("\nEvaluation on Prompt 1")
print(tabulate(complete_result_prompt1, headers="keys", tablefmt="fancy_grid", showindex=False))

print("\nEvaluation on Prompt 2")
print(tabulate(complete_result_prompt2, headers="keys", tablefmt="fancy_grid", showindex=False))

Illustrative output for prompt 1, your tags depend on your dataset and eval run:

Row 1’s opener_1 scored Poor on Appropriateness in the table above. The value proposition is about location insights for ad targeting, but the opener leans on a generic “building a pipeline” framing instead of naming which post the location angle actually came from:

“I recently saw your post about leveraging LinkedIn for building a pipeline; location insights could enhance your ad strategies even further!”

The Appropriateness criterion asks whether the opener selected the right post to justify the value proposition, and this opener never points at a specific post, so the eval has nothing concrete to confirm. Tighten the criterion’s description to require that the opener name or clearly reference a specific post before it can score Good:

JUDGING_CRITERIA["Appropriateness"] = (
    "Evaluate whether the opener explicitly references a specific post from combined_posts "
    "(a metric, a quote, or a named topic) to justify the value proposition. Choose Good only if "
    "the opener ties back to something concrete in one of the posts. Choose Poor if the opener is "
    "generic and could have been sent regardless of which posts the prospect wrote."
)

rerun = evaluator.evaluate(
    eval_templates="custom_deterministic_eval",
    inputs={
        "opener": dataset.iloc[0]["opener_1"],
        "combined_posts": dataset.iloc[0]["combined_posts"],
        "value_proposition": dataset.iloc[0]["value_proposition"],
        "description": JUDGING_CRITERIA["Appropriateness"],
    },
    model_name="turing_flash",
)
print(rerun.eval_results[0].output)

You should see Poor still, since opener_1 itself never names a post. Rerun the same call against opener_2, which also stays generic, then rewrite opener_1 to cite the follower-growth post directly (“Congrats on growing to 58k followers, location data could help you double down on the accounts already engaging you”) and rerun once more. That version should flip the tag to Good, confirming the tightened criterion rewards openers that anchor to a real post instead of penalizing both candidates equally.

Take the majority tag across the five criteria for each row, then count how many rows land Good per prompt.

def get_majority(row):
    frequency = row[:5].value_counts()
    return frequency.idxmax()

df1_majority = complete_result_prompt1.apply(get_majority, axis=1)
df2_majority = complete_result_prompt2.apply(get_majority, axis=1)

good_count_prompt1 = (df1_majority == "Good").sum()
good_count_prompt2 = (df2_majority == "Good").sum()

if good_count_prompt1 > good_count_prompt2:
    winner = "Prompt 1"
elif good_count_prompt2 > good_count_prompt1:
    winner = "Prompt 2"
else:
    winner = "TIE"

print(f"\nWinner Prompt: {winner}")

You should see a Winner Prompt line naming the prompt with more majority-Good rows. Ties mean neither opener-generation prompt is clearly stronger on these five criteria, run a larger dataset before deciding.