Bayesian Search

What Bayesian Search tunes

Bayesian Search tunes the few-shot examples in a prompt rather than its wording. It searches over how many examples to include and which ones, building a model of which configurations score well and spending its trial budget on the promising ones instead of trying every combination. That makes it a fit when the prompt’s wording is already fine and the examples are what’s left to tune. See Choosing an optimizer for how it compares to Random Search and the rest.

The candidate examples come from the same dataset you pass to optimize(): each trial borrows a handful of dataset rows and formats them as few-shot examples, so min_examples and max_examples bound how many rows a single trial can borrow. See Optimize from the SDK for how to build one.

Parameters

The On-screen label column is the field name shown when you run this optimizer from the UI; see Run an optimization for the full form walkthrough. All three keys are required when you submit the optimization from the platform; in the SDK they’re optional keyword arguments. The Default column gives the UI’s prefilled value and the SDK constructor’s fallback when the argument is omitted.

Parameter	On-screen label	Default	Description
min_examples	Min examples	2 prefilled in the UI, 2 in the SDK	Minimum number of few-shot examples to include in a trial. Fewer examples means less context per trial and a cheaper run
max_examples	Max examples	4 prefilled in the UI, 8 in the SDK	Maximum number of few-shot examples to include in a trial. More examples means more context but a longer, costlier prompt per trial
n_trials	No.of trials	5 prefilled in the UI, 10 in the SDK	Number of configurations the optimizer tries. Raising it searches more configurations at the cost of more evaluator calls

This table covers only the parameters specific to Bayesian Search that appear in the UI. Two other groups of arguments show up in the code below but are documented in the SDK reference instead:

• inference_model_name, the constructor argument that sets which model generates completions during the search
• the optimize() arguments shared by every optimizer: evaluator, data_mapper, dataset, and early_stopping

initial_prompts isn’t one of those shared arguments: it’s required on this optimizer’s optimize() call specifically.

Usage

• Install: pip install agent-opt
• Keys: get fi_api_key and fi_secret_key from Admin Settings (or set FI_API_KEY/FI_SECRET_KEY as environment variables and drop them from the Evaluator call below)

from fi.opt.optimizers import BayesianSearchOptimizer
from fi.opt.datamappers import BasicDataMapper
from fi.opt.base.evaluator import Evaluator

# Dataset: each row can also be drawn as a few-shot example.
# Keep max_examples at or below your row count.
# Each row pairs the `article` input with a target `summary`, so the few-shot examples
# the optimizer samples show the input to output pattern, not just inputs.
my_dataset = [
    {
        "article": "The James Webb Space Telescope has captured its clearest images yet of a distant exoplanet's atmosphere, revealing traces of carbon dioxide and methane.",
        "summary": "JWST found carbon dioxide and methane in a distant exoplanet's atmosphere.",
    },
    {
        "article": "Researchers have discovered a new enzyme that breaks down PET plastic at room temperature, far faster than any previously known enzyme.",
        "summary": "A newly discovered enzyme breaks down PET plastic at room temperature faster than any known before it.",
    },
    {
        "article": "A team of engineers unveiled a compact fusion reactor prototype that sustained plasma for a record twelve minutes under laboratory conditions.",
        "summary": "Engineers unveiled a compact fusion reactor that sustained plasma for a record twelve minutes.",
    },
    {
        "article": "City officials broke ground on a new light rail line intended to cut downtown commute times by nearly half once completed in 2028.",
        "summary": "City officials broke ground on a light rail line meant to cut downtown commute times nearly in half by 2028.",
    },
    {
        "article": "A previously undocumented species of deep-sea octopus was filmed for the first time near hydrothermal vents off the coast of Costa Rica.",
        "summary": "A previously undocumented deep-sea octopus species was filmed for the first time near hydrothermal vents off Costa Rica.",
    },
    # ... add more rows here
]

# Evaluator that scores each configuration.
# "summary_quality" and "turing_flash" are just this example's choices; fi_api_key/fi_secret_key are placeholders (see Admin Settings above)
evaluator = Evaluator(
    eval_template="summary_quality",
    eval_model_name="turing_flash",
    fi_api_key="your_key",
    fi_secret_key="your_secret"
)

# Maps generator output and dataset fields to what the evaluator expects
# "article" here must match the dataset's key above and the {article} placeholder in initial_prompts below
data_mapper = BasicDataMapper(
    key_map={"input": "article", "output": "generated_output"}
)

# --- What's specific to Bayesian Search ---
optimizer = BayesianSearchOptimizer(
    min_examples=2,
    max_examples=4,
    n_trials=10,  # this is the default; raise it to search more configurations
    inference_model_name="gpt-4o-mini"
)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=my_dataset,
    initial_prompts=["Summarize this article: {article}"]
)

print(f"Final score: {result.final_score:.4f}")
print(f"Best prompt:\n{result.best_generator.get_prompt_template()}")

eval_template accepts any of the built-in evaluation templates; eval_model_name accepts any of the evaluator models.

result.best_generator holds the original prompt wording plus the winning set of few-shot examples; result.final_score is that combination’s average score from the evaluator.

Keep exploring