SDK & API

Install the library with pip install agent-opt; every import on this page comes from the fi.opt package it provides. For a full walkthrough that builds and runs an optimization end to end, see Optimize from the SDK; for the environment variables and authentication this page’s examples assume, see Install and authenticate.

The shared surface

Every optimizer in agent-opt is built on the same four pieces:

• Evaluator scores outputs
• BasicDataMapper maps dataset fields to what the evaluator expects
• LiteLLMGenerator holds the prompt being optimized
• optimizer.optimize() runs the optimizer and returns an OptimizationResult

This page is the reference for that shared surface. Constructor parameters specific to a single optimizer live on that optimizer’s own reference page.

Evaluator

Evaluator has two construction modes. Provide metric for local evaluation, or provide eval_template together with eval_model_name for evaluation on the Future AGI platform.

Platform mode runs a pre-built Future AGI eval template with no custom code. Local mode uses a custom metric, such as a local LLM-as-a-judge or a rule-based heuristic. See Choosing Evaluation Metrics for Prompt Optimization for worked examples of both modes, including where a local metric instance like my_metric below comes from.

Argument	Type	Default	Mode	Description
eval_template	str	required (Platform)	Platform	Name of the Future AGI platform eval template to run
eval_model_name	str	required (Platform)	Platform	Model the platform eval template runs under
fi_api_key	str	None	Platform	Future AGI API key; falls back to the FI_API_KEY environment variable when omitted
fi_secret_key	str	None	Platform	Future AGI secret key; falls back to the FI_SECRET_KEY environment variable when omitted
metric	BaseMetric	required (Local)	Local	A local metric instance that performs the evaluation
provider	LiteLLMProvider	None	Local	Optional, only used with a local LLM-as-judge metric; defaults from environment variables when omitted (see Install and authenticate for which ones)

from fi.opt.base.evaluator import Evaluator

# Platform mode
evaluator = Evaluator(
    eval_template="summary_quality",
    eval_model_name="turing_flash",
    fi_api_key="your_key",
    fi_secret_key="your_secret"
)

# Local mode
evaluator = Evaluator(metric=my_metric)

BasicDataMapper

BasicDataMapper is the only exported data mapper.

Argument	Type	Default	Description
key_map	dict[str, str]	required	Maps each key the evaluator expects to the dataset column or generator-output key it should read from, as {evaluator_key: source_key}

from fi.opt.datamappers import BasicDataMapper

data_mapper = BasicDataMapper(
    key_map={
        "input": "article",            # dataset column "article" -> evaluator's "input"
        "output": "generated_output"   # generator's output -> evaluator's "output"
    }
)

LiteLLMGenerator

LiteLLMGenerator is the only exported generator.

Argument	Type	Default	Description
model	str	required	LiteLLM-formatted model identifier
prompt_template	str	required	Prompt template the generator fills to produce outputs

from fi.opt.generators import LiteLLMGenerator

generator = LiteLLMGenerator(
    model="gpt-4o-mini",
    prompt_template="Summarize this article: {article}"
)

model is routed through LiteLLM, so it needs its provider’s API key set as an environment variable, for example OPENAI_API_KEY for gpt-4o-mini above. See Install and authenticate for the pattern.

optimize()

Every optimizer implements optimize() with the same core parameters below as explicit named arguments, not as **kwargs; a runnable instance follows the argument table. optimizer is an instance of one of the optimizer classes on the Optimizers reference page, built in that example as RandomSearchOptimizer from the generator above.

optimizer.optimize(evaluator, data_mapper, dataset, initial_prompts, early_stopping=None, **kwargs) -> OptimizationResult

Argument	Type	Default	Description
evaluator	Evaluator	required	The Evaluator instance that scores generated outputs
data_mapper	BasicDataMapper	required	The BasicDataMapper instance that maps dataset and output keys
dataset	list[dict]	required	The dataset to evaluate against
initial_prompts	list[str]	required (not accepted by Random Search)	Starting prompt(s) the optimizer refines; see each optimizer’s reference page
early_stopping	EarlyStoppingConfig	None	Stops the run before it reaches its maximum iterations; see EarlyStoppingConfig below
**kwargs		optional	Optimizer-specific keyword arguments, where the optimizer accepts them (see each optimizer’s reference page); Meta-Prompt and GEPA accept no **kwargs passthrough at all, so anything beyond their own named parameters (task_description, num_rounds, and eval_subset_size for Meta-Prompt; max_metric_calls for GEPA) raises a TypeError

from fi.opt.optimizers import RandomSearchOptimizer

dataset = [
    {"article": "..."},
    {"article": "..."}
]

optimizer = RandomSearchOptimizer(generator=generator, num_variations=3)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset
)

evaluator and data_mapper are the values built in the sections above; optimizer is built just above from generator. See Optimize from the SDK for a full walkthrough that builds dataset.

EarlyStoppingConfig

Pass an EarlyStoppingConfig instance as the early_stopping keyword argument to optimize() to stop a run before it reaches its maximum iterations. All fields are optional. Early stopping turns on when patience, min_score_threshold, or max_evaluations is set; min_delta alone does not enable it, it only tunes the patience counter. When more than one field is set, optimization stops as soon as any one of them is satisfied.

Field	Bounds	Description
patience	greater than 0	Stop after this many consecutive iterations with no score improvement
min_score_threshold	0.0 to 1.0	Stop once the score reaches or exceeds this threshold
max_evaluations	greater than 0	Stop once this many total dataset evaluations have run across all iterations; checked before the score threshold
min_delta	0.0 or greater	Minimum score improvement counted as progress

optimize() always returns an OptimizationResult, whether or not a stopping criterion triggered; check early_stopped and stop_reason on the result to see what happened.

from fi.opt.utils.early_stopping import EarlyStoppingConfig

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset,
    early_stopping=EarlyStoppingConfig(
        patience=3,
        min_score_threshold=0.9,
        min_delta=0.01
    )
)

Return values

OptimizationResult

The object optimize() returns.

Field	Description
best_generator	The generator holding the best-performing prompt found
history	List of IterationHistory records, one per iteration
final_score	The best score achieved during the run
early_stopped	Whether the run was terminated early by a stopping criterion
stop_reason	Explanation for early stopping, when applicable
total_iterations	Total number of iterations completed
total_evaluations	Total number of dataset evaluations performed

print(result.final_score)
print(result.best_generator.prompt_template)

IterationHistory

A single iteration’s record inside history.

Field	Description
prompt	The prompt evaluated in this iteration
average_score	Mean score across this iteration’s evaluations
individual_results	List of EvaluationResult, one per dataset row

EvaluationResult

A single evaluation’s result, returned inside individual_results.

Field	Description
score	Normalized score, 0.0 to 1.0
reason	Explanation for the score
metadata	Additional evaluator-specific metadata

Keep exploring