Experimentation: Compare Prompts and Models on a Dataset

Go to app.futureagi.com. Select Dataset > Add Dataset > Upload a file (JSON, CSV).

Save as experiment-data.csv and upload:

question,context,expected_answer
"What year was the Eiffel Tower completed?","The Eiffel Tower was completed in 1889 after two years of construction.","1889"
"Who designed the Eiffel Tower?","The Eiffel Tower was designed by engineer Gustave Eiffel and his team.","Gustave Eiffel and his team"
"How tall is the Eiffel Tower?","The Eiffel Tower stands 330 metres tall including the antenna at the top.","330 metres"
"Where is the Eiffel Tower located?","The Eiffel Tower is located on the Champ de Mars in Paris, France.","Champ de Mars in Paris, France"
"How many visitors does the Eiffel Tower receive annually?","The Eiffel Tower receives approximately 7 million visitors per year.","Around 7 million"

Open your dataset → click Experiment in the dataset toolbar.

The Run Experiment drawer opens with the subtitle “Test, validate, and compare different prompt configurations.”

Fill in the top-level fields:

The first Prompt Template 1 accordion is already open. Fill in:

1. Prompt Name: baseline-prompt
2. Choose a model type: Select LLM (other options: Text-to-Speech, Speech-to-Text, Image Generation)
3. Models: Select one or more models — e.g. gpt-4o-mini and gpt-4o. You can select multiple models per prompt to compare model performance too.
4. Write the prompt messages:

System message:

You are a helpful assistant. Answer questions using only the provided context.

User message:

Context: {{context}}
Question: {{question}}

Click Add Another Prompt.

A new Prompt Template 2 accordion appears. Fill in:

1. Prompt Name: cot-prompt
2. Choose a model type: LLM
3. Models: Select the same models (gpt-4o-mini, gpt-4o)
4. Write the prompt messages:

System message:

You are a precise question-answering assistant. Use only the information provided in the context — do not add any external knowledge.

User message:

Step 1: Read the context carefully.
Step 2: Identify the specific fact that answers the question.
Step 3: Write a concise answer using only that fact.

Context: {{context}}
Question: {{question}}

Answer:

Click Run.

The platform runs both prompt templates across all selected models on every dataset row and generates outputs.

Once the experiment finishes, go to the Data tab in the experiment detail view.

1. Click Evaluate (top-right of the Data tab)
2. The Evaluation drawer opens — add groundedness
3. Map keys: output → the generated output column, context → context, input → question
4. Run the evaluation

Eval scores appear as grouped columns under the evaluation metric name (e.g. groundedness). Within each group, each prompt variant’s score is shown side by side — e.g. groundedness-baseline-prompt-gpt-4o-mini, groundedness-cot-prompt-gpt-4o-mini — so you can compare scores across variants at a glance.

Switch to the Summary tab to see:

• Summary table — aggregate scores per prompt variant and model, including average response time, total tokens, and completion tokens
• Spider chart — visual comparison of all evaluation metrics across variants
• Evaluation charts — per-metric score distribution across prompt/model combinations

1. Click Choose Winner (crown icon) in the Summary tab
2. The Winner Settings drawer opens — adjust importance weights (0–10 scale) for:
   • Evaluation metrics (e.g. groundedness)
   • Average Response Time
   • Completion Tokens
   • Total Tokens
3. Click Save & Run

The winning variant is ranked in the summary table.