Comparing Prompts and Models

Go to app.futureagi.com. Select Dataset > Add Dataset > Upload a file (JSONl/ JSON/ CSV).

Save as experiment-data.csv and upload:

question,context,expected_answer
"What is the return window for a purchase?","Items can be returned within 30 days of delivery for a full refund, provided the item is unused and in its original packaging.","30 days"
"Do you ship internationally?","We currently ship to over 40 countries. International orders take 7-14 business days and customs fees are the customer's responsibility.","Yes, to over 40 countries"
"Can I cancel an order after placing it?","Orders can be cancelled free of charge within 1 hour of placement. After that, the order enters fulfillment and cannot be cancelled.","Yes, within 1 hour of placement"
"How do I reset my account password?","Go to Settings > Security and click 'Reset Password'. A reset link is emailed to the account's registered address and expires after 24 hours.","Use Settings > Security > Reset Password"
"How are loyalty points earned and redeemed?","Customers earn 1 point per dollar spent. Points can be redeemed at checkout starting at 500 points for a $5 discount.","1 point per dollar, redeemable from 500 points"

You should see a 5-row dataset with question, context, and expected_answer columns in the dataset viewer.

Open your dataset and click Experiment in the dataset toolbar.

The Run Experiment drawer opens with the subtitle “Test, validate, and compare different prompt configurations.” Fill in the top-level fields:

The first Prompt Template 1 accordion is already open. Fill in:

1. Prompt Name: baseline-prompt
2. Choose a model type: select LLM (other options: Text-to-Speech, Speech-to-Text, Image Generation)
3. Models: select one or more models, e.g. gpt-4o-mini and gpt-4o. You can select multiple models per prompt to compare model performance too
4. Write the prompt messages:

System message:

You are a helpful assistant. Answer questions using only the provided context.

User message:

Context: {{context}}
Question: {{question}}

You should see baseline-prompt with both models selected and context and question picked up as detected variables from the messages.

Click Add Another Prompt. A new Prompt Template 2 accordion appears. Fill in:

1. Prompt Name: cot-prompt
2. Choose a model type: LLM
3. Models: select the same models (gpt-4o-mini, gpt-4o)
4. Write the prompt messages:

System message:

You are a precise question-answering assistant. Use only the information provided in the context. Do not add any external knowledge.

User message:

Step 1: Read the context carefully.
Step 2: Identify the specific fact that answers the question.
Step 3: Write a concise answer using only that fact.

Context: {{context}}
Question: {{question}}

Answer:

You should see two prompt template accordions, baseline-prompt and cot-prompt, each with its own models and messages.

Click Run.

The platform runs both prompt templates across all selected models on every dataset row and generates outputs. You should see a run status that moves to complete once every row has an output for every prompt/model combination.

Once the experiment finishes, go to the Data tab in the experiment detail view.

1. Click Evaluate (top-right of the Data tab)
2. The Evaluation drawer opens. Add groundedness
3. Map keys: output to the generated output column, context to context, input to question
4. Run the evaluation

Eval scores appear as grouped columns under the evaluation metric name (e.g. groundedness). Within each group, each prompt variant’s score is shown side by side, e.g. groundedness-baseline-prompt-gpt-4o-mini, groundedness-cot-prompt-gpt-4o-mini, so you can compare scores across variants at a glance.

Switch to the Summary tab to see:

• Summary table: aggregate scores per prompt variant and model, including average response time, total tokens, and completion tokens
• Spider chart: visual comparison of all evaluation metrics across variants
• Evaluation charts: per-metric score distribution across prompt/model combinations

You should see cot-prompt and baseline-prompt as separate rows in the summary table, each with its own groundedness score.

The actual groundedness scores from the run behind these videos were not recorded, so no baseline-vs-cot-prompt delta is reported here. Re-run this experiment and note the two averages before publishing an outcome.

1. Click Choose Winner (crown icon) in the Summary tab
2. The Winner Settings drawer opens. Set importance weights (0-10 scale) for:
   • Evaluation metrics (e.g. groundedness)
   • Average Response Time
   • Completion Tokens
   • Total Tokens
3. Click Save & Run

The winning variant is ranked at the top of the summary table.