Epoch

Finding a fix for your agent's failing scenarios, and proving it before anything changes

What an epoch is

Epoch searches for a better version of your agent. It starts from the failing scenarios of a finished run in an environment, tries changes to your agent, and keeps the one that fixes the most without breaking anything. Each search is one epoch, numbered in the order you start them: Epoch 1, Epoch 2, and so on. Each version of your agent that an epoch tries is a candidate, numbered the same way: Candidate 1, Candidate 2.

An epoch never changes your agent by itself. It hands you the best candidate to review, with the evidence for it. Your agent changes only when you ship that candidate, as a pull request on your repository or as the environment’s new default version.

%%{init: {"flowchart": {"curve": "basis", "rankSpacing": 46, "nodeSpacing": 36, "padding": 16}}}%%
flowchart TD
  accTitle: How an epoch turns a failing run into a change you can ship
  accDescr: A finished run splits into the failing scenarios to fix, the failing scenarios held back to prove the fix, and the scenarios that must keep passing. The epoch reproduces the failures, tries candidates on the scenarios to fix, and checks the best one on the held-back and must-pass scenarios. It ends ready for review, or with nothing changed. You ship a candidate that is ready for review.
  R["A finished run"] --> T["Failing scenarios to fix"]
  R --> H["Failing scenarios held back<br/>the epoch never sees them"]
  R --> M["Scenarios that must keep passing<br/>including the ones you protect"]
  T --> S["Reproduce, then search<br/>Candidate 1, Candidate 2, …"]
  S --> V["Verify the best candidate"]
  H --> V
  M --> V
  V --> E1["Ready for review"]
  V --> E2["Ended with nothing changed"]
  E1 --> Y["You ship it<br/>a pull request, or the new default version"]

The scenarios an epoch works with

You start an epoch from Debug failures on a finished run, and you choose which of the run’s scenarios it works on. Each scenario takes one of three roles:

  • Target scenarios are the failing scenarios the epoch works on. Every candidate is tried against them
  • Held-back scenarios are failing scenarios the epoch never sees while it searches. Once you pick 12 or more failing scenarios, some of them are held back, and the launch form shows how many. After the epoch picks its best candidate, it runs the held-back scenarios to check that the fix works on cases it wasn’t tuned on
  • Must-pass scenarios are all the other scenarios in the run. A scenario that passed every time before must not get worse beyond normal run-to-run variation. The passing scenarios you protect at launch are in this group by name

An epoch needs 12 failing scenarios. With fewer, it first writes three variations of each one you picked, and keeps the variations your current agent also fails. It needs at least three failing scenarios to start this way, and it holds back some of them together with their variations. With fewer than six, it can’t hold back enough to tell whether a fix generalizes.

A candidate that fixes only the target scenarios has learned those scenarios, not the problem. The held-back scenarios catch that. The must-pass scenarios catch a fix for one case that breaks another.

What an epoch is held to

When you start an epoch, you set its limits, and it works only inside them:

  • it changes only the parts of your agent you allow: its Instructions, Tools, Settings or Code
  • it never edits a file you lock
  • its candidates call only the models you approve, which start as the models your agent called in the run
  • its candidates can’t reply more slowly, or cost more for each conversation, than the margin you allow
  • it stops at its budget in credits or its time limit, whichever comes first. Reaching the budget ends it as Stopped at budget, and reaching the time limit ends it as Failed, because it ran out of time

A candidate that changes something it may not, or goes over a model, reply time or cost limit, is rejected however well it does on the target scenarios. The model and cost checks need your agent’s model calls to be measured; Run Epoch covers when they aren’t.

How an epoch ends

An epoch that finds a candidate that passes every check ends Ready for review. Every other ending leaves your agent as it was: No safe improvement, Could not reproduce, Stopped at budget, Stopped, Too few failing scenarios, or Failed. A failed epoch can be retried once. Review an epoch says what each ending means and what to do next.

A result that is Ready for review is only as strong as the scenarios it was proven on, so the page says when the proof is weaker than usual:

  • Limited validation: one to three failing scenarios were held back, too few to tell whether the fix works on cases the epoch never saw
  • Proven on the targets only: nothing was held back, so the winner is proven only on the scenarios it was tuned on

For an agent on GitHub, a result with either label opens its pull request as a draft. When neither label shows, the winner held up on four or more scenarios the epoch never saw.

Why it matters

Fixing an agent by hand is a loop: read the failing calls, change the instructions, run again, and hope the change didn’t break something that worked. A change that fixes the calls you read can still fail the next caller who says it differently.

An epoch runs that loop for you, with the checks that are easy to skip by hand. It tries each change on the target scenarios, then checks the best one on scenarios it never saw and on every scenario that must keep passing, and it shows you the numbers before you decide. You review a change together with its evidence.

Keep exploring

Was this page helpful?

Questions & Discussion