Composite evals

What are they

A composite eval rolls several child templates into a single score, so one check can stand for several at once. Every template is either single or composite. A composite template holds an ordered list of child templates. When it runs, the parent scores each child on the same input, then combines the child scores into one parent score with an aggregation function.

Children share one output type, either pass/fail, score, or choices, so their scores are comparable before they are combined. A composite made of graded children produces a graded score; one made of pass/fail children produces a pass/fail-style score.

Reach for a composite when “good” is several checks at once, for example faithful and on-tone and complete. Instead of reading three separate results, you get one score that already reflects all three, and one threshold to gate on.

Aggregation functions

The parent combines child scores with one of five functions. Every child score is normalized to a 0 to 1 range first, so they can be compared on the same scale.

The examples below all score the same response with three graded children: Groundedness 0.9 with weight 2.0, Tone 0.7 and Completeness 0.8 with weight 1.0.

Weighted average

Each child score times its weight, divided by the total weight. This is the default. Use it when some children matter more than others.

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart TD
  C1["Groundedness 0.9<br/>weight 2.0"] --> AGG["Weighted average"]
  C2["Tone 0.7<br/>weight 1.0"] --> AGG
  C3["Completeness 0.8<br/>weight 1.0"] --> AGG
  AGG --> P["Composite score 0.825"]

On the example: (0.9 × 2.0 + 0.7 + 0.8) / 4.0 = 0.825, and one threshold on that single number gates all three checks at once.

Average

The plain mean of the child scores; weights are ignored. On the example: (0.9 + 0.7 + 0.8) / 3 = 0.8.

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart TD
  C1["Groundedness 0.9"] --> AGG["Average"]
  C2["Tone 0.7"] --> AGG
  C3["Completeness 0.8"] --> AGG
  AGG --> P["Composite score 0.8"]

Minimum

The lowest child score, a strict gate where every child has to do well. Use it when any one failure should sink the whole score. On the example: 0.7, the Tone score.

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart TD
  C1["Groundedness 0.9"] --> AGG["Minimum"]
  C2["Tone 0.7"] --> AGG
  C3["Completeness 0.8"] --> AGG
  AGG --> P["Composite score 0.7"]

Maximum

The highest child score, when clearing the bar on any one check is enough. On the example: 0.9, the Groundedness score.

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart TD
  C1["Groundedness 0.9"] --> AGG["Maximum"]
  C2["Tone 0.7"] --> AGG
  C3["Completeness 0.8"] --> AGG
  AGG --> P["Composite score 0.9"]

Pass rate

The fraction of children that individually meet their own pass threshold. Use it when you care how many checks passed rather than by how much. On the example, with each child’s threshold at 0.8: two of the three pass, so 0.67.

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart TD
  C1["Groundedness 0.9 passes"] --> AGG["Pass rate"]
  C2["Tone 0.7 fails"] --> AGG
  C3["Completeness 0.8 passes"] --> AGG
  AGG --> P["Composite score 0.67"]

Weights and pinned versions

Two settings on each child keep a composite stable as it grows.

• Weight sets how much a child counts in a weighted average, from 0.0 to 10.0, defaulting to 1.0. A child you care about more gets a higher weight, and a child at 0.0 drops out of the weighted score
• Pinned version locks a child to a specific template version, so the composite keeps scoring the same way even after that child template is edited later

Weights only change the weighted average function; the other four ignore them. Pinning matters most for a composite you rely on in CI, where a silent change to a child would quietly move the gate.

Keep exploring