Leaf routing guide v2 — which model gets which job

Emergent implementation is the one type where opus-5 didn't beat fable. Confirmed defects: fable 1·1 (2 total) vs opus-5 0·3 (3 total). But those 3 are concentrated in one run — the first was flawless and only the second carried them.

The practical rule: fable if it has to land in one shot. opus-5 twice if a gate is in your loop anyway — and since this type makes the gate mandatory, the second is the default.

The first four (9·8·4·5) have something in common: they all reward sweeping wide and landing precisely. The design judge's own words point at that axis — “it even accounted for the dark-surface contrast, which shows how wide a field it was considering.” That is the ability to take care of what nobody asked for.

The last two are different in kind. Type 7 wins by being tied at half the price, and 1·2·3 are cases where all three models are identical, so there's no reason to pay up.

Same two models, and the multiplier varies by more than 2× depending on the task type (opus-5 ÷ sonnet, mean of 2 runs):

Why it varies this much is not explained by this measurement. We hypothesised "the better model just stays on it longer" and checked it against the turn-count ratio — it broke on 4 of the 8 types: research spends 1.77× the turns for 1.17× the cost, review 1.16× the turns for 1.95× the cost. Per-turn price, cache hit rate and output length are presumably tangled differently per type, but we did not verify that.

What fable's premium actually is — against both opus generations

For the three types fable was run on: how many times more expensive fable is than opus-4-8, and than opus-5. Read this separately from the table above (which is opus5 ÷ sonnet).

Identifying question

Does the review instruction contain a checklist of what to look at and through which lens? If nothing is specified and the review is open-ended, that's type 9.

When you use this type

• Pre-merge gate review (the typical case) — "review this change for privacy, performance and test effectiveness"
• Directed review of a change touching security or privacy
• Checking a plan or document against a list of done-criteria

The checklist is written and supplied by the delegating main session — deciding where the risk is is itself type 8 (discovery), so leaving that to a stronger model is consistent with the rest of the measurement.

The actual task measured

What it was

While building a feature that publishes a document to a public web URL, a change landed that added a "badge showing the number of child items" to that page. The task is to review it before merge.

Lenses supplied

Privacy leakage · correctness/performance · test effectiveness — the instruction specified all three.

The defect hiding in it

The badge-counting code re-walks each item's whole subtree from scratch — the work grows with the square of the item count. The instruction only says "look at performance"; it doesn't say where the problem is.

How it was graded

Does it find that defect on its own.

What the answer key is

• What — a confirmed list of one merge-blocking defect (the quadratic blow-up)
• Which model made it — found by a review run on opus at the time
• How it got there — that finding led to a real repair commit, which proves the defect was real. It's a settled fact, not someone's opinion

2nd pass results

Model	confirmed	minor
opus5	0 · 0	4 · 3
opus4.8	0 · 0	1 · 2
sonnet	0 · 0	1 · 3

(the two numbers are run 1 · run 2; lower is better)

Verdict

Keep sonnet — all three models hit zero critical defects, so nothing separated them. On minors opus-5 is actually the worst (4·3). If a better model gives the same result, the cheaper one is the rational pick.

Cost (2 runs)

Model	run 1	run 2	total
opus5	$5.20	$6.45	$11.64
sonnet	$3.37	$2.60	$5.97
opus4.8	$2.39	$2.84	$5.22

opus-5 costs about 2× for the same result.

Identifying question

Is "what to change and how" already in the instruction, together with an oracle (tests, output comparison) that mechanically decides whether it worked? If the cause isn't established yet, that's type 7.

When you use this type

• Behaviour-preserving refactors — "the output bytes must be identical" is the done-criterion
• Fixing a bug whose cause is already agreed
• A migration to a specified shape

The actual task measured

What it was

In the rendering code two functions call each other back and forth (mutual recursion). When the input nests deeply, the call stack overflows and the program dies. Convert it to an iterative form.

What the instruction supplied

The method — "convert the mutual recursion to an explicit stack-based loop" — and the machine criterion — "the rendered output bytes must be byte-identical to before." So what, how, and how to check were all given.

How it was graded

Automatic oracle — byte-for-byte comparison against the reference output + no crash on input far deeper than the old limit + no regression in the existing suite.

What the answer key is

• What — byte copies of 3 reference HTML outputs produced by the pre-repair code. It's for machine comparison, so no human interpretation enters
• Which model made it — no model made it; it's the output of the pre-repair code. This type's answer key has no model attribution

2nd pass results

Automatic oracle: opus5, opus4.8 and sonnet all scored full marks — 3 byte-identical outputs + no crash + suite passing.

Model	confirmed	minor
opus5	0 · 0	2 · 1
opus4.8	0 · 0	2 · 2
sonnet	0 · 0	4 · 3

Critical defects are zero for all three — with the method and the criterion both handed over, the ceiling is low. The only spread is in minors, and sonnet has the most (sonnet 7 · opus4.8 4 · opus5 3). But minors don't block a merge, so that difference is not grounds for moving up a tier.

Verdict

Keep sonnet — neither the oracle nor the review separated them. opus-5 has the fewest minors (2·1 vs 4·3), but a 2-item difference across 2 runs is indistinguishable from run variance.

This type is work where the answer's location is already fixed, so the ceiling is low. A stronger model has nowhere to spend its headroom.

Cost (2 runs)

Model	run 1	run 2	total
opus5	$13.74	$27.95	$41.69
sonnet	$9.45	$6.70	$16.15
opus4.8	$6.67	$10.86	$17.53

opus-5 costs 2.5× for the same result. ⚠ Its second run spiked to 120 turns (the first took 24) — 5× the turns and 2× the cost on identical work.

Identifying question

Is what to build already decided, leaving only how to build it to design? And is the output a document rather than code? — that's what "scoped" points at. Goal fixed, route open.
Work where you must find the goal itself is an "exploratory plan" — for example "go find out what we should build," or "find what this product is missing." There is no exploratory plan in this measurement. For how models split on open-ended work, see type 8.

When you use this type

• Restructure or migration plans — "we've decided to change it this way; design the steps"
• Writing up the trade-offs of a technical choice

The output is always a "pre-review draft" — the delegator decides how much review to apply based on blast radius.

The actual task measured

What it was

The same kind of data lives split across three different stores, so different screens read different sources. Write a plan document that unifies them into one source. No code changes allowed — read-only investigation plus one document.

How it was graded

Defects in the plan itself (infeasible steps, wrong ordering, self-contradiction, no rollback path) plus a hindsight check: did it foresee the 4 pitfalls the real implementation later hit?

What the answer key is

• What — the 4 pitfalls the real implementation later exposed (recursive-query duplication blow-up · no-op move handling · migration re-run safety · a missing user-facing screen)
• How it got there — the list was only settled after the implementation finished. Nobody knew it when the plan was written

2nd pass results

Model	confirmed	minor
opus5	1 · 0	2 · 3
opus4.8	0 · 1	2 · 2
sonnet	1 · 0	2 · 2

Each of the three produced a critical defect in exactly one of its two runs.

Verdict

Keep sonnet — the defect distribution is identical (one each). That reads as run variance, not as one model being better.

What this means in practice: raising the model does not reduce critical defects in a plan document. If a plan has a large blast radius, add an independent review rather than a bigger model — that's the only response this measurement supports.

Cost (2 runs)

Model	run 1	run 2	total
opus5	$6.58	$5.43	$12.00
opus4.8	$3.09	$3.57	$6.66
sonnet	$2.72	$4.80	$7.52

opus-5 costs 1.60× sonnet and 1.80× opus-4-8, and all three land on one confirmed defect — in this type a better model only raises the price. Run-to-run spread is similar across all three, too.

Identifying question

Is the problem clear but the solution open? — the instruction points at what needs fixing, but there is more than one right route and the candidate has to choose which one to take. If the instruction also spells out how to solve it, that's a closed problem (type 2); if you must first establish what the problem even is, that's type 7.

When you use this type

• Areas where several subsystems' rules must hold simultaneously — sync, concurrency, data integrity
• Work where you're given "make this problem go away" and nothing else; where and how is the candidate's call

The artifact must pass a type 9 gate before merge — regardless of model.

The actual task measured

What it was

In a feature where several people edit the same document at once, characters disappear when two people edit the same line simultaneously. When the other side's edit arrives mid-keystroke, the position arithmetic goes out of alignment.

Why it's emergent

The instruction goes only as far as "close this loss window." Where to fix it is the candidate's call, there are several options, and fixing one place easily breaks another — the transmit, settle and recover paths are entangled.

How it was graded

Automatic oracle (build, typecheck, no regression in existing tests) + defect hunting by 3 reviewers.

What the answer key is — ⚠ read this

The model that actually ran the original session for this task was fable (verified from the conversation log). So "it didn't beat the answer key" is a fable-versus-fable comparison here. Before quoting any "versus the anchor" claim, check whose output that task's answer key is.

⚠ this table mixes campaigns

Only opus-5's two runs are genuinely fresh here. The fable, opus4.8 and sonnet rows are 1st-pass (C1) output reused as-is and simply re-graded by the new reviewer panel (which now includes opus-5) — they were not re-run under 2nd-pass conditions. So the opus-5-versus-the-other-three comparison carries cross-campaign drift (CLI version, environment) — this is "re-graded under corrected conditions," not "re-run under identical conditions."

2nd pass — only defects both verifiers confirmed (lower is better)

Model	confirmed	Note
fable	1 · 1	fewest · no spread
opus5	0 · 3	extreme spread
opus4.8	1 · 3
sonnet	4 · 3	most

All four models scored full marks on the automatic oracle — that layer discriminates nothing here.

opus-5's run-to-run spread

It split 0 vs 3. Same model, same instruction: one run flawless, the other carrying 3 confirmed defects. Don't settle a model on two runs.
What this means in practice: for an emergent implementation that matters, running opus-5 twice and letting the gate pick beats raising the model — two runs total $37.64, less than one fable run ($46.94), and you get to take the flawless one.

The gate stays mandatory

No model was flawless. An independent adversarial review before merge is required regardless of model — every confirmed defect in this measurement was caught by the gate and by nothing else.

Cost (2 runs)

Model	run 1	run 2	total
fable	$42.65	$51.22	$93.87
opus5	$20.20	$17.44	$37.64
opus4.8	$12.16	$22.34	$34.51
sonnet	$17.50	$13.20	$30.70

fable costs 2.5× opus-5 — fewest defects, but the most expensive cell in the benchmark.

Identifying question

Is the bug's cause not yet established (hypothesis-level), with diagnosis through repair delegated in one piece? If the repair method is already in the instruction, that's type 2.

How it differs from type 6

Type 6 is work where you know what to build but how is open; type 7 is work where you have to find out what the problem is first. Six is design; seven is closer to detective work.

The actual task measured

What it was

A bug where creating a new item bounces you straight back to the first screen. You press create, the item should appear and you should navigate to it — instead it flashes and bounces back.

Why it's diagnostic

The instruction gives only the symptom. Several layers are entangled, so the candidate has to establish which layer the cause is in before anything else.

How it was graded

No automatic oracle — defect hunting by 3 reviewers is the only verdict. The crux is "did it just patch the surface symptom, or did it reach the causal layer."

What the answer key is — ⚠ same caveat as type 6

This task's original session also ran on fable. Fable versus fable here too.

⚠ this table mixes campaigns too (same as type 6)

Only opus-5 is a fresh 2nd-pass run; fable, opus4.8 and sonnet are 1st-pass output re-graded by the new reviewer panel (not re-run). Cross-campaign CLI/environment drift is baked into this comparison.

2nd pass results (lower is better)

Model	confirmed
fable	1 · 0
opus5	0 · 1
opus4.8	2 · 1
sonnet	2 · 1

fable and opus-5 tie for fewest (1 each); opus4.8 and sonnet tie at 3.

Verdict

Switch to opus-5 — opus-4-8, the first pass's recommendation, split with 3 confirmed defects, while opus-5 ties fable at 1 for half the price ($10.82 vs $21.15). Keep fable as the escalation card for harder problems.

The gate stays mandatory here too — none of the four models was flawless.

Cost (2 runs)

Model	run 1	run 2	total
fable	$6.60	$14.55	$21.15
opus4.8	$6.57	$11.75	$18.33
opus5	$5.88	$4.94	$10.82
sonnet	$2.53	$5.70	$8.22

opus-5 is half of fable and ties it — on value per dollar opus-5 is ahead.

What the pattern is

Having a stronger model plan the research strategy only, then having a cheaper model execute that plan. The idea is "the expensive model does the thinking and the cheap one does the legwork" — and the measurement keeps saying the opposite.

How it was measured

On the same research task as type 8, run twice in a two-stage setup where opus writes the strategy and sonnet executes it. Every other model did the whole thing end to end.

2nd pass results

	hidden finds	coverage	grounding
split opus4.8 → sonnet	1 · 2	83.0%	57.1%
best single model	11 · 12	93.6%	90.6%

Last on every metric. Discovery is about 1/8 of the best, and grounding accuracy at 57% is far off every other model (75–91%).

Why this happens (interpretation, not measurement)

Discovery is work whose direction changes while you're doing it. Freeze the strategy and hand it over, and the executor sweeps only inside that list — nobody ever looks outside it.

The moment you split it, discovery degrades into coverage.

The cost is bad too

The split is the most expensive — $7.90 per run, 47% more than the best single model ($5.36) (§4). 1/8 the quality at 1.5× the price.