Leaf Routing Guide — Which Work Goes to Which Model

Terms Used in This Document

① Perspective-specified review sonnet

When to use this type

• Pre-merge gate review (representative) — in the original project, every time a feature was finished it was standard practice to run a pre-merge adversarial review: "review this diff from privacy / performance / test-effectiveness perspectives." The measurement sample is all of this pattern.
• Specified-perspective review of a change involving security or privacy.
• Verification comparing a plan or document against a list of completion criteria.
• The checklist is written and assigned by the delegating main session just before merge:
  • The work of "pointing out where the risk is" is itself a discovery-type judgment (⑧), so leaving it to the higher model is consistent with this benchmark's other measurements.
  • But checklist-writing ability was not directly measured this time — we note that this is inference, not measurement.

The actual task used for measurement

• What the work was: While building a feature to publish notes to a public web URL, code came in that added "a badge showing the number of child items" to that page. The task is to review the correctness of this code before merge.
• Perspectives specified: privacy leakage, correctness/performance, test effectiveness — a checklist of three perspectives was assigned.
• An answer key exists: This code hides a merge-blocking defect — its identity is in "What the answer key is" below. The code that counts the badge number re-counts the child tree from scratch for every item, so the compute cost explodes as the square of the item count, and this page can be reached by anyone without login, making it a channel for a service-crashing attack.
• Grading criterion: Does it find this defect on its own?

What the answer key is

• What: The confirmed list of one merge-blocking defect — a service-crash channel caused by compute explosion.
• Model that produced it: A review by a standalone opus run at the time found it.
• Process it went through: That finding led to an actual repair commit, proving the defect was real — this is a confirmed fact, not someone's opinion.
• Caution when comparing: Only the presence/absence of the finding has a ground truth; report quality has no ground truth, so it was judged separately by a blind panel.

Test method

• The same review instruction was delegated to opus and sonnet twice each, 4 runs total.
• Each run is a single delegation — completed with one instruction, no mid-course intervention. However, the instruction prescribes 3 rounds, one per perspective, and requires "re-reading the diff/source each round, until findings per perspective are exhausted," so internally it is a multi-round review. All 4 runs stated in their reports that they completed all 3 rounds.
• Grading is two-layer. Layer 1 is whether the ground-truth defect was caught — a mechanical judgment comparing against the answer key. Layer 2 is the quality of the review report itself — a blind panel with model names hidden finds flaws in the report such as false positives, inaccurate descriptions, and exaggeration.
• All four runs didn't just assert their defect claims verbally but demonstrated them themselves: running the full test suite, a performance benchmark, and mutation testing — a technique of deliberately planting a defect in the code (e.g. removing the privacy filter) and checking whether an existing test flips to FAIL. If it doesn't, that test is decoration. With this they proved "the tests attached to the diff are not hollow."

^*panel findings = the count of flaws the blind adjudication found in the review report itself (claims of nonexistent defects, inaccurate file/figure citations, exaggeration, etc. — all minor-grade, 0 critical).

Insight

• Perspective-specified review is fine with sonnet. All 4 runs caught the answer, and the demonstration density was comparable too — when "where to look" is settled upstream via a checklist, what remains is the execution of digging into the specified spots to the end, and here the two models were indistinguishable.
• Two reviews beat one review by a good model. The cleanest and the messiest report both came from the same model (opus) — the same model's run-to-run variance is larger than the between-model gap.
• sonnet isn't cheaper in dollar terms either. Total of the 2 runs of this task: opus $5.23 vs sonnet $5.97 — the payoff of downgrading is not dollars but higher-model quota savings (§2).
• Checklist-writing stays with the higher model. Even if you push the review execution down, the upstream judgment of "pointing out where the risk is" is a discovery-type ability and shouldn't be economized on.

Caution

Open deep review without a checklist is treated as ⑨ Adversarial review gate — a completely different ability of "inventing where to dig on your own," and there the higher-model gap was real.

② Method-specified repair / refactor sonnet

When to use this type

• Execution delegation of a bug whose cause and repair method are already agreed on — diagnosis is done and only the fixing remains.
• A refactor that must preserve behavior — work like structural conversion, module separation, or performance improvement where "the visible result stays the same" is required.
• Enclosing an oracle is the constitutive condition of this type. The moment it becomes "just fix it however" without a judging means such as tests or output comparison, it escalates to ⑦ Pure diagnosis / repair, and the recommended model and safeguards change.

The actual task used for measurement

• What the work was: The server code of the publish feature that deploys notes to a public web URL had a structure where two functions that render the page recursively called each other. When it encounters a document nested more than 1,900 levels deep, the calls get too deep and the whole server dies — since it's a URL reachable without login, a single request can kill the server.
• Method specified: The repair method is already in the instruction — convert the recursion to an iterative-loop structure. But the page's render result must be byte-for-byte identical to before. Because this is not a behavior change but a structure-only refactor.
• Grading criterion: Did the crash disappear on deep documents, and is the output completely identical?

What the answer key is

• Why "the output of the pre-fix code" is the answer: It seems paradoxical at first — how can the output of buggy code be the answer? Because this work's requirement is "eliminate only the crash, and keep the rendered-page result exactly the same as before." The pre-fix code's only defect is that it dies on very deeply nested documents, and it renders ordinary documents correctly. So the output taken from ordinary documents becomes the answer as-is, and if the post-fix code produces a character-for-character identical result on the same document, it passes.
• What: The output of the 3 documents extracted that way — a document with shared items, an 1,800-level document, a formatted document. Plus one separate criterion: on the 3,000-level document that killed the pre-fix code, it must not die after the fix.
• Model that produced it: None — it is the output the code itself produced, so there is no room for model bias. The most objective answer key in this benchmark.
• Process it went through: The grader generated it directly in a separate environment, and first reproduced the pre-fix code actually dying on the 3,000-level document to confirm the answer key was valid.
• Note — who did the repair in the original session: The one that actually performed this repair was the fable run at the time (default dispatch with no model specified). That deliverable is not the grading standard but a reference comparison — the code output above serves as the ground truth instead.

Test method

• The same instruction was delegated to opus and sonnet twice each, 4 runs total.
• Grading was done with an independent harness that does not trust the candidate's claims. In a separate environment the grader built the 3 answer outputs from the pre-fix code — since it's the code itself, not a particular model's output, there's no source bias — a document with shared items, an 1,800-level document, a formatted document — and compared the candidate code's output byte by byte. It also first reproduced the pre-fix code actually crashing on the 3,000-level document, confirming the validity of the oracle itself.
• Added to this were re-running the 132 existing tests and a quality adjudication by a blind panel with model names hidden.

^*The content of the panel findings is interesting — all 4 of opus's flaws were not in the code but mismatches between the report and the deliverable: a verification tool it claimed to use was absent from the commit, it exaggerated a gap in existing test coverage, or a newly built oracle verified in only one direction. The code itself was clean in all 4 runs.

Insight

• A repair where method and oracle are provided together is fine with sonnet. All 4 runs score perfect on the oracle, and 0 panel flaws is actually on the sonnet side.
• Even a hard constraint like "byte-for-byte identical" becomes an execution problem when the judging means comes with it. It's not that a hard constraint requires a higher model, but whether there's an oracle to verify the constraint that matters.
• Comparing report claims against the deliverable is an effective gate for this type too. A case actually surfaced where the claim "verified with 500 random cases" was not in the commit — even when the code is perfect, the report can inflate.
• Dollar cost is comparable — 2-run total opus $17.53 vs sonnet $16.15. The payoff of downgrading is quota.

Caution

The oracle must be derived from a merged / confirmed standard. If the party doing the repair uses tests it wrote itself as the oracle, it shares its own blind spot — this trap was confirmed in the flesh in ⑥ Emergent implementation.

③ Scoped plan sonnet + review intensity is set by blast radius — see the decision procedure in the body

When to use this type

• A scoped plan for work like a structural change or migration where the steps and risks must be organized before implementation.
• Code modification is forbidden, and the deliverable is a single document containing the researched rationale, step breakdown, completion criteria, and a list of open decisions.
• Why it's named "draft": This leaf's deliverable is always a pre-review draft rather than a finalized plan. If the blast radius is large it is finalized via independent review per the standard below; if it's local, the draft may be adopted as-is — finality is decided by the gate, not the name.

Decision procedure — who writes and who verifies this plan

• Plan writing — who writes it
  • Scoped plan (a known goal) — organizes steps, risks, and completion criteria based on the current state → sonnet — the same even if the blast radius is large. The basis is measurement: the measured task below was exactly a large-blast-radius plan, and sonnet did not fall behind.
  • Exploratory plan (you must first find the goal) — where the work is searching broadly for what to do or devising a new design → opus or higher. This is outside this card's measurement, a conservative choice inferred from the evidence of ⑥ Emergent implementation.
• Review — who verifies it
  • Blast radius local — at the level of organizing the reorder of a single screen → review can be skipped.
  • Blast radius large → opus independent review. "Large blast radius" holds if any one of the following:
    • does a lot of downstream work stand on this plan
    • does it move something hard to reverse — migration, etc.
    • does an interpretation difference decide user-visible behavior
    The reviewer basis is one measurement: the adjudicator that caught this measurement's critical defect was opus. If you specify the perspective to look at, there's room to push down to sonnet, per the result of ①.
  • Top-level risk — a decision that decides product direction, an irreversible data move → fable open deep review. Unmeasured — a conservative choice inferred from the evidence of ⑨.

What size changes is not "who writes the plan" but "how hard the review is" — this is the answer this card's measurement gave.

The actual task used for measurement

• What the work was: In an early validation phase, the same kind of data lived split across several different stores — the list and tree on a temporary one, the body content on the real one — so different screens read different sources. A contradiction actually arose where an item visible in one view didn't exist in another. The task is to write a document with a migration plan that unifies this split into a single data model.
• This plan meets the "large blast radius plan" criterion above — a lot of downstream work stands on it, and a migration is involved, so it's hard to reverse if it goes wrong.
• The grading method is distinctive — hindsight grading: This plan actually went all the way to implementation in the original project — the implementation run was fable at the time — and 4 traps stepped into in that process remain on record. Those traps are back-computed to check "did the plan warn about them in advance" — a way of grading plan quality not by taste but by facts confirmed after the fact.
• The 4 traps:
  • The explosion problem of counting the same item redundantly in deep-tree lookups
  • Handling the edge case of page navigation
  • Is the migration safe to run twice, and can it be rolled back if it fails
  • Did the plan include not just the data structure but the screen the user will operate on

What the answer key is

• What: The list of 4 traps the plan should have foreseen.
• Model that produced it: It's not a particular model's output — it was back-computed from the event record of what the later actual implementation (the fable run at the time) and the independent review actually stepped into or caught.
• Process it went through: All confirmed after-the-fact facts from commit and review records.
• Caution when comparing: The adopted plan document from the original session is an improved version reflecting 26 rounds of independent review, so it was not used for comparison — directly comparing a reviewed version against a single run would be unfair.

Test method

• The same instruction was delegated to opus and sonnet twice each, 4 runs total.
• Grading judged whether each of the 4 traps was foreseen, together with document-citation evidence, and separately a blind panel found defects in the plan itself — unrealizable steps, self-contradiction, wrong attribution.
• Direct comparison against the original session's adopted version was not done. The adopted version is an improved version that went through 26 rounds of independent review, so comparing it against a single run with no review would be unfair.

Insight

• A scoped plan is fine with sonnet — more precisely, "sonnet does not fall behind." The side that produced a critical defect was actually opus. That said, since these are results of 2 runs each, we can't go so far as "sonnet is safer."
• The best single output was also opus, and the only critical defect was also opus. The pattern that the same model's run-to-run variance exceeds the between-model gap repeated in planning too.
• The safeguard for a plan is not a model upgrade but independent review. The fact that the side producing a critical defect was the higher model backs this up — who does which review is in the decision table above.

Caution

If the plan document is a strategy paper for "what to research and how to find it," it is not this type — the very construction of writing the strategy in advance and injecting it into the executor was rejected in ⑩.

④ Coverage-type research sonnet

When to use this type

• Exhaustive collection and comparison against the current state of competing products, standards, or docs — research where "gather without omission" is the body of the value.
• The typical case is work that compares the collected results against our code/docs to build a classification table.

The actual task used for measurement

• What the work was: The task is to exhaustively collect the keyboard shortcuts of a competing note product from the official docs (Korean and English editions) and community write-ups, read our app's code directly, and classify each shortcut as "already supported / conflicts with another action / not implemented" to build a compatibility matrix.
• Answer key: The confirmed matrix, 49 items — its identity and bias are in "What the answer key is" below. "The best being 47/49, short by 2" should be read not as a performance shortfall but as the gap between a single run and a review-reflected version.
• The classifications' correctness was also separately verified: The grading panel picked at least 8 of each run's classifications and opened the code directly to re-verify.

What the answer key is

• What: The confirmed compatibility matrix, 49 items.
• Model that produced it: A run that started with opus and continued into fable mid-way wrote the draft.
• Process it went through: A confirmed version reflecting 11 findings from 2 rounds of independent review — i.e., a reviewed, improved version.
• Caution when comparing: A coverage comparison against a single run tilts in the answer key's favor. Moreover, in this grading 3 errors in the answer key itself surfaced, and candidates also found valid items not in the answer key — the answer key is not an absolute standard but a comparison standard.

Test method

• The same instruction was delegated to opus and sonnet twice each.
• The panel was instructed to "doubt the answer key too" — and indeed 3 errors in the answer key itself were surfaced in this process (missing official shortcuts, etc.). The original session's deliverable is not flawless either.

Insight

• Coverage-type research is fine with sonnet. The coverage record-holder is sonnet, and the classification errors on both sides were all trivial local misclassifications — accuracy is slightly opus's edge (24 of 25 vs 24 of 29), but critical defects are 0 on both sides.
• Once the collection scope is defined, it becomes a problem of diligent traversal. Discovery-type research where "where to look" is open produced completely different results — ⑧.
• Make it doubt the answer key too. 3 errors in the grading standard itself were surfaced by the candidate runs — penalizing anything that differs from the standard unconditionally punishes the better answer.

⑤ Design mockup / HTML production sonnet + specify automated capture-and-review of the rendered screen

When to use this type

• A round where, before implementing a new screen or component, you make a mockup HTML comparing several variants side by side for a human to choose.
• The deliverable is a self-contained page that includes the selection-collection controls — radio buttons, a copy button.

The actual tasks used for measurement (2)

• Task A: The editor's command menu got the feedback "the text is too big." The work is to design 4–6 variants along the axes of font size and density and make a comparison page.
• Task B: The first mockup round for a trash-bin screen and a top menu panel that had never been built — 3 structural variants each, including per-state screens.
• What "the original session's adopted version" is: The very comparison page that was produced when this mockup round actually ran in the original project, from which the user viewed and chose a variant — its identity is in "What the comparison standard is" below. The originals live in the project repository and can be opened directly for comparison.
• Comparison design: For each task, opus ×2 and sonnet ×2 plus the adopted version — 5 items — were shuffled anonymously, and with which one was the adopted version also hidden, a human picked the best and the reject.

What the comparison standard is — visual preference has no ground truth, so the adopted version is used instead of an "answer key"

• What: The 2 comparison pages the user viewed and chose a variant from in the original round.
• Model that produced it: Both were outputs of an opus run at the time.
• Process it went through: Adopted as-is with no edits after production — per repository history, 0 changes since the introduction commit, hash-matched with the blind copy.
• Caution when comparing: It's a version that once passed human selection, but in this blind round it went 1 win / 1 reject — an adoption record is not a quality guarantee.

Test method

• Human blind selection is the primary verdict — because visual quality has no automated grading. The grader's connotation criteria: reject = a defect at the not-adoptable level, "so-so" = the level of not catching the eye.
• Separately, the panel adjudicated only verifiable items: the required variant count, per-state screens, accuracy of numeric annotations, dark mode, self-containedness.

Insight

• Mockups are not a one-shot job. Of the 8 new items, only 2–3 met the human's bar — more than half were mediocre or below. The correct operation of this type is a structure that generates multiple candidates for a human to pick, and the mockup round is that structure by design.
• There is no model direction. The one "best" was sonnet, the one "good" was opus, and both sides appear among the mediocre and rejected — nothing was observed to be lost by downgrading to sonnet, and run unit-cost is not the dominant factor either. So increasing the number of candidates on sonnet is rational.
• The adopted version doesn't always win either. The adopted version — an opus output at the time — was the best on one task but got a not-adoptable verdict on another, and even a content defect nobody caught in practice surfaced in the blind re-check. Anonymous comparison verifies not just the deliverables but past choices too.
• Required condition: specify in the instruction "automatically capture the rendered screen and review it yourself." This is a verification that opens the page in a headless browser and takes a screenshot; the class of bug that declares a style but never wires it can't be caught from source alone and requires looking at the rendered screen — indeed the rejected run stepped into this trap.

⑥ Emergent implementation opus or fable + independent adversarial review before merge is mandatory

When to use this type

• An implementation with algorithm/protocol-level design judgment still remaining — work where there's a reference direction but no correct design.
• Areas where the cost of a mistake is high, like synchronization, concurrency, and data integrity, are typical.

The actual task used for measurement

• What the work was: There was a defect where, when two people edit the same line simultaneously, one side's input gets erased. You need to build an algorithm that transforms the position of a remote change and merges it onto my not-yet-confirmed edit, and the instruction gave only the cause family and direction, saying "verify the current code directly, then judge the design yourself." There's the overlapping constraint that it must coexist without conflict with the resend / reconnect / server-sync rules.
• How it went in the original session: The original implementation of this task was a fable run. Even that fable built a critical defect into its first implementation, and it was merged only after a pre-merge adversarial review caught it and it went through repair — i.e., "you can't trust a single implementation" is this card's conclusion, common to all three models including fable.

What the answer key is

• What: There is no answer list in advance — it's an answer key whose confirmed-defect count is created after the fact.
• Model that produced it: A cross-adjudication team (opus and fable each) found the defects, and major claims were confirmed only if they passed a separate disconfirming verification.
• Caution when comparing: The original session implementation (fable at the time, 1 critical defect on the first pass then repaired via review) is a reference comparison, not the grading standard.

Test method

• Delegated to fable, opus, and sonnet twice each, 6 runs total. Automated grading was a build plus a full re-run of the existing tests — all passed.
• The quality layer was cross-adjudication: an opus panel and a fable panel each blind-adjudicated the same 6 deliverables, and major claims were admitted only if they passed disconfirming verification. This cross design produced an unexpected harvest — a gap between the panels.

Asymmetry-of-verification-depth footnote: An adjudicator can build verification tools itself and has discretion over how far to dig, so not every candidate received 70,000-scale random verification — the three candidates with a defect had that defect confirmed by a replayer (positive evidence), and the two fable candidates are clean even under aggressive replay (negative evidence), but opus run 1's cleanness has no record of being attacked at the same intensity. Keep this in mind when reading the "fable 2/2 > opus 1/2" ladder.

When and how good was fable — the measurement in the verification phase

• In the implementation phase, fable is 2/2 clean as in the table above. Separately, the place where fable's ability showed most dramatically is the verification phase.
• The behavior of the two panels that adjudicated the original experiment's 4 candidates diverged. The opus panel: at the level of reading the code carefully and checking the existing tests — found 0 confirmed defects.
• The fable panel: didn't stop at reading the code carefully but invented and ran on the spot a verification harness that randomly shuffles and replays the edit orders of the two clients. It ran 30,000 orders, reproduced 291 divergences, and confirmed the cause by seeing them drop to 0 after fixing one suspect line. By this method it found all 3 confirmed defects.
• That is, fable's edge was not "reads the code better" but "invents on its own what verification means it needs" — the same mechanism as ⑧ Discovery, and the direct basis for entrusting the merge gate of emergent implementation to fable (⑨).

Insight

• 3 of 6 runs built in a merge-blocking defect while passing all their own tests. Self-written tests share one's own blind spot — hard evidence that you must not use "passes its own tests" as the completion criterion for emergent implementation.
• The model ladder was measured: fable clean 2/2 > opus 1/2 > sonnet 0/2. The only model that passed this benchmark's decision rule (0 critical) is fable. But since the original session's fable first pass had a critical defect (a 6-run sample), fable is not always clean either — keep the gate mandatory regardless of model.
• There are two options: for quality-first, fable + gate ($43–51 per run — about 3× opus); to save quota, opus + gate (on the premise that the gate catches criticals). Downgrading to sonnet is still not recommended, being 0/2 clean.
• It's the most expensive type — $12–22 per run. And yet without the gate you're buying a defect with that money.

⑦ Pure diagnosis / repair opus, fable if important + same gate mandatory

When to use this type

• Whole-cloth delegation of diagnosis ＋ repair for a bug that reproduces but whose cause is unknown.
• Boundary with ②: if the repair method is in the instruction it's ②, if not it's here.

The actual task used for measurement

• What the work was: There was a bug where, when you create a new page, it should navigate to that page but instead bounces to the home screen. The instruction gave only the hypothesis "a race entangled with screen-refresh timing is suspected," and nailed down even that hypothesis with "this is an estimate at review time, so re-verify it with code."
• A layered answer key: In the original session this bug was fixed in two passes. The first repair — a fable run — merely blocked the one path where the problem occurred, and only after an independent review — an opus run — pointed out "it recurs via another path" did fable's re-repair reach a containment that doesn't recur via any path. So the grading scale is three-layer:
  • Path repair — blocked only the one path where the problem occurred. The level of the original session's first repair.
  • A third valid source repair — a different method from the answer, but blocks all currently reachable paths.
  • Source containment — a structural repair that doesn't recur via any path.

What the answer key is

• What: The 3-layer repair scale — path repair / third valid repair / source containment.
• Model that produced it: Not a single output of a particular model but back-computed from the original session's event history — the first repair was fable, the "it recurs via another path" pointer was an opus review, and the source-containment re-repair was fable again.
• Caution when comparing: Not a mechanical diff comparison but the panel judging by code-tracing "does this repair actually block the other paths too" — a third valid repair different from the answer is also admitted.

Test method

• Delegated to fable, opus, and sonnet twice each, 6 runs total. An opus panel and a fable panel judged, by code-tracing, the diagnosis level of each deliverable — down to surface race or down to the root of the recurrence structure — and the repair layer. Not a mechanical diff comparison but confirming per-path "does this repair actually block the other paths too."

An interesting fact — both of the fresh single-run fable runs surpassed the first repair (path-repair level) that fable itself produced in the original session. The original session's first pass being shallow was more likely that day's variance than a model limit.

Insight

• Looking only at the 4 opus·sonnet runs, the best is also sonnet and the only critical defect is also sonnet. The best run went deeper than the original session's first repair, and the worst run stepped into an approach the original session had rejected — variance dominates the model gap in the diagnosis type too.
• The recommendation is opus, fable if important. sonnet did not pass this benchmark's decision rule — no downgrade if even one critical defect appears. opus was a clean middle on both runs, and fable's 2/2 was root diagnosis ＋ at least a third valid repair — superior in both layer and cleanness.
• All 4 runs re-verified the hypothesis with measurement rather than blindly trusting it — down to instrumenting directly to observe the 3-stage screen transition. Diligence of diagnosis was high regardless of model.
• For an important diagnosis, "the same model twice + gate selection" can be measurably more rational than "a higher model once" — because the variance is large.

⑧ Discovery-type research opus or higher

When to use this type

• Research that digs out features, constraints, and workarounds not in the official docs from the community and experiments.
• It's often mixed inside the same task as ④ Coverage type — when writing the instruction, think of the two axes separately. The recommended model splits depending on which axis is the body.

The actual task used for measurement

• Not a separate run — it's the same 4 shortcut-research runs of ④ measured with a different yardstick. That instruction explicitly included the discovery axis: "find hidden shortcuts that the community reports but the official docs don't have, and mark them separately."

What the answer key is

• What: There's no answer list in advance — the count of the hidden shortcuts each run found whose source is actually confirmed becomes the score after the fact.
• Model that produced it: The grading standard itself is not a particular model's output (a post-hoc-verification type that verifies the source in the flesh per finding, with no answer list in advance). But the run that actually performed this discovery in the original session is the same run as ④ — it started with opus at the time and continued into fable, and the 3 hidden shortcuts of ④'s confirmed matrix used for cross-checking are also that deliverable (reflecting 2 rounds of independent review).
• Caution when comparing: For reference, ④'s confirmed matrix also carries 3 hidden shortcuts, used for cross-checking. And fable was not measured for this type — the fable supplementary runs were only in ⑥ (implementation target A) · ⑦ (diagnosis target A), and this research task compared only opus · sonnet.

Test method

• A panel with model names hidden verified each run's discovery claims against the source in the flesh — official announcements, community docs — and counted only the valid ones.

Cost is the same run as ④, so see ④'s table. This is the clearest and most consistent model gap in the entire benchmark, opening up in both runs.

Insight

• The ability to dig where specified is tied, but the ability to broaden where to dig on your own was consistently the higher model's edge.
• This gap cannot be corrected by the gate. Review catches what's wrong, but review can't create a find that wasn't made — so for work where discovery is the body, don't economize on the model.

Caution

Since there's only 1 measurement task, read it as a direction signal — but ⑨'s panel gap is the same mechanism, so they reinforce each other.

⑨ Adversarial review gate keep fable

When to use this type

• The last checkpoint before merge for emergent implementation and diagnostic repair. Since every "sonnet possible" verdict in this document presumes the gate's existence, the quality of the gate itself is the floor of the whole system.
• Boundary with ①: if a checklist is given it's ① and sonnet is fine. This is a review where you must decide where to dig on your own.

The actual task used for measurement

• Not a designed replay but a natural experiment created by the benchmark's own adjudication process — the 4 implementation deliverables of ⑥ and the 4 diagnosis deliverables of ⑦ were blind-adjudicated independently by an opus panel and a fable panel. Since two panels saw the same targets, it becomes a direct comparison of adjudication ability.

What the answer key is

• What: No answer in advance — the set of confirmed defects that combine the two panels' findings and pass disconfirming verification becomes the post-hoc standard, and how many of them each panel found is the score.
• Model that produced it: None — confirmed after the fact by cross-finding + disconfirming verification.
• Caution when comparing: Since it's a structure where the panels' own findings become the standard, a defect that both panels missed is outside the measurement — it measures only the lower bound of the gap.

Test method

• The two panels each adjudicated 4 candidates each with model names hidden, and merge-blocking claims were admitted only if they passed a separate disconfirming verification.

Insight

• The gap in open deep review is the gap in discovery — the same mechanism as ⑧. The panels diverged on the ability to decide, on their own, what to suspect and what verification tools to build and run.
• Economize on the gate and the whole thing collapses. Since the safety basis of every sonnet-downgrade row is "the gate catches it," the moment you lower the gate model, that premise disappears.

Caution

It's a conservative status-quo where the whole basis is 1 implementation-adjudication task + the direction agreement of the diagnosis adjudication — the precise statement is not "opus must not be used" but "there is no basis to lower it." Checklist-type review (①) is evidence in the opposite direction.

⑩ Strategy/execution two-stage split forbidden

When this pattern is tempting

• The intuition "let the expensive model use only its head, and let the cheap model use its hands" — it comes from the rational motive of saving quota. It looks plausible but was rejected in measurement.

The actual task used for measurement

• Using ④'s shortcut-research task, three configurations were compared twice each: opus alone, sonnet alone, and the split configuration — opus writes a strategy document and sonnet takes over and executes.

What the answer key is

• Same as ④ — coverage against the confirmed matrix of 49 items (review-reflected version), code re-verification of classifications, and the hidden-find count of ⑧.

Test method

• The 4 alone-configuration runs are the same runs as ④·⑧, and only the 2 split-configuration runs were newly run — comparing by summing the cost of the strategy stage and the execution stage.

Why it fails

• The strategy document becomes the ceiling of exploration. Both runs had 0 hidden finds — the executor dug only where the strategy pointed, and unexpected finds outside the strategy vanished.
• A hole in the strategy becomes a hole in the result as-is. Tracing the cause of the worst run, one core file was missing from the file list the strategy said to scan, and the executor never opened that file — 4–5 misclassifications cascaded from here. A file a single run would have found and gone into on its own.
• It's also more expensive. Calling two models makes the summed cost higher than alone.

Insight

• Core mechanism: research scope must be dynamically updated as the research results come in, and the split configuration blocks that — so performance drops. Both failures above are this one mechanism: a find during execution should create "next I should look over there too" (hidden finds), but when the scope is fixed in a strategy document that update doesn't happen (0 finds), and a place missing from the strategy is never visited (core file unvisited → misclassification cascade).
• The alternative is a single well-written instruction. A single delegation that carries the background and completion criteria while leaving "the details are your judgment after verifying the current state directly" was better than the 2-stage pipeline in both performance and cost — every measurement in this document was done on that form.

Aside — what model for the main session? keep fable (unmeasured — inferred from adjacent measurements)

• Every measurement in this document is at the leaf layer — there is no experiment directly comparing the main session's duties (decomposition, instruction-writing, orchestration, gate operation) across models. What follows is inference from adjacent measurements.
• The main session's duties are precisely the set of abilities where a higher-model gap was measured: open deep adjudication (⑨ — fable 3 vs opus 0), discovery (⑧), dynamic updating of research scope (⑩'s core mechanism — the very judgment the main session performs constantly), checklist writing (the upstream of ①) and blast-radius judgment (③).
• Consistent with the quota structure too: The body of quota savings is pushing leaves out of fable (§2), and the use for the Fable quota saved that way is the main session. If you push down the leaves and lower the main too, there's nowhere to spend the saved quota.
• Warning signal: The closest proxy measurement for "opus main session" is ⑨'s opus panel, which caught 0 of 4 confirmed defects. If Fable's quota is short, route leaves more thoroughly rather than lowering the main.