Tiered routing A/B — safe to enable `FLYER_TAGGER_TIERED_ROUTING`? (issue #3132, epic #3134)

TL;DR — recommendation: ENABLE-WITH-TIGHTER-THRESHOLDS (safe, but wire prompt caching into the Haiku lane first)

The router is safe from a quality standpoint: it is correctly conservative. Across the 16-page benchmark set it sent only 2 pages to Haiku (contact-form p0, math-review’s pure-prose intro p0) and kept the other 14 on Sonnet — including every page the #3131 benchmark flagged as unsafe (figure-heavy, equation-dense, form-dense). All 10 tagged outputs (5 docs × 2 arms) are veraPDF PDF/UA-1 compliant, and the two Haiku-routed pages showed no dropped figures, no corrupted alt text, and no equations-as-figures — the specific regressions that made a flat Haiku switch a NO-GO. On complex docs, tiered output is quality-identical to baseline by construction (same model, same prompt).

But the measured cost win is ~0 on this doc set — and for a fixable reason: the Haiku (no-thinking) lane does not emit the ephemeral prompt-cache breakpoint that the Sonnet path uses, so Haiku re-bills the full instruction prompt at the full input rate every call (billable input ~4× Sonnet’s cached input). On small pages this erases nearly all of Haiku’s per-token discount. The router is safe to enable, but enabling it today buys essentially no savings. Fix prompt caching on the Haiku lane, then the safe-lane savings materialize.

What ran

Real flyer-tagger CLI process() pipeline (inventory → vision analysis → injection → structure → verify) per doc, twice:

BASELINE: FLYER_TAGGER_TIERED_ROUTING unset → every page on the base model (Sonnet).
TIERED: FLYER_TAGGER_TIERED_ROUTING=1 → per-page routing.

Both arms passed the same Sonnet base id (us.anthropic.claude-sonnet-4-6), mode="text", run from the main checkout so local fixtures resolve. Client is anthropic.AnthropicBedrock with ANTHROPIC_API_KEY unset — ZDR posture; both tiers are Bedrock Anthropic ids. Driver: /tmp/ab-3134/driver.py (untracked, measurement only — no tracked code was modified). FLYER_TAGGER_EFFORT was left at its main default (medium).

Doc	Profile	Pages	Ran?
beacom-flyer	figure-heavy flyer (1 image inv. on this render)	1	✅
math-review	prose + inline equations	6	✅
bsee-plan	table/form-dense degree plan (banded)	3	✅
ut-resources	mixed prose + figures	5	✅
contact-form	sparse form / text-only	1	✅
Total		16	all 5 ran

No benchmark docs were skipped — all five fixtures resolved from the main checkout. Required coverage is satisfied: figure-heavy (beacom), prose+equation (math-review), form/dense (bsee-plan), and the sparse safe-lane (contact-form).

Routing split (live, tiered arm)

Doc	Pages	→ Haiku	→ Sonnet	Why Sonnet
beacom-flyer	1	0	1	`images>0`
math-review	6	1	5	equation-like text (p1–p5); p0 prose intro → Haiku
bsee-plan	3	0	3	widget/density (banded form)
ut-resources	5	0	5	`images>0` (p0,p2) · equation-like text (p1,p3) · prose_fraction<0.60 (p4)
contact-form	1	1	0	sparse safe lane → Haiku
Total	16	2	14

Divergence from the #3139 design doc (which predicted 4 Haiku / 12 Sonnet, including 2 Haiku for ut-resources): the merged router is stricter than that earlier estimate. Every ut-resources page is now blocked by a real gate — the deterministic route_reason() returns images=1>0 on p0/p2, equation-like text on p1/p3, and prose_fraction=0.152<0.60 on p4. The live run matches the current code exactly; the doc’s table reflected an earlier heuristic. Net: the router is even more Sonnet-biased than documented, which only strengthens the safety case (and further shrinks the savings).

Cost (measured)

Per-1M-token USD (in/out): Sonnet $3/$15, Haiku $1/$5.

Doc	Baseline $	Tiered $	Δ	Routing (H/S)
beacom-flyer	0.0819	0.0597	−27.1%	0H/1S
math-review	0.2650	0.2395	−9.6%	1H/5S
bsee-plan	0.6728	0.7311	+8.7%	0H/3S
ut-resources	0.1693	0.1541	−9.0%	0H/5S
contact-form	0.0141	0.0130	−7.8%	1H/0S
TOTAL	1.2031	1.1974	−0.5%	2H/14S

Read these deltas with care. On the three docs that routed 100% Sonnet in both arms (beacom, bsee-plan, ut-resources), the model, prompt, and thinking config are identical — so the −27% / +8.7% / −9% swings are pure run-to-run adaptive-thinking token variance, NOT routing savings. They net to noise. The only routing-attributable cost change is on the 2 Haiku-routed pages:

Haiku-routed page	Baseline (Sonnet) $	Tiered (Haiku) $	Billable input tok (S→H)
contact-form p0	0.0141	0.0130	2,424 → 9,643
math-review p0	0.0137	0.0138	2,619 → 10,034

Cost is flat on both. The culprit is prompt caching (next section), not the model rate.

Why the Haiku lane shows no savings — prompt-cache regression

The analysis call marks its stable instruction prefix with cache_control: ephemeral. On the Sonnet path this works: billable input_tokens is ~2.4k with a separate cache_creation_input_tokens=2179 on the first page and cache_read_input_tokens=2179 (billed ~0.1×) on later pages. On the Haiku (no-thinking) lane the usage shows cache_read=0 and cache_creation=0 — the breakpoint is not being emitted at all, so the full ~9.6k-token prompt is billed at the full input rate every call. Haiku’s lower per-token rate ($1 vs $3 in) cannot overcome billing 4× the input, so on small pages the net is a wash.

This is a fixable implementation gap, not a model limitation: the no-thinking/Haiku call path needs the same ephemeral cache_control block the thinking path already sends. With caching restored, the Haiku lane’s billable input drops back to ~2.4k and the safe-lane savings (output-token discount + cheaper input) reappear — consistent with the #3131 forced-Haiku projection.

Total live Bedrock spend for this A/B: ≈ $2.40 (both arms, 32 page-analyses).

Quality parity

veraPDF PDF/UA-1 — all outputs clean

Doc	Baseline	Tiered
beacom-flyer	PASS	PASS
math-review	PASS	PASS
bsee-plan	PASS	PASS
ut-resources	PASS (validator)	PASS (validator)
contact-form	PASS	PASS

Strict-gate verdict (`evaluate`, require_verapdf=True)

Identical across arms per doc. beacom, math-review, bsee-plan, contact-form all PASS both arms. ut-resources FAILS both arms identically — page 2: 1 text op left untagged. This is a pre-existing model-omission on a Sonnet page (the page routes to Sonnet in both arms), not introduced by tiered routing — tiered is byte-for-byte the same failure. Flagged for separate follow-up; it does not bear on the routing decision.

Sonnet-routed pages — identical by construction

The 14 Sonnet-routed pages use the same model + prompt + thinking in both arms. Any diff is run-to-run sampling noise; no structural/reading-order/veraPDF divergence was observed.

Haiku-routed pages — inspected individually (the NO-GO tripwires)

The task’s tripwire: a single dropped figure or corrupted alt on a Haiku page is a NO-GO. Neither occurred.

contact-form p0 (sparse form):

Roles: baseline H1,H2,P×10 → Haiku H1,P×11. One H2 subheading demoted to P (role agreement 0.917, 1 disagreement). Reading-order similarity 1.0.
Figures: 0 → 0 (none to drop). Alt text: none involved. Title: identical.
veraPDF: PASS. Verdict: PASS.
Assessment: a minor heading flattening — the benign end of the benchmark’s “Haiku flattens headings” pattern. No content lost, no alt corrupted. On a 1-heading form this is low-impact but is a real, if small, quality nick.

math-review p0 (pure-prose chapter intro, no equations):

Roles: baseline H1,H1,P,LI×6,P×3 → Haiku H1,H2,H3,LI×6,P×3. Haiku is arguably better here — the Sonnet baseline emitted two H1s (an invalid heading structure) and flattened a line to P; Haiku produced a proper H1→H2→H3 ladder. Reading-order similarity 1.0.
Figures: 1 → 1 (no drop). No equations on this page → no equations-as-figures.
veraPDF: PASS. Verdict: PASS.
Assessment: no regression; if anything a small improvement.

Neither Haiku page exhibited the disqualifying failures (dropped figure, wrong/lost alt on named people, equation-as-figure). The router’s equation and image gates kept every risky page on Sonnet, which is exactly the design intent.

Recommendation: ENABLE-WITH-TIGHTER-THRESHOLDS

Scoped to what ran (all 5 benchmark docs, 16 pages, live on Bedrock):

Quality is safe to enable. The router is aggressively conservative (2/16 pages to Haiku here — even more Sonnet-biased than the design doc), every risky page stayed on Sonnet, all outputs are PDF/UA-1 clean, and the two Haiku-routed pages had no figure drops or alt corruption. On complex docs tiered == baseline by construction.
But do not enable expecting savings yet — fix the Haiku prompt-cache gap first. The Haiku lane bills full uncached input every call (cache_read=cache_creation=0), erasing its rate advantage on small pages; the measured net was −0.5% (noise). Wire the ephemeral cache_control breakpoint into the no-thinking/Haiku call path (the thinking path already has it), then re-run this A/B. That is the one change standing between “safe but pointless” and “safe and cheaper.”
Tighten the contact-form-class heading nick (optional). The one real quality delta was an H2→P demotion on the sparse form. If the Haiku lane is widened later, add the #3131-recommended semantic gate (heading-count sanity in addition to veraPDF) so a heading-flattening page is caught, since veraPDF does not flag it.

Do NOT flip the flag in prod as part of this measurement. No tracked code was changed and the flag remains OFF. Suggested order: (a) land the Haiku-lane cache fix, (b) re-run this A/B and confirm a real cost delta with unchanged quality, (c) then enable in prod behind the existing env knobs, monitoring the semantic signals above.

Reproduce

# from the MAIN checkout (fixtures resolve here)
cd services/flyer-tagger
export AWS_PROFILE=accessible-sso AWS_REGION=us-east-1 PYTHONPATH=$(pwd) FT_ROOT=$(pwd)
unset ANTHROPIC_API_KEY            # Bedrock / ZDR
AB_OUT=/tmp/ab-3134/out python3 /tmp/ab-3134/driver.py    # both arms, all docs
python3 tools/haiku_quality_diff.py /tmp/ab-3134/out       # after symlinking
                                                           # baseline→sonnet, tiered→haiku

Raw artifacts (tagged PDFs, analysis JSON, results.json) were generated under /tmp/ab-3134/out and are not committed (full tagged renditions).

#3142 caching fix — verification (addendum)

The A/B above found “no savings materialized” partly because the Haiku (no-thinking) lane’s stable instruction prefix was not being cached at all — every Haiku call returned cache_creation = cache_read = 0, re-billing the full instruction prompt at full input rate. Root cause (#3142): prompt caching has a per-model minimum-cacheable- prefix floor. Our analysis prefix bills ~2179 cached tokens — above Sonnet 4.6’s 2048-token floor (so the Sonnet lane cached fine) but below Haiku 4.5’s 4096-token floor, so Haiku silently cached nothing. The fix pads the Haiku-lane cached prefix past the 4096 floor (byte-stable, instruction-neutral; the Sonnet lane is byte-identical / no-op).

Live Bedrock verification (ZDR, us-east-1, `AnthropicBedrock`)

cache_read before → after: 0 → 13956 on math-review’s Haiku page (0 → 6066 in a controlled 2-call proof of the same prefix).
Haiku-page cost: on a multi-band / multi-Haiku-page doc where the prefix is re-sent, math-review’s Haiku page dropped −54% ($0.0234 → $0.0108, cached read at ~0.1×). A lone single-call Haiku page (contact-form) costs marginally more (a cache write bills ~1.25× with nothing yet to amortize against) — expected; caching only pays back on re-use.
veraPDF: both docs still PDF/UA-1 clean (0 failed rules) after the fix.

Effect on the enablement decision

This clears the #3142 caching gate — Haiku now benefits from prefix caching like Sonnet. It does not overturn the recommendation above: FLYER_TAGGER_TIERED_ROUTING stays OFF. Enabling it is a separate decision, and the savings remain marginal because the conservative predicate keeps the expensive dense/table/figure/equation pages on Sonnet and the Haiku-eligible pages are often sparse single-page docs where a lone call cannot amortize a cached prefix. The value of this fix is that when tiered routing is used (opt-in for known-sparse workloads), the Haiku lane no longer silently throws away its caching.