ZDR cheap-model eval — Nova Lite vs Claude Haiku vs Claude Sonnet

Why this eval exists

We promise Zero Data Retention (ZDR), which today means all document AI is rerouted onto Amazon Bedrock (bedrock-gemini-shim.ts, issue #2242). Gemini, Mathpix, and Datalab Marker are not ZDR and are off the table for ZDR jobs.

The shim currently maps Gemini-class work onto Claude:

Gemini class	Bedrock target today	Price (in/out per 1M)
Flash (bulk, cheap)	`us.anthropic.claude-haiku-4-5-20251001-v1:0`	$1.00 / $5.00
Pro (hi-fi, escalation)	`us.anthropic.claude-sonnet-4-6`	$3.00 / $15.00

That is the ZDR cost tax: bulk vision moved from Gemini Flash (~~$0.15/$2.50) onto Claude Haiku (~~$1/$5), roughly 6× more expensive on input.

The cost ledger (production, lifetime through 2026-08-27) shows a ZDR-compatible alternative already runs in our pipeline and is dramatically cheaper:

Model	Calls	Input tok	Output tok	Cost
`claude-sonnet-4-6` (direct)	277	6.32 M	1.91 M	$54.92
`amazon.nova-lite-v1:0` (Bedrock, ZDR)	402	6.11 M	1.43 M	$0.71

Comparable token volume, ~55× cheaper, already ZDR. But Nova Lite is only wired into the ai-fix path today — not the conversion vision path. Before we repoint bulk conversion at Nova we need head-to-head quality data, because a cheaper model that drops content or breaks tables costs us in remediation and trust, not just tokens.

Question this eval answers: on our real page mix, does Nova Lite hold acceptable conversion quality vs Claude Haiku (current ZDR bulk default) and Claude Sonnet (ZDR ceiling) — and what is the true $/page of each?

conversion is 79.7 % of all AI spend, so this is the lever that matters.

Models under test (all Bedrock, all ZDR)

Backend id (new)	Bedrock model	Role
`bedrock-nova-lite`	`amazon.nova-lite-v1:0`	Candidate cheap tier
`bedrock-nova-micro` (optional)	`amazon.nova-micro-v1:0`	Stretch-cheap (text-only pages)
`bedrock-claude-haiku`	`us.anthropic.claude-haiku-4-5-20251001-v1:0`	Current ZDR bulk default (baseline)
`bedrock-claude-sonnet`	`us.anthropic.claude-sonnet-4-6`	ZDR quality ceiling (reference)

Gemini backends (gemini-flash, gemini-pro) may be run out of band as an upper-bound reference only — they are non-ZDR and cannot ship to ZDR customers, so they never win the eval, they only calibrate the ceiling.

Reusing the existing harness

The benchmark backend from #2016 already produces every axis we need per pipeline — no new scoring code required:

Content fidelity — content-fidelity-scorer.ts → ContentFidelityScore (wordCoverage, placeholderCount). Catches Nova dropping/ truncating text.
Visual-layout fidelity — visual-layout-scorer.ts → VisualLayoutFidelityScore (breakdown: text, spatial, tableStructure, imagePlacement, headingHierarchy). Catches table/layout collapse.
WCAG — custom validator + axe-core (AuditResult). Post-remediation pass rate.
HTML quality — HtmlQualityMetrics (semantic ratio, heading validity, tables-with-headers, mathTagsWithValidMathml).
Cost — real TokenUsage.estimatedCostUsd per pipeline (not the BACKEND_COST_PER_PAGE estimate).

convertWithBedrockClaude(pdf, filename, { creds, model }) already takes a model id, so Haiku and Sonnet need no converter changes — only backend registration.

Code changes required (small, one PR)

packages/shared/src/benchmark-types.ts
- Add the four ids to the ConversionBackend union.
- Add BACKEND_COST_PER_PAGE entries (placeholders; real cost comes from token usage): bedrock-nova-lite: 0.002, bedrock-nova-micro: 0.001, bedrock-claude-haiku: 0.01, bedrock-claude-sonnet: 0.02.
workers/api/src/services/benchmark-orchestrator.ts (backend switch)
- bedrock-claude-haiku / bedrock-claude-sonnet → call convertWithBedrockClaude(pdf, name, { creds, model }) with the matching us.anthropic.… id.
- bedrock-nova-lite / bedrock-nova-micro → new convertWithBedrockNova.
Nova converter — workers/api/src/services/bedrock-nova-converter.ts (new)
- Nova speaks the same Bedrock Converse API, but the message shape and document handling differ from Claude, so callClaudeBedrock cannot be reused verbatim. Either generalize bedrock-claude.ts to a callBedrockConverse(model, …) or add a thin Nova twin. Reuse assemblePrompt() unchanged so the only variable under test is the model.
- Verify Nova accepts native PDF document blocks in Converse. If it does not, fall back to the per-page image path (segmented-converter.ts already renders page PNGs for the ZDR image route) — note this in results because it changes the token profile.
Cost attribution — confirm is_bedrock_model() (migration 254) matches amazon.nova-* ids, not just us.anthropic.*. Nova already appears as Bedrock spend in the ledger, so it likely does; verify before trusting the eval’s cost rollup.

Sample set (representative, not cherry-picked)

The router (pdf-complexity-detector.ts) branches on page content type, so the sample must exercise every branch. Target 12–20 real PDFs, ~3 per bucket, biased toward what the incoming order actually contains:

Bucket	Why it stresses the cheap model
Text-heavy prose	Baseline; cheap model should win outright
Dense tables	Table-structure collapse is Nova’s likeliest failure
Math / equations	Non-ZDR Mathpix is excluded — measure native model math
Scanned / image-only (OCR)	No text layer; pure vision
Mixed / multi-column	Reading order + spatial layout
Long doc (30+ pp)	Nova output-token cap (see risks) surfaces here

Selection:

Start from the curated set in tools/benchmark-cli/calibration/ if it already spans these buckets.
Top up from production: pick real converted files by bucket (e.g. join cost_ledger / file metadata on has_math, page count, image density). Use files representative of the prospective customer’s documents if we have a sample from them — that is the single most predictive input.
Freeze the set (store file ids in the eval config) so re-runs are comparable.

Metrics & decision rubric

For each (file, backend) pipeline, record: WCAG pass (custom + axe), content fidelity score + wordCoverage, visual-layout score + tableStructure, placeholderCount, and actual $/page from token usage.

Primary decision axes: content fidelity, table-structure fidelity, cost.

Go / no-go for Nova Lite as the ZDR bulk default

Adopt Nova Lite for a bucket if, versus the Claude Haiku baseline, on that bucket:

wordCoverage regression ≤ 2 % (absolute), AND
placeholderCount no worse, AND
visual-layout tableStructure regression ≤ 5 points, AND
post-remediation WCAG pass rate not lower, AND
cost materially lower (expected ~15–20×).

If Nova passes on some buckets but not others (likely: wins text/prose, struggles on dense tables), ship a per-content-type routing rule: Nova for the buckets it passes, Haiku/Sonnet escalation for the rest. The router already classifies pages, so this is a routing-table change, not new plumbing.

Nova Micro is a stretch: only consider it for the text-only bucket, same rubric.

Cost projection (feeds the order quote)

Plug the eval’s measured $/page into the order size. Indicative, pre-eval:

ZDR path	~$/page	100 K pages	500 K pages
Claude Sonnet hi-fi	~$0.10	~$10,000	~$50,000
Claude Haiku (current bulk default)	~$0.04	~$4,000	~$20,000
Nova Lite (this eval’s candidate)	~$0.002	~$200	~$1,000

Layer two more ZDR-safe multipliers after the model choice is settled:

Bedrock batch inference = 50 % off (our pipeline is already SQS/async).
Bedrock prompt caching on the Claude escalation tier (already live; $18.78 saved in the ledger to date).

Risks & things to watch

Nova output-token cap. Nova Lite truncated ACR output at ~5 K tokens historically (see project_acr_vpat_parse_claude_haiku). A dense full page of HTML can exceed that → silent truncation. The placeholderCount + wordCoverage scorers will catch it; set maxTokens explicitly and watch the long-doc bucket.
Document-block support. If Nova won’t take a native PDF document block in Converse, we fall back to per-page images — different token profile, must be noted so the cost comparison stays apples-to-apples.
Escalation multipliers. sonnet-hifi-escalation and text-coverage retries are pure cost multipliers. Measure how often a Nova base would trigger escalation — a cheap base that escalates half the time isn’t cheap.
Compliance-claim wording. This eval keeps us fully inside Bedrock ZDR. If we later add a self-hosted VLM (cheaper still, ZDR-by-construction), confirm customer contracts say “zero data retention,” not “Amazon Bedrock” specifically.

Run procedure

Land the backend-registration PR (above); keep PDF_BEDROCK_ENABLED / production routing untouched — the eval runs through the benchmark backend, not the live path.
Freeze the sample set (file ids) in the benchmark config.
Run the harness across the four backends × the sample set. Options:
- CLI: tools/benchmark-cli (offline, scriptable, cheapest to iterate).
- UI: /benchmark in apps/web for visual side-by-side diffing.
Export the per-pipeline table (fidelity, WCAG, $/page) and apply the rubric per bucket.
Write the go/no-go + the per-content-type routing table into a short results addendum at the bottom of this doc.
If Nova wins a bucket: repoint the shim’s flash-class target for that bucket (GEMINI_BEDROCK_FLASH_MODEL override) and roll out one runtime at a time, watching the ledger’s per-model rollup.

Results

(Fill in after the run: per-bucket table, go/no-go, routing decision, measured $/page, and any Nova truncation/table-collapse notes.)