Tag-tree golden corpus

Epic #3856. veraPDF tells us a Preserve output is conformant. It does not tell us the tree is correct: a budget table tagged as a list, two columns fused into one paragraph, a running footer tagged as H1, and a formula tagged as a garbled paragraph all pass PDF/UA-1. The golden corpus is a set of known-good tagged PDFs, each with a reviewed spec of what its tree must say. A runner tags each document with the local flyer-tagger and scores the output against its spec, then reports pass rates per document type.

Code: tools/tag-corpus/ (Python package tagcorpus). Corpus data lives in R2 only, never in git, because documents may carry customer text under the public-corpus exception.

RCL2DEV:accessible-dev-assets/tag-corpus/
  README.txt
  inbox/<type>/            Laura drops known-good tagged PDFs here (+ optional .txt notes)
  docs/<type>-<sha10>/
    source.pdf             the known-good original, never modified
    input.pdf              the same pages with every tag removed (what Preserve is tested on)
    spec.yaml              expected-tree assertions + review_flags (draft until reviewed)
    spec.generated.yaml    a newer draft when spec.yaml is already reviewed (diff and merge)
    meta.json              original name, sha256, strip report
    notes.txt              the dropper's notes, if any
    recordings/            recorded model + Textract responses (calls/<sha256>.json, manifest.json)
  rejected/<type>/         inbox files that could not be ingested, with a .reason.txt

Types: checklist, financial-schedule, newsletter, catalog, policy-manual, formula-handout, flyer, scanned, form, report, slides-export, other.

rclone writes to this bucket must pass --s3-no-check-bucket, or R2 answers the implicit CreateBucket with a 403. The tooling does this for you.

1. Adding documents (Laura)

Pick a PDF whose tags are already right: tagged by a careful human, or a public document from a source known for good tagging. The spec is generated from these tags, so a wrongly tagged source becomes a wrong expected answer.
Customer files need written approval under the public-corpus exception. Public or synthetic files need nothing.
Upload it to tag-corpus/inbox/<type>/ using the type that best describes it. Add a .txt file with the same name for notes. A line such as source: public:https://example.gov/report.pdf or source: approved-customer:<ticket> goes straight into the spec.
Terminal window
```
rclone copyto budget.pdf RCL2DEV:accessible-dev-assets/tag-corpus/inbox/financial-schedule/budget.pdf --s3-no-check-bucket
```

A developer then runs ingest. Each file is moved to docs/<id>/, and the inbox copy is deleted only after every upload has succeeded. A file with no structure tree, or one that cannot be stripped pixel-identically, goes to rejected/ with a reason. Dropping the same bytes twice does nothing.

2. Reviewing a spec (about 5 minutes per document)

Open docs/<id>/source.pdf in Acrobat (Tags panel) next to spec.yaml.

Start with review_flags. These mark places where the source tree looks suspicious:

flag	meaning
`h1_flood`	H1 on most pages; usually a running title
`heading_level_skip`	H1 followed by H3, for example
`table_without_th` / `table_mostly_empty` / `table_ragged_rows` / `single_row_table`	layout table, split table or broken grid
`figure_without_alt` / `formula_without_alt`	missing alternative text
`element_spans_columns`	one element contains text from both sides of a column gutter
`margin_text_tagged`	a running header, footer or page number is in the tree instead of being an Artifact
`untagged_text`	more than 2% of the painted text is neither tagged nor an Artifact
`undecodable_text`	some text could not be decoded, so spec text may be wrong in those places
`mcid_referenced_twice`, `tree_problem`, `empty_elements`, `no_lang`	structural issues

For each flag, either fix the spec (for example delete an assertion that encodes the defect, or add advisory: true) or drop the document.

Read outline.exact. It should be the document’s real heading outline. Remove running titles. If the level of a heading is debatable, add level_tolerance: 1.
Check each table, list, figure and formula entry against the page it names.
Delete review_flags entries you have handled, then set reviewed_by: <name> and reviewed_at: <YYYY-MM-DD>. Upload the edited file to docs/<id>/spec.yaml.

ingest and respec never overwrite a reviewed spec. When the generator improves, respec writes spec.generated.yaml beside a reviewed spec so you can diff the two.

3. Spec format

spec_version: 1
doc: budget.pdf
type: financial-schedule
source: public:https://…      # | synthetic | approved-customer:<ref>
reviewed_by: null
reviewed_at: null
pipeline: preserve
document: {lang: en-US, title_contains: "Annual Budget", pages: 2}
outline:
  exact: [{H1: "Annual Budget Report 2026"}, {H2: "Required Courses"}]   # or in_order + count_between for >80 headings
  forbid_levels_skipped: true
  max_h1: 1                    # catches the H1 flood
  # optional: allow_missing: 0, allow_extra: 0, level_tolerance: 0
figures:
- {id: figure-1, page: 2, bbox: [72, 200, 272, 320], near_text: "…", role: Figure,
   source_alt: "Bar chart of tuition growth by year", alt_must_mention: [tuition], alt_max_chars: 250}
  # role: Artifact asserts the image must NOT be a Figure (decorative)
tables:
- id: table-1
  pages: [1]
  caption_or_heading: "Tuition and fees"
  role: Table
  min_rows: 3
  columns: 3
  min_full_rows: 3             # rows with every column filled; fused or split cells empty some
  header_row: true             # first row is all TH
  row_headers: false           # most body rows start with a TH
  must_contain_row: ["Tuition", "$5,000", "Per term"]   # distinct cells, in order
  last_row_contains: "Housing"
  bbox_first_page: [72, 360, 455, 410]
  # optional: allow_row_deficit, columns_tolerance
lists:
- {id: list-1, pages: [2], after_heading: "Required Courses", role: L, min_items: 3,
   first_item: "1. Algebra fundamentals", last_item: "3. Research methods seminar", each_item_matches: '^\d+[.)]'}
formulas:
- {id: formula-1, page: 2, bbox: [72, 360, 114, 372], near_text: "…", role: Formula,
   source_alt: "E equals m c squared", alt_required: true}
invariants:
- no_element_spans_columns: {gutters: {1: [[281.0, 330.0]]}}   # page -> gutter x-ranges
- no_duplicate_text: true      # no MCID used twice, no layered duplicate elements
- not_in_role: {text_matches: '^budgetreportpage\d+$', roles: [H1, H2, H3, H4, H5, H6, Title]}
- margin_text_is_artifact: [{text_matches: '^budgetreportpage\d+$', band: bottom, sample: "Budget Report - Page 1"}]
- reading_order: {before: "…end of column one", after: "Start of column two…", page: 1}
- tree_text_coverage: {min: 0.91}   # share of painted text that must be in the tree
- opaque_text: {max_chars: 2, hidden: [[1, 98, 114, '2019']], graphics: {1: [[72, 100, 372, 300]]}}
tolerated: {wrapper_roles: [Art, Aside, Div, Document, DocumentFragment, Index, NonStruct, Part, Private, Sect, Span]}
review_flags: [{code: …, page: …, message: …}]

Any assertion dict can take advisory: true. It is still scored and reported, but it does not fail the document.

The scorer fails closed

A check that could not run is a failure, never a pass:

An unknown key anywhere in the spec (min_row for min_rows, role for roles in not_in_role) fails spec.schema. So does an invariant body with a required key missing or empty. An entry that cannot be evaluated (a bad regex, a wrong type) fails with could not evaluate.
A spec with no required assertions fails spec.empty. outline.exact: [] is an assertion: the output must have no headings.
Content the walker could not read (an undecodable content stream, a Form XObject past the recursion cap) fails document.content_unread. A shared or cyclic struct element fails document.tree_problems.
tree_text_coverage fails when no painted text was found. margin_text_is_artifact fails when a rule’s running text is not found on any page. A list or reading-order anchor with no comparable text fails.
Figure, Formula and Artifact elements are opaque, as they are to assistive technology: headings, lists and tables inside them do not count, a Figure that contains heading, list or table structure fails, and text under an Artifact element is not coverage. A Figure or Formula reads as its alt text in reading-order checks, and opaque_text fails when more painted text sits inside figures than in the source (a paragraph swallowed into a Figure).
not_in_role fails when its pattern matches no painted text (patterns apply to squashed text: lowercase, no spaces or punctuation) or names a non-standard role. A Figure only matches a spec box when its area is within 4x of the box, so a page-sized Figure cannot satisfy every figure on its page.
A table split into several Table elements on its first page fails (only continuation tables on later pages are merged; a table is located by its source box, and by row text only when the spec has no box).
opaque_text catches text hidden inside a Figure or Formula (AT reads only the alt). The spec records, from the source, every text fragment hidden inside figures (page, centre, text) and the visible area of each figure’s graphics (clipping paths honoured). Output fragments the source also hid are fine however they are grouped. The rest are judged per segment (a line split at wide gaps): on a source figure’s graphic, a segment of at most 80 characters that is not a sentence (under 7 words, and not 3+ words ending in a stop: axis titles, legend entries, ticks such as “Jan.”) is fine; inside a Formula, non-prose text (inline math) is fine. Captions, photo credits, paragraphs and prose (three 4-letter words plus a function word or a full stop) swallowed into a figure or formula fail; 2 characters of slack.
Outline text is kept up to 300 characters. A heading only matches by prefix when the spec text was cut at that limit, so a heading that swallowed the next paragraph does not match.

How matching stays tolerant

Wrapper roles are transparent. Adding or removing Div, Sect, Part, Span and similar wrappers never changes a verdict, including wrappers between TR and TD or between L and LI.
Text is compared in squashed form: NFKC, case-folded, with punctuation and all whitespace removed. Long strings also get a fuzzy match. Source and output text come from the same painted glyphs, because stripping never touches drawing operators. That means differences are in grouping, not characters.
Geometry anchors elements. Figures and formulas are matched by page and bounding box, and tables by first-page box or row content. Continuation tables on later pages are merged back together.
reading_order pairs are generated only where a top-to-bottom sort would get the order wrong, such as a column jump. Pairs inside figures are not generated.

Text extraction

The walker (content.py, walk.py) interprets every page and Form XObject content stream itself. It tracks the CTM, text state and marked-content stack, and decodes strings through the font’s ToUnicode keyed by the codespace of the Encoding CMap. That is why a CID TrueType font with a custom embedded CMap and 1-byte codes decodes correctly. pdfminer returns empty text for that font shape, which is the WeasyPrint Liberation subset case. A unit test builds such a font. Runs that still cannot be decoded are filled from PyMuPDF’s text at the same position. The walker also handles RoleMap chains, MCR with /Pg and /Stm, OBJR, array or single /K, inherited /Pg and cycles.

4. Developers

Setup: Python 3.11+ with pikepdf, PyMuPDF, PyYAML and fontTools (the flyer-tagger’s dependencies). The runner imports services/flyer-tagger and packages/pdf-enhance from the monorepo and does not need them installed.

cd tools/tag-corpus
python -m pytest -q                               # synthetic fixtures only

python -m tagcorpus ingest                        # inbox/ -> docs/<id>/
python -m tagcorpus respec [--id ID]              # regenerate unreviewed specs after spec_gen changes
python -m tagcorpus run --mode replay             # CI mode: no model calls; any cache miss fails the doc
python -m tagcorpus run --mode record --budget 5  # needs `aws sso login`; records missing calls only
python -m tagcorpus run --mode replay --baseline last-good.json   # exit 1 on any per-doc or per-type regression

# single-file helpers
python -m tagcorpus walk tagged.pdf               # tree with per-element text
python -m tagcorpus spec tagged.pdf -o spec.yaml --type flyer
python -m tagcorpus strip source.pdf -o input.pdf # + pixel-identity and no-tags verification
python -m tagcorpus score output.pdf spec.yaml

The working directory defaults to tools/tag-corpus/.work/ (gitignored). Run reports go to .work/reports/run-<timestamp>.json. Each report holds the per-type pass rates, the environment used, and for each document every assertion with ✓ or ✗ and a short reason.

Only reviewed specs gate. A spec counts as reviewed when it has both reviewed_by and reviewed_at; setting only one fails spec.schema and is a run error, and respec treats either one as protection. by_type covers reviewed specs only; drafts are reported under drafts_by_type and never fail a run.

Exit codes. run stops with an error when it would score no documents, or when an --id/--type is not in the corpus. It exits 1 when any document errors. Without --baseline it also exits 1 when any reviewed document fails. With --baseline it exits 1 on a regression: a reviewed document that passed in the baseline and now fails or is missing, or a type whose pass rate dropped. A baseline file that is not a run report exits 2 rather than comparing against nothing.

Fidelity. Besides the spec, every document must render pixel-identical to its input.pdf (output.pixels): Preserve never changes pixels, so a changed page means content was lost or added. When the tagger reports a pixel_baseline (it embedded substitute fonts or prepared form widgets first), the output is compared with that intermediate instead, and the report says so. The tagger’s own verdict must also be a pass (output.tagger_verdict). The previous run’s output.pdf is deleted before tagging, so it can never be scored by mistake.

How `run` mirrors production

run works like deploy/handler.py’s _run. Preflight chooses the mode, then process() runs with the production Lambda’s tagging environment (PROD_ENV in run.py, read from the flyer-tagger function on 2026-09-28):

FLYER_TAGGER_MODEL=us.anthropic.claude-sonnet-4-6
FLYER_TAGGER_TEXTRACT_ENABLED=true
FLYER_TAGGER_INTERACTIVE_FORMS=1
FLYER_TAGGER_FORM_ENHANCE=1
FLYER_TAGGER_MATH_ALT=true

There is no flatten and no checkpoint. ANTHROPIC_API_KEY is unset for the run, because Preserve runs on Bedrock. Documents that preflight would send to reflow are still tagged, and the report records the route. When prod changes a flag, update PROD_ENV and re-record.

Record and replay

All of the tagger’s model calls go through flyer_tagger.analyze._client_and_model: page analysis, OCR analysis, figure re-description, adjudication and Textract-lane alt text. All Textract calls go through flyer_tagger.textract_ocr._client. recorder.py wraps both.

The key is a SHA-256 of the full request: model id, system prompt, messages including the page-image bytes, schema, temperature and token limits. A prompt, schema, model or rendering change therefore produces a new key. Checkpoint keys cover page text and model but not the prompt, so reusing them would silently replay stale analyses. That is why the checkpoint store is not used here.
Replay makes no network calls. A miss raises ReplayMiss, which is a BaseException, so flyer-tagger’s fail-open except Exception blocks cannot swallow it. The miss is also recorded on the store, and the document then fails with the call’s model and prompt head.
Record serves what is already recorded and calls Bedrock only for the rest. It stops at --budget (default $5). It first writes the live spend to the prod cost_ledger and only then uploads recordings/. If ledgering fails, the recordings are not uploaded and the document errors, so unledgered spend can’t hide behind later free replays. Local recordings are reset to the store’s copy before every document, so a recording that was never uploaded is never served. It ledgers through flyer_tagger.devcost.record_dev_run (backend bedrock-dev, served Bedrock model id), so the Bedrock divergence monitor reconciles it. Credentials come from SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY, or from the local-cost-sync file ~/.config/accessible-cost-sync/env.
Re-recording after a prompt or model change: run run --mode replay and expect misses. Then run run --mode record. Only the changed calls are paid for. Diff the replay report against the previous one and review every change in pass rate as part of the PR. To force a full re-record of one document, delete docs/<id>/recordings/ in R2 first.
Keys include the rendered page JPEG, so a PyMuPDF upgrade that changes rasterization also forces a re-record. The miss says so.

Reading a report

FAIL  other-fed2e6cd3d   14/21 live=0 replayed=11 $0.0000
per-type pass rates:
  other   1/3 (33%) errors=0

In the JSON, docs[].results[] holds {id, kind, passed, reason, advisory}. Reasons name the defect directly. Examples: no Table found; 'Tuition' found as LI, row … not found as separate cells (fused: 'pdfd:conformsTo URI'), 10 H1s (max 1) — H1 flood, running text in the tree on 6 page(s), and no Formula at p2 (tagged as ['P']).

5. CI job (sketch; not wired, and not a required check yet)

Replay needs only R2 read access. It makes no Bedrock calls and needs no AWS credentials. Wire it in once enough documents are reviewed for the pass rates to mean something.

# .github/workflows/tag-corpus.yml  (sketch)
name: tag-corpus
on:
  pull_request:
    paths: ["services/flyer-tagger/**", "tools/tag-corpus/**"]
jobs:
  replay:
    runs-on: [self-hosted]          # has rclone + the RCL2DEV remote (R2 read)
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e services/flyer-tagger -e tools/tag-corpus
      - run: python -m pytest -q tools/tag-corpus
      - run: rclone copyto RCL2DEV:accessible-dev-assets/tag-corpus/baseline.json baseline.json
      - run: >
          python -m tagcorpus run --mode replay --baseline baseline.json
          --report tag-corpus-report.json
        working-directory: tools/tag-corpus
      - uses: actions/upload-artifact@v4
        with: {name: tag-corpus-report, path: tools/tag-corpus/tag-corpus-report.json}

The job fails on a replay miss, which means the PR changed a prompt or model without re-recording. It also fails when a reviewed document that passed in baseline.json (the report from the last accepted run on main) now fails or is missing, or when a type’s pass rate drops. After a reviewed improvement lands, upload the new report as the baseline.

Pilot (2026-09-28)

Three public or synthetic documents in inbox/other/ went through the full path: ingest, strip, spec, record, score. Stripping was pixel-identical at 100 dpi on every page, with no MCIDs left. Recording cost $0.36 in total and was ledgered as bedrock-dev.

id	document	flags	score	live cost
other-68b91530a1	WeasyPrint ACR table (repo fixture, synthetic)	0	9/9 PASS	$0.023
other-4c9e96740c	PDF Association PDF 2.0 AN001 (5 pp)	0	9/14 FAIL	$0.137
other-fed2e6cd3d	PDF Association PDF Declarations (10 pp)	2 (`margin_text_tagged`)	14/21 FAIL	$0.197

The failures are real Preserve defects of the classes in #3856:

Declarations tables have columns fused into one TH (pdfd:conformsTo URI) and a continuation table whose header is fused (Type Property content).
Running headers are tagged instead of being Artifacts.
H1 appears 10 times where the source has one H1, and 4.1/4.2 headings come out as P.
The vector chart on AN001 p4 has no Figure; its labels are tagged as Captions.

Some failures are judgement calls for the reviewer. Examples are the cover title becoming H1, and whether the logo should be a Figure or an Artifact. None of the three specs has been reviewed yet.