Tag-studio task benchmark

Epic #3866, Phase 0. This is the benchmark we use to compare Gevell Tags with Adobe Acrobat Pro (Tags panel, Reading Order tool, Accessibility Checker), Grackle and Siteimprove, and to gate each phase of the epic. It targets PDF/UA-1 and WCAG 2.1 AA. PDF/UA-2 tasks are added in Phase 6.

The rule is simple: “better” is a number from this benchmark, not a claim. A task counts as done only when a machine check (or a blind human check where no machine check is possible) says the end state is right. Speed without a correct result counts as a failure.

1. Design

1.1 Seeded defects on known-good documents

Most tasks start from a golden source: a correctly tagged PDF from the tag-tree corpus (epic #3856, PR #3862, tools/tag-corpus). A new tagcorpus inject command applies one defect class to a chosen scope (pages, elements) and writes start.pdf. The expected end state is then the golden tree inside that scope, which the corpus scorer can already check.

This has three advantages over hand-built task files:

The expected answer exists before anyone runs the task, and it was reviewed once (the corpus spec review), not per task.
We can make several isomorphic variants of the same task (same defect, different pages or documents), so repeat runs don’t reward memory.
Anything outside the injected scope must be unchanged, so collateral damage (errors introduced) is measurable as a tree diff.

Tasks that cannot be seeded (untagged, scanned, OCR) start from a real untagged file and are scored against a hand-reviewed spec.

1.2 Task package

Each task lives in R2 at RCL2DEV:accessible-dev-assets/tags-bench/tasks/<task-id>/ (same bucket and rules as the corpus; no documents in git):

task.yaml          id, title, brief, golden doc id, scope, defect recipe, assertions, time cap
brief.md           tool-neutral instructions shown to the operator
variants/<v>/      start.pdf, golden.pdf (scope-limited reference), task-spec.yaml,
                   expected-sr.txt (screen-reader transcript of the scope), inject.json

task-spec.yaml uses the corpus spec format (outline, figures, tables, lists, formulas, invariants) plus the extensions listed in section 5.2, restricted to the task scope.

1.3 What we record for every run

Metric	Definition	Source
`success`	every non-advisory task assertion passes	`tagcorpus bench score`
`wall_s`	seconds from “file open and interactive” to “save completed”	harness timer + screen recording
`actions`	clicks + key presses + drag gestures (a drag = 1); scroll recorded separately as `scrolls`	OS-level input counter, same for every tool
`errors_introduced`	sum of: new veraPDF failed checks vs `start.pdf`; new tag-lint errors/warnings; tree changes outside scope (collateral diff); any pixel diff	`bench score`
`verapdf_ua1`	document passes veraPDF PDF/UA-1, plus failed-rule list and delta vs start	`verify.verapdf_check`
`pac`	PAC 2024 PDF/UA and WCAG result, failed checkpoints	PAC report export (manual, see 4.4)
`semantic_pass`	share of task assertions passed (0–1) and list of failures	corpus scorer
`sr_match`	1 − normalised edit distance between the scope’s screen-reader transcript and `expected-sr.txt`	serializer + diff
`sr_human`	0/1/2 rubric from a blind NVDA listen (sampled runs only)	rater
`human_quality`	0/1/2 rubric for judgment items (alt text, note placement), blind	rater
`features_used`	auto-tag, AI suggestions, OCR, bulk ops, validator fix-all	operator log + recording
`crashes`, `hangs_s`	app crashes and time spent unresponsive over 2 s	recording
`dnf`	`unsupported` (tool cannot do it) or `timeout`	operator

A run is correct when success is true, errors_introduced is 0 and the PDF/UA-1 result is not worse than the start file on any rule inside the task’s scope. Only correct runs count as successes in time comparisons; the rest are scored at the time cap (section 3.4).

2. Tasks

Twenty-two tasks. “Golden” names a document with a correct tree; “start” is what the operator receives. Sources marked (gap) need a document we don’t have yet (section 6). Repo paths are relative to the monorepo root; corpus ids refer to tag-corpus/docs/<id>.

Candidate golden sources already on hand (role counts measured 2026-09-28):

Doc	Pages	Useful structure
`docs/standards/Well-Tagged-PDF-WTPDF-1.0.pdf`	57	TOC/45 TOCI, 26 L, 105 Lbl, 10 tables with 139 span attrs and 114 Headers attrs, 173 Link, 28 `Lang ja` spans
`docs/standards/Tagged-PDF-Best-Practice-Guide.pdf`	72	TOC/42 TOCI, 2 Note, 26 L, 8 tables, 8 Figure, 73 Link
`docs/standards/PDF-Declarations.pdf` (corpus pilot)	10	3 tables with spans and Headers, TOC, vector chart
`docs/standards/PDF20_AN001-BPC.pdf` (corpus pilot)	5	3 Figure, TOC, short and clean
`docs/standards/ISO-14289-1-2014-sponsored.pdf`	25	TOC/50 TOCI, 17 L, running headers/footers, 1 `Lang fr` span
`docs/standards/ISO_32000-2_sponsored_EC3.pdf`	1023	tagged, 988 outline entries, 6031 links, many tables
WeasyPrint ACR table (corpus pilot, synthetic)	1	clean header-row table
`TheAccessible-Contact-Form.pdf`	1	tagged form, 7 fields

The two PDF Association “sponsored” ISO standards are licensed copies. Keep them in the dev-assets bucket for internal benchmarking only; never publish a benchmark output containing their pages.

ID	Task	Golden / start	Starting defect	Expected end state (machine check unless noted)	Scoring
T01	Reading order: multi-column page with sidebar	(gap) golden newsletter, 3 columns + boxed sidebar + pull quote; start = inject	Page’s struct kids sorted top-to-bottom across columns, so column 2’s first paragraph interrupts column 1; sidebar in the middle of a paragraph run	`reading_order` pairs at every column jump pass; new `contiguous` assertion: the sidebar’s text is one contiguous run in `text_sequence` and sits either after the main story or at its golden position; `no_element_spans_columns` passes	Auto + `sr_match`
T02	Retag misidentified headings, fix hierarchy	Best Practice Guide pp. 1–20; start = inject	6 real headings tagged P; 2 bold callout paragraphs tagged H2; one H1→H3 skip; running title tagged H1 on every page	`outline.exact` equals golden for the scope; `max_h1: 1`; `forbid_levels_skipped`; running title not in any H role	Auto
T03	Alt text; mark decorative images	Best Practice Guide figures + (gap) 1 page with 3 decorative images (rules, background, ornament); start = inject	5 informative Figures lose `/Alt`; 3 decorative images are tagged Figure with junk alt (“image1.png”)	Informative: `role: Figure`, Alt present, `alt_must_mention` keywords, `alt_max_chars: 250`, no filename-like alt. Decorative: `role: Artifact` (not in tree). Alt quality: blind human rubric	Auto + human
T04	Artifact running headers, footers, page numbers across a long document	ISO 14289-1 (25 pp) and WTPDF (57 pp) variants; start = inject	Running header, footer and page number tagged P (and page numbers as H6) on every page	`margin_text_is_artifact` on every page; `not_in_role` for headings; `tree_text_coverage` ≥ golden − 0.5 pt; body untouched (collateral diff = 0)	Auto; `actions` per page is the headline metric
T05	Fix a simple data table	WeasyPrint ACR table (corpus) + Declarations table 1; start = inject	Header row tagged TD; one variant has the whole table tagged as P lines (the budget-as-list class, #3746)	Corpus table assertions: `role: Table`, `columns`, `min_full_rows`, `header_row: true`, `must_contain_row`; extension: every TH has `Scope Column` (or `Row` for row headers)	Auto + `sr_match`
T06	Fix a complex table (multi-level headers, spans, Headers/IDs)	WTPDF tables (spans + Headers) and Declarations table 2; start = inject	RowSpan/ColSpan removed and spanned cells split into empty TDs; second header row tagged TD; all `/Headers` and TH `/ID` removed	New `cells` assertion: grid shape equals golden after span resolution; for each TD, the resolved header set (explicit Headers/IDs, else Scope algorithm) equals golden’s; no empty cells that golden lacks	Auto + `sr_match` on 3 sample cells
T07	Lists: nested, with Lbl/LBody	WTPDF / Best Practice Guide lists; start = inject	Nested list flattened into parent (or `L` directly inside `L`); bullets left in LBody text; one list tagged as P lines; ListNumbering missing	Corpus `lists` assertions (`min_items`, `first_item`, `last_item`); extension: nesting depth per item equals golden; every LI has Lbl + LBody; tag-lint `list-l-direct-child`, `li-missing-lbl`, `lbody-bullet-in-text` all clean; ListNumbering matches golden	Auto
T08	Links: Link + OBJR + alternate description	Best Practice Guide pp. 1–10 and `workers/api/src/__tests__/fixtures/link-annots/weasyprint-inline-formatted-links.pdf`; start = inject	Link struct elements removed (link text left in P); annots lose `/StructParent`; `/Contents` stripped; one link’s text split across two Links	`verify.link_coverage` clean: every Link annot has a Link element with OBJR and its visible text; annot `/Contents` present and non-empty; one Link per annot target run; tag-lint clean	Auto
T09	Form fields: tooltips, tab order, Form tags	`TheAccessible-Contact-Form.pdf` (7 fields) + (gap) a 30+ field form with a golden tree	TU stripped; tab order scrambled; widgets not under Form elements; page `/Tabs` missing	Every widget has TU containing its label keywords; each widget has a Form element with OBJR (`verify.widget_coverage` clean); page `/Tabs /S`; new `tab_order` assertion equals golden field sequence	Auto; TU wording human-sampled
T10	Footnotes / notes	Best Practice Guide (2 Note) + (gap) a report with 10+ footnotes and endnotes	Note bodies left as P at the page bottom (in reading order after the page), reference markers plain text	New `notes` assertion: each note is `Note` (or `FENote` for UA-2) with a unique `/ID`; the Note follows the paragraph that references it, or the marker is a Link/Reference to it (both accepted); markers inside the referencing paragraph	Auto + human placement check
T11	Language changes	WTPDF (28 `Lang ja` spans) + ISO 14289-1 (`fr`) + (gap) English doc with Spanish passages; start = inject	Catalog `/Lang` removed; all element-level `/Lang` removed	Catalog Lang equals golden; new `lang` assertion: each golden foreign-language span/element has the same Lang (primary subtag match) and no English text inherits a foreign Lang	Auto + NVDA voice-switch check (human, sampled)
T12	Tag untagged content	Corpus `input.pdf` of AN001 (5 pp, stripped by `tagcorpus strip`) and `services/flyer-tagger/fixtures/beacom-flyer.pdf`	No structure tree at all	Full golden spec for the doc passes (outline, figures, tables, lists, invariants), `tree_text_coverage ≥ 0.97`, veraPDF UA-1 pass	Auto; auto-tag features allowed and recorded, except Acrobat, which tags manually (3.2.3)
T13	Split a merged paragraph	Declarations / AN001; start = inject	Three consecutive P plus the heading above them merged into one P	New `boundaries` assertion: the scope yields exactly the golden elements (H2 + 3 P) with the golden text split points	Auto
T14	Merge fragmented spans	Best Practice Guide; start = inject	One paragraph broken into a P per line; one heading split into two H2; a sentence split into 6 sibling Spans	Scope yields golden element count and text; H2 count in scope = golden; no sibling Spans with identical attributes	Auto
T15	Fix a table of contents	WTPDF / ISO 14289-1 TOC pages; start = inject	TOC tagged as P lines (one variant: as a Table); TOCI Links missing OBJR	New `toc` assertion: TOC with n TOCI (n = golden), each TOCI contains a Link with OBJR whose annot targets the golden destination page; entry text matches golden	Auto
T16	300+ page document stress task	ISO 32000-2 (1023 pp), first 350 pages as one variant, whole file as a second; start = inject	Running headers/footers tagged on every page (T04 at scale) + 20 headings demoted to P, scattered + one continued table with no TH	T04, T02 and T21 assertions over the whole scope; also record `open_s`, `save_s`, peak RSS, crashes	Auto; headline metrics are wall time and responsiveness
T17	Scanned / OCR page	`tools/contrast-sweep/fixtures/scanned.pdf`, `tools/benchmark-cli/calibration/scanned-ocr.pdf`, `tools/pdf-inventory/src/census/fixtures/pdfs/c-image-only-notice.pdf`	Image-only page, no text layer, no tags	Text layer present; character error rate vs a ground-truth transcript ≤ 2% (gap: transcripts); hand-reviewed spec (outline, lists, table if any) passes; coverage ≥ 0.95	Auto; any OCR engine the tool ships is allowed
T18	Math / formula	`services/flyer-tagger/fixtures/Mathematics Review.pdf` (6 pp, untagged) + `tools/benchmark-cli/calibration/text-equations.pdf` (gap: golden tree for both)	Formulas tagged as garbled P, or untagged	Corpus `formulas` assertions: each display formula `role: Formula`, `alt_required`, alt contains spoken-math keywords from the spec; inline formulas stay inside their P; no garbled P left. Alt correctness: human rubric	Auto + human
T19	Group a vector chart into one Figure with Caption	Declarations (the pilot showed its vector chart has no Figure); needs its golden fixed in spec review	Chart’s labels and axis numbers tagged as dozens of P/Span; no Figure; caption is a loose P	Figure anchored at chart bbox with Alt; no tagged text inside the bbox except via the Figure; Caption element adjacent to the Figure with golden text	Auto + human alt rubric
T20	Document setup	Any stripped corpus input (AN001) with a tree added back but document properties removed	No Title, `DisplayDocTitle` false, no catalog Lang, no `MarkInfo /Marked`, no PDF/UA identifier in XMP, no `/Tabs /S`	No veraPDF UA-1 failures for any of those properties (PDF/UA identifier, dc:title + DisplayDocTitle, Marked, catalog Lang, page Tabs); veraPDF result otherwise identical to start	Auto; a baseline “easy task”
T21	Continued table across pages	Declarations tables (fused continuation header) + WTPDF long table; start = inject	Table split into one Table per page; the repeated header row on page 2 tagged as a body row (TD)	Corpus continuation merge passes: one logical table with golden row count; the repeated header is either Artifact or TH, never TD; TD header resolution as T06	Auto
T22	Clean up auto-tag debris	Best Practice Guide pp. 20–30; start = inject	15 empty tags, 4 MCIDs referenced twice (layered text), redundant Div/Span wrappers, one vector rule tagged inside a P	tag-lint `empty-tag`, `whitespace-only-tag`, `duplicate-adjacent-tag`, `decorative-path-in-text` all 0 in scope; corpus `no_duplicate_text`; collateral diff 0	Auto

Each task has three variants (different pages or documents with the same defect recipe). The variant a run uses is assigned by the protocol (3.3).

2.1 Task tiers

Core (T01–T09, T12, T20): the Phase 2 and 3 gates are computed on these.
Extended (T10, T11, T13–T15, T17–T19, T21, T22): in the Phase 3 gate for the task modes that exist by then, and in every gate from Phase 4.
Stress (T16): Phase 1 (engine) and Phase 5 (bulk) gates.

3. Measurement protocol

3.1 Who

Operators: at least two professional remediators per tool, each with ≥ 1 year of paid PDF remediation. At least one operator per tool should be a daily user of that tool (an Acrobat expert runs Acrobat). Nobody who worked on Gevell Tags operates Gevell Tags in a gated run; our own team only runs pilots.
Proficiency check: before timed runs, each operator completes the practice set (one variant of T02, T05 and T08 not used in scoring) in each tool they will run. Anyone who cannot finish a practice task within its cap gets a 30-minute tool orientation and retries once.
Scorers: a different person runs the automated scoring and PAC. Human rubric items are scored by two raters who do not know which tool produced the file (section 3.5).

3.2 Environment and fairness

Same hardware for everything: one Windows 11 machine (Acrobat Pro, PAC, NVDA are Windows-first), 1920×1080, same mouse and keyboard, Chrome for web tools. Gevell Tags runs in the same Chrome. Record OS, browser and every tool version in the run record; pin versions for the duration of a gate.
Same bytes: every tool starts from the identical start.pdf (SHA-256 recorded). Web tools that must upload count upload/processing time inside wall_s because the user waits for it.
Features allowed. Each tool may use the features it ships, inside the tool under test only. No external scripts, no second tool, no pre-made templates built for these files. Features used are recorded, so results can be broken down by “manual” vs “assisted”.
- Acrobat (decision, Larry, 2026-10-01): Acrobat may use every manual tool it ships: the Tags panel, the Reading Order tool, the Table Editor, Set tab order (page properties) and the Accessibility Checker. It may not use Autotag (“Add Tags to Document” / “Autotag Document”), on any task. On T12 Acrobat is therefore restricted to manual tagging.
- Other tools (Gevell Tags, Grackle, Siteimprove, CommonLook): any shipped feature (auto-tag, AI suggestions, OCR, fix-all, templates), recorded in features_used.
Tool-neutral briefs: brief.md says what must be true at the end (“the sidebar is read after the article”), never how to do it in any UI.
Validation: the operator may use any validator built into the tool. External validators (veraPDF, PAC, NVDA) are not available during the run. The operator declares done and saves; that save is what gets scored.
Time cap: per task in task.yaml, set to 3× the pilot median of an Acrobat expert, minimum 10 minutes, maximum 60 (T16: 120).
Unsupported: if a tool cannot do the task at all, the run is dnf: unsupported, scored as a failure at the cap. Report these separately so a competitor’s missing feature is visible rather than hidden in a median.
Breaks between tasks; no session longer than 2 hours.

3.3 Runs and ordering

Each operator runs every task in scope once per variant (three variants), in each tool they cover: 2 operators × 3 variants = 6 runs per (tool, task) minimum.
Tool order is counterbalanced with a Latin square across operators, and a given operator never sees the same variant in two tools. Variant-to-tool assignment rotates so every variant is used by every tool.
Task order within a session is randomised per operator (fixed seed recorded).
Phase 0 pilot: one operator, Acrobat only, one variant per task. This sets the time caps and validates every task package before real runs.

3.4 Aggregation

Per (tool, task): success rate, median wall_s over correct runs, and a penalised median where failed runs count as the cap. Gates use the penalised median, so a fast wrong answer never helps.
Per tool: geometric mean of the per-task penalised-median time ratio against the comparison tool, with a bootstrap 95% CI (10,000 resamples over runs within tasks). “Beats” means the CI’s upper bound is below 1.0.
Per task comparison: a tool beats another on a task when its penalised median is lower and its success rate is not lower.
Paired Wilcoxon signed-rank across tasks on the per-task medians is reported alongside, not used for gates.
errors_introduced, verapdf_ua1, pac, sr_match, actions are reported per task as medians and totals.

3.5 Recording

Harness (tools/tag-corpus → tagcorpus bench): shows the brief, starts the timer when the operator confirms the file is open and interactive, stops it on save, and starts the input counter and screen recording.
Input counting: one OS-level counter process for every tool (mouse down, key down, drag start; scroll separately). Never use Gevell Tags’ own telemetry for comparison numbers, because competitors have none.
Screen recording (OBS) of every run, kept for audit and for hangs_s.
Run record: one JSON line per run appended to tags-bench/results/<gate>/runs.jsonl:

{"run_id": "…", "gate": "phase3", "task": "T06", "variant": "b", "tool": "acrobat",
 "tool_version": "…", "operator": "op2", "session_seq": 7, "start_sha256": "…",
 "output_sha256": "…", "wall_s": 412.3, "actions": 318, "scrolls": 41,
 "features_used": ["table-editor"], "dnf": null, "crashes": 0, "hangs_s": 0,
 "score": {"success": true, "semantic_pass": 1.0, "errors_introduced": 0,
           "errors_detail": {"verapdf_new": 0, "taglint_new": 0, "collateral": 0, "pixel": 0},
           "verapdf_ua1": true, "pac_ua": "pass", "pac_wcag": "pass",
           "sr_match": 0.97, "sr_human": null, "human_quality": null}}

Blinding: before human rating, output files are renamed to their SHA-256, and /Producer, /Creator and XMP tool entries are replaced in a scoring copy (never in the scored original).

4. Scoring pieces: reuse vs build

4.1 Already exists

Piece	Where	Used for
Tree walker with per-MCID text + bbox, RoleMap, OBJR, Form XObjects	`tools/tag-corpus/tagcorpus/walk.py`, `content.py` (PR #3862, unmerged)	Every automated assertion
Spec generator + reviewed spec format	`tagcorpus spec`	Golden specs; task specs are scope-limited copies
Scorer: outline, figures, tables (incl. continuation merge), lists, formulas; invariants (`no_element_spans_columns`, `no_duplicate_text`, `margin_text_is_artifact`, `not_in_role`, `reading_order`, `tree_text_coverage`)	`tagcorpus score`, `invariants.py`	T01–T07, T12–T14, T16, T18, T19, T21, T22
Strip to untagged, pixel-identity verified	`tagcorpus strip`	T12 and T20 start files
Ingest, R2 layout, idempotent storage	`tagcorpus ingest`, `storage.py`	Task package storage
veraPDF PDF/UA-1 JSON wrapper	`services/flyer-tagger/flyer_tagger/verify.py` `verapdf_check` (image with veraPDF: `deploy/Dockerfile.test`)	`verapdf_ua1`, `verapdf_new`
Pixel diff	`verify.py` `pixel_diff`	`errors_detail.pixel`
Link and widget coverage	`verify.py` `link_coverage`, `widget_coverage`	T08, T09
Reading-order text helpers	`verify.py` `text_in_structure_order`, `assert_order`	T01 cross-check
Tag-tree linter (Matterhorn-mapped rules: list nesting, missing alt, Lbl, empty/duplicate tags, heading skip, decorative paths)	`tools/tag-lint`	`taglint_new`, T07, T22
PDF/UA-1 clause → fix-class map	`tools/pdf-inventory/src/census/clause-class-map.json`	Grouping veraPDF deltas in reports
Screen-reader serializer from the struct tree (TS)	`workers/api/src/services/pdf-screen-reader-serializer.ts`	Reference for the `sr_match` transcript format
Struct-tree edit ops	`services/flyer-tagger/flyer_tagger/structedit.py`, `altedit.py`	Building blocks for `inject`
Object-level PDF inspection	`tools/pdfdump/pdfdump.py`	Debugging task packages

4.2 Must build

Merge PR #3862 (or land tools/tag-corpus another way). Everything below extends it.
tagcorpus inject: deterministic defect recipes (reorder-kids, retag, drop-alt, figure-to-artifact and reverse, artifact-to-tagged, merge-elements, split-elements, flatten-list, drop-table-attrs, split-spans, untag-links, drop-tu, scramble-tabs, drop-lang, toc-to-p, add-debris), seeded and recorded in inject.json. Must keep content streams byte-identical except MCID markers, verified with pixel_diff. Reuse structedit.py ops where they fit.
Spec extensions in score.py: contiguous, boundaries, cells (span-aware grid + resolved header sets, following the PDF/UA-1 / PDF 2.0 table header algorithm), notes, lang, toc, tab_order, tu, list_depth / lbl_lbody, th_scope, alt_quality_lint (filename, “image of”, empty, too long).
Scope-limited scoring + collateral diff (tagcorpus bench score): applies only the task’s assertions, then diffs the output tree against start.pdf outside the scope (wrapper-transparent, same text normalisation as the scorer). Any change counts toward errors_introduced.
veraPDF and tag-lint deltas: run both on start and output, report only new failures, grouped by clause class.
Python screen-reader transcript built on the corpus walker, matching the TS serializer’s announcement format (heading level, list n of m, table row/column with headers, link, figure alt, formula alt, form field label/TU, language switch markers). expected-sr.txt is generated from the golden; sr_match is a normalised edit distance over the scope.
Bench harness (tagcorpus bench run): brief display, timer, variant assignment from the Latin square, launch of the input counter and OBS, and the JSONL writer.
Input counter: a small Windows background process (low-level mouse and keyboard hooks) that writes counts, not keystroke content, to the run record. No keylogging: only counts and timestamps.
PAC runner: PAC 2024 is a Windows GUI. If the current build offers no command line, the scorer runs PAC after the session and enters the result through a small form that writes into runs.jsonl. Revisit if axes4 ships a CLI.
Report generator (tagcorpus bench report): per-task and per-tool tables, penalised medians, bootstrap CIs, gate verdict, published as an admin-docs page.
Human rating form: blind rubric capture (alt text, note placement, SR listen) with two raters and an adjudication column.

4.3 Scoring each task end to end

start.pdf ──(operator, tool)──▶ output.pdf
output.pdf ─▶ tagcorpus bench score --task T06 --variant b
   ├─ task assertions (task-spec.yaml)               → success, semantic_pass
   ├─ collateral diff vs start.pdf outside scope     → errors_detail.collateral
   ├─ pixel_diff(start, output)                      → errors_detail.pixel
   ├─ veraPDF UA-1 on start and output               → verapdf_ua1, verapdf_new
   ├─ tag-lint on start and output                   → taglint_new
   └─ SR transcript of scope vs expected-sr.txt      → sr_match
PAC 2024 (scorer, after session)                     → pac_ua, pac_wcag
Blind raters (sampled / judgment tasks)              → human_quality, sr_human

4.3a Build status (core tasks, 2026-09-30)

Items 2–6 of 4.2 are built for the core tasks (T01–T09, T12, T20): tagcorpus inject, the scope-limited bench score with collateral diff, veraPDF and tag-lint deltas, the Python screen-reader transcript and sr_match, and the 33 validated task packages. How to build, score, upload and run the Acrobat pilot, plus where the variants deviate from the table in section 2: benchmark-packages.md. Items 7–11 (harness, input counter, PAC form, report, rating form) are not built.

4.4 Human checks

Required for T03, T18, T19 alt text; T10 placement; T09 TU wording (sampled).
NVDA + Acrobat Reader listen for 1 in 4 runs of T01, T05, T06, T11, chosen at random, against expected-sr.txt. NVDA’s speech viewer log is saved with the run.
Rubric: 0 = wrong or misleading, 1 = acceptable but weak, 2 = what a careful remediator would write. A run with any 0 on a required human item is not correct.
Raters disagree by 2 points → a third rater decides.

5. Phase gates

All gates use the same pinned task packages. Gates are run by operators who did not build the phase being gated.

Phase	Gate	Threshold
0 Discovery	Benchmark is runnable	All 22 task packages built with 3 variants; every `task-spec.yaml` passes against its own `golden.pdf` and fails against its `start.pdf`; Acrobat pilot baseline recorded; time caps set
1 Engine	Engine can express and preserve every task	(a) open + save with no edits is lossless for every start and golden file: pixel-identical, identical tree after normalisation, no new veraPDF failures; (b) for every task, a scripted op sequence turns `start.pdf` into an output that passes the task spec with `errors_introduced = 0`; (c) T16 whole file opens in < 10 s and saves in < 30 s on the benchmark machine, peak memory < 2 GB in the browser
2 Workspace shell	Acrobat Tags-panel parity	Raw-tree mode only (no task modes): success rate ≥ Acrobat’s on every core task; `errors_introduced` total ≤ Acrobat’s; penalised median time ≤ 1.25× Acrobat on each core task; zero crashes
3 Task modes	Beats Acrobat on median time per task	Beats Acrobat (lower penalised median, success not lower) on ≥ 80% of core tasks and on every core task with a shipped task mode; aggregate time ratio vs Acrobat has 95% CI upper bound < 0.8; median `actions` ≤ Acrobat on ≥ 80% of core tasks; `errors_introduced` median 0 on every task
4 Live validation	No false passes; beats all three competitors on quality	Tag-studio’s validator reports clean on zero outputs that fail veraPDF UA-1 or the task spec (false pass rate 0 across all runs); veraPDF UA-1 pass and `semantic_pass` ≥ the best competitor on every task; beats Acrobat, Grackle and Siteimprove on penalised median time on ≥ 80% of all 22 tasks
5 AI + bulk	Scales and stays correct	T04 and T16 at least 3× faster (penalised median) than the best competitor; AI-assisted runs have `errors_introduced` no higher than manual Gevell Tags runs; accepted-without-edit rate of AI suggestions reported per task
Retire old editors	Parity with what they replace	Tag-studio ≥ the Gevell PDF tag editor (`packages/tag-editor-ui`) and `apps/tag-editor` on success and penalised median time for every task those editors can do
Launch claim	”Best-in-class” copy is allowed	Phase 4 gate holds on the most recent competitor versions (re-run within 90 days of the claim); no task where Gevell Tags is > 1.2× slower than any competitor or has a lower success rate

6. Corpus gaps

Documents we need before the related tasks can be built. Public or synthetic first; customer files only under the public-corpus exception with written approval.

Need	For	Suggested source
Multi-column newsletter with sidebar and pull quote, correctly tagged	T01	Synthetic (WeasyPrint/InDesign export, then hand-verified tags) or a public university/government newsletter retagged by an expert
Page with decorative images (rules, backgrounds, ornaments) next to informative ones	T03	Synthetic flyer
Form with 30+ fields, groups (radio, checkbox), golden tree and tab order	T09	Public government form retagged, or `docx2acroform` output hand-verified
Report with 10+ footnotes and endnotes	T10	Public government report or synthetic
English document with Spanish (and one RTL) passages	T11	Synthetic; a public bilingual notice
Golden trees for `Mathematics Review.pdf` and `text-equations.pdf`	T18	Hand-tagged by an expert; spoken-math Alt written once and reviewed
Ground-truth transcripts for the scanned fixtures	T17	Manual transcription (3 pages)
Financial schedule with row headers (budget-as-list class)	T05 variant	Public university budget PDF under the corpus `financial-schedule` type
Deeply nested list (3 levels, mixed ordered/unordered)	T07 variant	Check WTPDF lists first; else synthetic
A second 300+ page tagged document that is not an ISO standard	T16 variant	Public course catalog or government manual with a good tree

tools/pdfpolish/output/* files are Gevell outputs, not golden trees, so they are not benchmark sources.

7. Open questions

Siteimprove and Grackle capabilities for editing existing PDFs’ tag trees must be confirmed in the competitor teardown. Tasks a tool cannot do are reported as unsupported, never dropped.
~~Whether Acrobat’s “Autotag” is allowed as the first step of T12.~~ Decided 2026-10-01 (Larry): no. Acrobat may use every manual tool it ships but never Autotag, and tags T12 manually (rule 3.2.3). Other tools keep any shipped feature, broken down by features_used.
Operator recruiting and compensation for competitor experts is not yet budgeted.