Skip to content
AnglinAI Docs

Benchmark task packages: build, score, upload, pilot

Gevell Tags Phase 2, workstream P0 (issue #3959, epic #3866). This is the runbook for the core-task packages (T01–T09, T12, T20) defined in benchmark.md. The tooling lives in tools/tag-corpus (tagcorpus inject, tagcorpus bench build|score|sr) on top of the golden corpus tooling from PR #3862.

1. What exists

PieceWhereNotes
Task definitions (11 core tasks × 3 variants)tools/tag-corpus/bench-tasks/T*.yamlbrief, golden source, seed, scope, inject recipe, assertion families
tagcorpus injecttagcorpus/bench/inject.py, structedit.py19 seeded recipes (incl. strip_tags); touches only the structure tree, annotation entries, document properties and marked-content operators; ParentTree rebuilt from the tree
Synthetic goldenstagcorpus/bench/goldens.py, synth.pynewsletter (T01, T12c), decorative flyer (T03), 33-widget form (T09), nested lists (T07), wrapped links (T08). Embedded Liberation Sans; every one passes veraPDF PDF/UA-1
Task-spec buildertagcorpus/bench/taskspec.pythe expected answer is always the golden’s own structure, limited to the scope
Task assertionstagcorpus/bench/assertions.pythe corpus scorer limited to the scope, plus block_order, contiguous, th_scope, cells, list_structure, links, widgets, doc_props, alt_lint, band_text
Collateral difftagcorpus/bench/collateral.pyeverything the task did not ask to change, compared with the golden: blocks outside the scope (in order), blocks inside it that are not task elements (as a multiset), table and list structure, annotations, page /Tabs, document properties
Screen-reader transcripttagcorpus/bench/sr.pyPython counterpart of pdf-screen-reader-serializer.ts; sr_match is word-level similarity
Oraclestagcorpus/bench/oracles.pyveraPDF, qpdf, tag-lint, PDFium pixel diff; all fail closed
bench scoretagcorpus/bench/score_run.pythe run record’s score object (benchmark.md 3.5)

Not built yet (benchmark.md 4.2 items 7–11): the bench run harness (brief, timer, Latin square, OBS and input-counter launch, JSONL writer), the Windows input counter, the PAC entry form, the report generator and the human rating form. None is needed for the Phase 0 pilot; all are needed for the gated Phase 2 runs.

2. Build and validate the packages

Terminal window
cd tools/tag-corpus
python -m tagcorpus bench build                     # all core tasks -> .work/tags-bench/tasks/
python -m tagcorpus bench build --task T06 --variant b

Needs veraPDF (brew install verapdf), qpdf, pikepdf, pypdfium2, fontTools and the repo’s tools/tag-lint. No network, no model calls, no cost.

Each variant directory holds golden.pdf, start.pdf, task-spec.yaml, expected-sr.txt, inject.json (seed, every change, SHA-256 of golden and start), brief.md and validation.json. A variant counts as built only when all of these hold, and the build exits 1 otherwise:

  1. start.pdf renders pixel-identical to golden.pdf (PDFium, every page);
  2. the task spec passes on golden.pdf and fails on start.pdf;
  3. the collateral diff between golden and start outside the scope is 0;
  4. bench score treats golden.pdf as a correct run (errors_introduced 0, sr_match 1.0) and start.pdf as not a success.

This is the “every task-spec.yaml passes against its own golden.pdf and fails against its start.pdf” part of the Phase 0 gate row.

Variants

Taskabc
T01 reading ordersynthetic newsletter anewsletter bnewsletter c
T02 headingsBest Practice Guide pp. 5–20WTPDF pp. 4–20ISO 14289-1 pp. 8–20
T03 alt / decorativesynthetic flyer aflyer bflyer c
T04 running textISO 14289-1 (25 pp)Best Practice Guide (72 pp)AN001 (5 pp)
T05 simple tableBPG p. 10 (header row as TD)ISO 14289-1 p. 10 (header row as TD)BPG p. 5 (table as P lines)
T06 complex tableWTPDF Table B.1 p. 51WTPDF p. 52BPG p. 59
T07 listsBPG pp. 60–72synthetic lists bsynthetic lists c
T08 linksDeclarations pp. 1–6WTPDF pp. 1–4synthetic links c
T09 formsTheAccessible-Contact-Form (7 fields)synthetic form b (33 widgets)synthetic form c
T12 untaggedAN001Declarationssynthetic newsletter b
T20 document setupAN001DeclarationsBest Practice Guide

Deviations from the table in benchmark.md section 2, and why:

  • Synthetic goldens fill the section 6 gaps for T01 (newsletter), T03 (decorative images), T07 (3-level nested list), T08 (links that wrap, so a link can be split) and T09 (30+ field form). They are built from the same description that paints them, so the tree is right by construction; a person still looks at each once before a gated run (section 4).
  • T04 uses the Best Practice Guide and AN001 instead of WTPDF, whose footer is already tagged on one page of the golden.
  • T05 and T06 use tables whose header cells carry Scope or real RowSpan/ColSpan, so removing them is a real defect. Declarations table 2 has Scope on every header, which makes dropping its Headers harmless.
  • T08 untags 4 links (3 on the synthetic page) and splits one more; the split needs a link whose text is two marked-content runs, which only the Declarations TOC, one WTPDF TOC entry and the synthetic page have.
  • T12 requires “veraPDF not worse than the golden” instead of “veraPDF pass”: the AN001 golden itself fails rule 7.21.4.2 (a font program issue no tagging can fix).
  • T20 puts page /Tabs in the document scope, with the other document properties.

Scoring rules worth knowing

  • Collateral covers the whole document except the task elements (the elements the inject record changed, and what lies inside them). Inside the scope, other blocks keep their order (and each element the task changed in place keeps its position: a rebuilt table or list left elsewhere in the tree is collateral), except in a reading-order task (one with block_order assertions, T01), where they may be reordered but not retagged, merged, split or re-alted. Block signatures include role, text, alt, effective language (own, inherited or catalog; the catalog is left out when the document is in scope), structural ancestors, spans and Headers; tables and lists also get a structure signature (grid, resolved headers, empty cells; item depth, Lbl/LBody, ListNumbering). T12 compares outside the scope only (collateral: outside_only): its start has no tree.

  • Pixels: rendered at 150 dpi; a page counts as changed when more than 8 pixels differ (absolute, so a deleted ornament or word always shows).

  • Decorative graphics (T03) pass only when no tagged content paints them, under any role; junk alt (file names, “image of”) fails on any element.

  • Tables (T06): a data cell passes when its header set matches either the golden’s explicit Headers or what the golden’s Scope attributes resolve to; the brief allows both methods.

  • Alt keywords are not machine-checked on T03 (they come from the hidden golden alt); alt quality is the blind human rubric.

  • Block order fails when two golden blocks are merged into one.

  • errors_introduced = new veraPDF failed checks + new tag-lint errors/warnings + collateral changes + changed pages. “New” means beyond the per-rule worse of start and golden, so reproducing the golden exactly introduces nothing.

  • A run is correct when every task assertion and oracle assertion passes and errors_introduced is 0.

  • band_text (T02, T04) works on whole visual lines in the top/bottom 12% of the page; the golden’s own tagged lines there are allowed. Both tasks require the running text to be out of the tree, not only out of headings.

3. Score a run

Terminal window
python -m tagcorpus bench score .work/tags-bench/tasks/T06/variants/b output.pdf --json score.json

Exit 0 = correct, 1 = not correct, 2 = no verdict. Exit 2 means a package file or an oracle was missing or gave an answer we could not read (veraPDF absent, crashing or using another profile, qpdf absent, tag-lint not importable, a page PDFium could not render), or scoring crashed. It is never a pass, and never recorded as a failed run. An output qpdf cannot read is a failed run (exit 1), not a scoring error.

python -m tagcorpus bench sr file.pdf --pages 1-3 prints the transcript.

4. Upload to R2 (manual, explicit)

The builder only writes locally. Packages go to the dev-assets bucket, never to git (benchmark.md 1.2). The two “sponsored” ISO standards are licensed copies: keep them in this bucket for internal benchmarking only.

Terminal window
cd tools/tag-corpus
python -m tagcorpus bench build                     # must end "all valid"
rclone copy .work/tags-bench/tasks RCL2DEV:accessible-dev-assets/tags-bench/tasks --s3-no-check-bucket
rclone check .work/tags-bench/tasks RCL2DEV:accessible-dev-assets/tags-bench/tasks --one-way

--s3-no-check-bucket is required on writes (R2 answers the implicit CreateBucket with a 403). Re-uploading after a rebuild replaces files; the inject.json SHA-256s say which bytes a run started from. Once runs have been recorded against a package, do not rebuild it in place: bump the seed or add a variant instead, so start_sha256 in old run records stays meaningful.

5. Phase 0 pilot (needs a human operator)

benchmark.md 3.3: one operator, Acrobat only, one variant per task. It sets the time caps and shakes out the packages. Exactly what to do:

  1. Machine. The Windows 11 benchmark machine at 1920×1080. Record the Windows build, Acrobat Pro version (Help > About) and the date. Install OBS; NVDA and PAC are not used during the run.
  2. Files. Copy tags-bench/tasks/ from R2 (or a zip from the builder). For each core task use variant a. Open nothing else.
  3. Rules (benchmark.md 3.2.3). Use only Acrobat’s manual tools: the Tags panel, the Reading Order tool, the Table Editor, Set tab order (page properties) and the Accessibility Checker. Never Autotag (“Add Tags to Document” / “Autotag Document”), on any task; T12 is tagged by hand. No other application, no scripts, no plug-ins.
  4. Each task. Read brief.md (and its page note). Start OBS, open start.pdf, and start the stopwatch when the file is open and interactive. Do the task. Save with File > Save As to <task>-a-output.pdf. Stop the stopwatch when the save completes. Note the seconds, every tool or feature used, any crash or hang over 2 seconds, and whether you gave up (dnf: unsupported if Acrobat cannot do it, dnf: timeout past 60 minutes). Take a break between tasks; no session longer than 2 hours.
  5. Hand over the 11 output PDFs, the notes and the recordings to the scorer (a different person). The scorer runs bench score on each and writes one JSON line per run to tags-bench/results/phase0-pilot/runs.jsonl in the section 3.5 format (gate: "phase0-pilot", tool: "acrobat", operator, wall_s, features_used, dnf, crashes, hangs_s, the score object, and start_sha256 / output_sha256). actions and scrolls are null until the input counter exists. Run PAC 2024 on each output and enter pac_ua / pac_wcag by hand.
  6. Time caps. With one pilot run per task, the cap is clamp(3 × wall_s / 60, 10, 60) minutes (T16: 120). Write it into time_cap_min in each bench-tasks/T*.yaml, rebuild, and re-upload task.yaml (the PDFs do not change).
  7. Triage. Any run that scored wrong in a way the operator disputes (an ambiguous brief, an assertion that rejects a reasonable fix) gets an issue before real runs; fix the package or the brief and record the change.

6. What the Phase 0 gate row still needs (core tasks)

  • Upload the validated packages to R2 (section 4).
  • A person checks each synthetic golden once against its render and the task specs against their goldens, then sets reviewed_by / reviewed_at in the task definitions (the corpus review rule).
  • The Acrobat pilot baseline (section 5) and the time caps from it.
  • The extended tasks (T10, T11, T13–T19, T21, T22) are outside this workstream; the row as written covers all 22.