Compatibility test set

How to add a real vendor accessibility-report entry to the test set that backs Gevell Tags’ “Compatibility mode” (task-redesign §1 decision 6, §8) and its pinned-version chip. The harness code lives in tools/tag-compat (issue #4181).

As of this runbook’s last update there are zero real entries — this PR is the infrastructure, not the data. See the follow-up issue (linked from #4181) for exactly which PDFs to run first.

Why this exists

Gevell Tags’ Analyze screen shows an agreement line in Acrobat mode: “Agrees with the Acrobat Pro … Full Check on X of Y rules.” That claim has to be backed by a real report, not an assumption about what Acrobat would say. This test set is where those real reports live, and tools/tag-compat is what turns them into the agreement number and the pinned version string shown in the UI.

Never hand-write or guess a vendor verdict. Every verdict in the test set must come from the vendor’s own tool, actually run.

Layout

tools/tag-compat/test-set/
  <entry-id>/
    metadata.json
    acrobat-full-check.html   (the raw export, as Acrobat wrote it)

metadata.json:

{
  "id": "headings-nesting-01",
  "pdf": "headings-nesting-01.pdf",
  "description": "Multi-level heading nesting, one skipped level (H2 -> H4)",
  "vendorRuns": {
    "acrobat-pro": {
      "product": "Acrobat Pro",
      "version": "26.001.20xxx",
      "settings": "Full Check, default rule set, accessibility permissions flag checked",
      "dateRun": "2026-10-20",
      "runBy": "[email protected]",
      "reportFiles": ["acrobat-full-check.html"]
    }
  }
}

• id — a short, stable, kebab-case identifier. Never reused for a different PDF.
• pdf — the filename (see Storage below for where the actual bytes live).
• vendorRuns — one key per compatibility mode actually tested (acrobat-pro, pac-2024, commonlook); omit modes you have not run.
• Every field in a vendor run must reflect what you actually did — the exact product version string (Help > About in Acrobat), the settings you chose, the real date, and your name/email.

src/test-set.ts validates every metadata.json against TestSetEntryMetadataSchema (src/types.ts) — a malformed file fails loudly (throws) rather than being silently skipped.

Storage

Following the tools/tag-corpus convention (#3862): the PDF and the raw vendor report file(s) are stored in R2 (RCL2DEV:accessible-dev-assets/tag-compat/<entry-id>/), not committed to git — they can be large and are not meaningful to diff. metadata.json is always committed; it is the only in-git record of what the test set contains. A small, text-diffable report may be committed directly if you judge it useful for review, but default to R2.

Running Acrobat Full Check and exporting the report

1. Open the PDF in Acrobat Pro.
2. Tools > Accessibility > Full Check (or Accessibility Checker panel if already in the workspace).
3. In the Options dialog, use a consistent rule set — the default “Accessibility Permissions, Image-only PDF, Tagged PDF, … ” full set of checks. Do not narrow the rule set unless the test-set entry’s settings field says so.
4. Run the check. In the Accessibility Checker panel, right-click the report root and choose Save Checker Report… (or, in the older UI, use Options > Create Accessibility Report) to export the HTML report.
5. Note the exact Acrobat version: Help > About Adobe Acrobat Pro.
6. Record product, version, settings, dateRun (today, ISO 8601), and runBy in metadata.json.
7. Upload the PDF and the exported HTML report to RCL2DEV:accessible-dev-assets/tag-compat/<entry-id>/ (see tools/tag-corpus’s R2 docs for the rclone remote setup if you have not used it before).
8. Commit metadata.json only.

PAC 2024: not supported yet (src/parsers/pac.ts is a stub) — its export format has not been confirmed. If you run PAC, save the raw export anyway (future-proofing) but do not expect the harness to parse it until the parser is implemented.

Computing agreement

npm run agreement -w @accessible-pdf/tag-compat

This parses every acrobat-pro vendor run in the test set, compares against Gevell’s own verdicts (see Gevell verdicts below), and prints a markdown summary. It exits non-zero only when a baseline is committed in tools/tag-compat/baseline.json and the observed agreement drops below it. With zero vendor reports (today), it always passes and prints “No vendor reports are in the test set yet.”

Gevell verdicts

The CLI reads an optional gevell-verdicts.json next to each entry’s metadata.json — {"engine:NoStructTreeRoot": "passed", ...} — mapping gevellCheckId (the same ids used in src/rules-table.ts) to a verdict. Generating that file from a real engine/scorer run on the test-set PDF is not built yet — this PR ships the comparison and CI machinery, not the engine-run wiring. Track that as a follow-up once the first real vendor report exists (no point wiring it against zero PDFs). Until then, an entry with a vendor run but no gevell-verdicts.json simply reports no comparable rules for Gevell’s side, which is reported accurately rather than silently assumed.

Re-running when a vendor updates its checker

Acrobat (or PAC/CommonLook) can change rule names, add/remove rules, or change wording between releases. When you upgrade:

1. Re-run the full test set with the new version.
2. Compare the new report’s rule names against src/rules-table.ts’s generic entry for that mode.
   • If nothing changed: just update each entry’s metadata.json version and re-export.
   • If a rule was renamed, added, or removed: fork a new RulesTableVersion in rules-table.ts keyed to that exact version (never edit the old one in place — it may still back older comparisons). Note the change in the PR description so it shows up in that release’s notes.
3. Re-run npm run agreement -w @accessible-pdf/tag-compat and consider whether baseline.json needs updating.
4. Note the rule change in the next release notes (task-redesign §1 decision 6: “We re-run the test set whenever the vendor changes its checker”).

Adding a new compatibility mode

1. Add the mode to CompatibilityModeSchema in src/types.ts.
2. Add a RulesTableVersion for it in src/rules-table.ts.
3. Implement (or stub, like src/parsers/pac.ts) its report parser.
4. Register it in src/cli.ts’s PARSERS map — runAgreement already loops over every mode present in each entry’s vendorRuns, so a registered parser is picked up with no other control-flow changes.
5. Update this runbook’s Storage/Running sections with that tool’s export steps.