Automated Document Scanning — Plan

Internal. Compiled 31 August 2026. Revised to make structural-vs-content classification the central output. Supersedes Phase 3.1 of the build brief, which was too narrow.

What this is

Point it at a domain, an LMS, a folder, or an inbox. It finds every document, classifies each one, and returns a prioritized worklist that answers the only question that matters operationally:

Which of these can we fix without asking a single author for anything?

Field experience says up to half of failing PDFs have structural-only failures — the content is fine, the tag tree is broken or missing. Those are fixable by tag-tree operations alone, with no new authored text and no change to the rendered page. The pipeline already makes this distinction at ingest. The scanner needs to make it at inventory scale, which is where it becomes a sales argument rather than a processing detail.

Why this is the most valuable thing to build next

It removes the author bottleneck, which is the real constraint. In higher ed the blocker isn’t budget, it’s organizational: getting two hundred faculty and staff to write alt text is a project that never finishes. Telling an institution that half their backlog can be cleared tonight without contacting anyone is a fundamentally different conversation from “here’s your list.”
It is the sales tool. A university with 800 PDFs faces a five-to-six figure bill at market rates of roughly $5–8 per page. Ohio State budgeted $20M. Nobody knows their own number until someone counts, and the number gets dramatically better once the structural half is carved out.
It feeds the remediation pipeline — the structural half to our automated path, the content half to authors or a partner under revenue share.
It powers the research study, which is the credibility play.
It’s the workflow Acquia already sells (crawl the domain, find the PDFs) and nobody does to depth. Their scanner flags issues. We validate to PDF/UA-1 and tell you which ones we can fix ourselves.

Part 1 — Discovery

Already our strongest asset and completely unmarketed.

Between the crawler, the public API, webhooks, MCP, email intake, shared folder, and the Canvas and Moodle integrations, we have more ingestion paths than anyone in the competitive set. Most competitors have exactly one: crawl a public domain.

Source	Status	Notes
Web crawl (link extraction)	To build	Extract `.pdf`, `.docx`, `.xlsx`, `.pptx` during page audit
sitemap.xml	Partial	Crawler reads it for pages; extend to documents
Search-engine enumeration	To build	`site:x.edu filetype:pdf` finds unlinked documents the crawler can never reach. Often a large share of a real corpus.
Canvas	Built, untested	Course files. Nobody else audits documents inside the LMS.
Moodle	Built, untested	Significant in community colleges and internationally
Shared folder	Built	Batch drop
Email intake	Built	Forward-to-scan
Public API	Built	Customer-driven bulk submission
Webhooks	Built	Continuous discovery on publish
MCP	Built	An agent can drive the whole workflow
Authenticated crawl	Phase 5	Documents behind a login. Nobody audits these.

Deduplicate by content hash, not URL. The same PDF sits at a dozen URLs across a university site. Charging twice for it, or reporting it twice, destroys trust in the inventory count.

Track versions. When a document at a known URL changes hash, that’s a new document, and on a scheduled scan it surfaces as “changed since last audit.”

Part 2 — Tiered triage

Most documents can be classified without a full validation run. Running veraPDF plus vision analysis across 10,000 documents is economically absurd. Running it on the subset that could plausibly pass is not.

Tier 0 — Structural probe. Milliseconds. No AI, no veraPDF.

Read the catalog and answer:

Is there a StructTreeRoot at all? (None → untagged, but see the classification note below: untagged is often the easiest structural fix, not the hardest)
/MarkInfo /Marked true?
Is /Lang set at document level?
Is there a document title in metadata, and is DisplayDocTitle set in ViewerPreferences?
Is there extractable text, or is this an image-only scan?
Is it encrypted or permission-locked?
Page count, file size, /Producer string, creation and modification dates

This dispositions the majority of a typical corpus for essentially zero cost. Producer strings are diagnostic: documents from a known-bad export path can be batched and fixed at the source rather than one at a time.

Output: a preliminary class plus hopeless / promising / likely-already-fine.

Tier 1 — veraPDF PDF/UA-1 validation. Seconds. CPU-bound, no AI.

Runs on anything Tier 0 flags as worth validating. Machine-verifiable conditions only, deterministic, cheap.

The failed-clause list is the input to classification — see Part 3.

Tier 2 — Judgment items. Expensive. Gated.

Alt-text quality, whether reading order makes sense, table semantics, heading logic. Vision and language model work. Only runs on a customer-selected priority subset or post-remediation QA. Never by default across a corpus.

Tier 3 — Conversion. Separately priced, explicit action only.

Never triggered by a scan. A document audit is cheap; a conversion is not, and blurring them destroys the margin on the inventory product.

Part 3 — Structural vs content classification

This is the core of the product. Build it as a first-class, testable component.

The test

Can this be fixed by tag-tree operations alone — with no new human-authored text, and no change to the rendered appearance of the page?

Yes → structural. No → content.

Four classes

Class A — Structural only. Fixable by building or repairing the tag tree. No author involvement. Automatable.

No StructTreeRoot / untagged, but with extractable text and clean layout
/MarkInfo /Marked absent or false
Document /Lang missing
Title metadata missing, or DisplayDocTitle not set
Headings mis-nested (H1 → H3) where the visual hierarchy is unambiguous
Data tables tagged as layout, or missing Scope / Headers
List content not tagged as L / LI / LBody
Decorative images, running heads, folios not marked as artifacts
Link annotations absent from the tag tree, missing /StructParent
Orphaned or malformed tag structure — the ParentTree/MCID class of bug

Class B — Content required. Needs new human-authored text or a content change. Author or remediator involvement.

Missing /Alt on meaningful images (someone must write it)
Missing form field /TU tooltips and labels
Complex tables needing a summary
Non-descriptive link text
Language changes within the document (requires knowing which language)
Charts and diagrams with no text equivalent
Colour contrast failures in the rendered content

Class C — Source or rebuild required. Cannot be fixed in place.

Image-only scans with no text layer (needs OCR, then re-tagging)
Content that only works as a web page → reflow pipeline
Documents so structurally damaged that rebuilding beats repairing

Class D — Already conformant. Passes Tier 1 with no judgment flags. Charge nothing.

A document is usually a mix

Report per-document class as the worst class present, and also report the counts within it. “Class B — 1 missing alt text, plus 14 structural issues” tells the customer they need one sentence from an author and we handle everything else. That’s a completely different task from “this needs rebuilding.”

Surface the near-miss set explicitly: documents that would be Class A but for a handful of Class B items. These are the highest-leverage documents in any corpus — a few minutes of author time each unlocks a fully automated fix.

Building the classifier

Build a clause-to-class mapping table from the veraPDF PDF/UA-1 failed-clause identifiers and the Matterhorn Protocol checkpoints. Matterhorn already splits checkpoints into machine-checkable and human-verifiable; our split is different but adjacent, and the mapping is the core IP here.

Requirements:

Versioned, checked into the repo, reviewable by a human as a table
Every clause maps to exactly one class with a written rationale
Unmapped clauses fail loudly rather than defaulting to a class
Regression-tested against the fixture corpus (Part 6)

Do not infer the class with a model. It is a lookup, it must be deterministic, and it has to be defensible when a customer asks why a document was classified the way it was.

Honest caveat to carry in the product

Class A means machine-fixable without new content. It does not mean the result is verified accessible. Reading order still has to make sense, headings still have to be logical, and existing alt text may still be useless. Those are human-judgment checks that happen after remediation, not before. Say this in the report. Overstating what automated structural repair achieves is exactly the claim category the FTC acted on.

Part 4 — The disposition engine

Each document gets a recommended action derived from its class:

Class	Disposition	Author involvement	Cost signal
D	Already accessible	None	$0 — and charge nothing
—	Delete or retire	Owner sign-off only	$0 — cheapest remediation there is
A	Automated tag-tree repair	None	Near-zero marginal; bulk priced
A+B (near miss)	Automated repair + short author task	Minutes	Low
B	Content remediation	Substantial	Market rate, or route to partner
C (image-only)	OCR then re-tag	Review only	Medium
C (better as HTML)	Reflow rebuild	Review only	Estimate
Any	Expert required	Specialist	Refer to partner

Attach a dollar estimate per disposition using market rates with assumptions stated openly. Present it as “what this would cost at prevailing rates”, never as a quote.

Prioritize the worklist by legal exposure first, not severity score: forms, public notices, required disclosures, anything a member of the public must read to receive a service. Then traffic, if we can get it.

Part 5 — The inventory report

Executive summary page, in this order:

N documents found across M sources
X% already conformant, no action, no charge
Y% fixable with no author involvement ← the headline
Z% recommended for deletion or retirement
Remaining W% requiring content work, with the near-miss subset called out
Estimated exposure $A, of which $B is avoidable

The second and third lines are what get this document forwarded to a vice president. “Half of your backlog needs nobody’s time but ours” is a stronger opening than any severity chart.

Detail sections: per-document class and disposition, near-miss list, producer-level patterns (“all 340 documents from this export path share one fixable setting”), and the delete candidates with the reasoning for each.

Delta on rescan: new, changed, resolved. This is what makes it a subscription rather than a one-off.

Part 6 — Operational constraints

Never retain source documents. Fetch, probe, validate, discard. Keep findings and hashes only. Keeps storage cheap and keeps us clear of copyright and PII problems when scanning an institution’s whole library.
Very large documents. We have hit trouble with large-document conversion, and a prospect has asked about multi-thousand-page files. Cap Tier 1 at a configurable page count; above it, Tier 0 only, reported as “too large for automated validation, manual scoping required.” One 4,000-page document must not stall a 10,000-document queue.
Encrypted and permission-locked files: detect, report, never bypass. Restrictive permissions are themselves an accessibility problem worth flagging.
Rate limiting and robots.txt on crawl-based discovery. We are hitting other people’s servers at volume.
Failure taxonomy: 404, auth wall, timeout, malformed, oversized. Each a distinct reportable state. A customer with 200 broken document links wants to know that.

Part 7 — Testing

There is currently no good way to test this, and that includes the Canvas and Moodle integrations sitting unvalidated for months.

Fixture corpus. Versioned test set with known-correct expected outcomes:

veraPDF’s own test corpus and the PDF Association’s Matterhorn-derived files, for validator correctness
100+ real .gov and .edu documents hand-classified by us, covering every class and every clause in the mapping table — untagged-but-clean, tagged-but-broken, image-only, encrypted, missing alt text only, complex tables, forms, and a spread of page counts
Near-miss cases specifically — Class A but for one or two Class B items — since those drive the highest-value recommendation and are the easiest to get wrong
Golden-file tests asserting Tier 0 verdict, Tier 1 clause list, assigned class, and final disposition

LMS test environments. Both are self-hostable, which solves the testing problem without needing a customer. Moodle ships official Docker images and has a public demo site. Canvas has an open-source edition that runs in Docker plus free instructor accounts for smoke testing. Stand both up, load the fixture corpus as course files, run the integrations against them. Do this before advertising either integration.

Load test the queue at 10,000 documents before offering inventory scans at that scale.

Part 8 — Pricing

“One credit per document” breaks at inventory scale. A 10,000-document scan cannot cost 10,000 credits when Tier 0 classifies most of them for fractions of a cent.

Four products, four price shapes:

Inventory scan (Tier 0 + Tier 1 + classification). Per domain or bulk tiers, priced cheap. A loss leader whose job is to produce a number and a worklist. Consider free up to a document count for qualified institutions.
Bulk structural repair (Class A). Priced per document, low, because it’s automated. This is the volume product and the clearest margin story we have — near-zero marginal cost against a market that charges $5–8 per page for work that requires a human.
Deep audit (Tier 2). Per document. Judgment work.
Conversion / rebuild (Tier 3). Per document or page, explicit action.

Never charge for a Class D document. It costs almost nothing at Tier 0, and “we didn’t charge you for the 312 files that were already fine” is worth more in the next conversation than the revenue.

Build order

Clause-to-class mapping table — the core IP. Written and reviewed before any code depends on it.
Tier 0 structural probe — smallest piece, biggest economic unlock.
Crawl-based discovery + dedup by content hash.
Tier 1 veraPDF at queue scale, with the large-document cap.
Classifier wiring Tier 0 + Tier 1 output through the mapping table.
Disposition engine + cost estimates + near-miss detection.
Inventory report.
Fixture corpus and golden-file tests — in parallel from step 1, not after.
Validate Canvas and Moodle against self-hosted instances, then fold into discovery.
Search-engine enumeration for unlinked documents.

Open decisions

Is the inventory scan free for qualified institutions, or cheap? Free is a stronger lead magnet and the Tier 0/1 cost supports it.
Do we sell bulk Class A repair as a self-serve product or a service? Self-serve is the better margin story and the better fit for the automated path.
Do we build content remediation capacity, or refer under revenue share? The classified worklist is genuinely valuable to a partner as scoping input, which is leverage in that negotiation either way.