Skip to content
AnglinAI Docs

Gevell Tags document sync API and verification job

Issue #3969 (Phase 2 workstream H, epic #3866, tracking #3972). Decision record: engine-adr.md §9. API contract: docs/api/tags-openapi.yaml.

The engine runs in the browser. The server stores four things and verifies saved versions:

  • the base PDF;
  • the append-only op log;
  • a single-editor lease;
  • saved versions.

Where the code is

PiecePath
Route (Lambda only, /api/v1/tag-docs)workers/api/src/routes/tag-docs.ts
Rules: ownership, conflicts, versions, costworkers/api/src/services/tag-docs/service.ts
DynamoDB conditionsworkers/api/src/services/tag-docs/store.ts
S3, presigned URLsworkers/api/src/services/tag-docs/blobs.ts
Config and allowlistworkers/api/src/services/tag-docs/config.ts
Verifier (SQS consumer, oracles)workers/api/src/tags-verifier/
Verifier imageservices/tags-verifier/Dockerfile
Infra (off by default)infra/cdk/lib/stacks/tags-stack.ts, tagsEnabled in infra/cdk/lib/env-config.ts
Cost op typesmigration 20261001000001_339_cost_ledger_tags_optypes.sql

routes/tags.ts is an unrelated, older feature: file labels at /api/tags. The document API therefore lives at /api/v1/tag-docs.

Storage layout

DynamoDB. These items go in the existing single-table MainTable (accessible-pdf-production-data). Ids are server-issued UUIDs.

pkskWhat
DOC#<docId>METAowner, title, status (awaiting_base/ready/deleting), baseSha256, headSeq, latestVersion, writeUnits (unbooked cost)
DOC#<docId>OP#<seq, 12 digits>one op record, verbatim (or, when over 350 KB, an s3Key), plus the sha256 of its JSON
DOC#<docId>LEASEholderUserId, holderSessionId, expiresAt (epoch ms)
DOC#<docId>VER#<n, 6 digits>a version: upToSeq, sha256, size, s3VersionId pinned at commit, status, verification
TAGUSER#<userId>DOC#<docId>owner index, used for listing (no GSI needed)

Version 0 is the base PDF.

S3. The bucket is accessible-pdf-production-tags-docs-<account>. It is versioned and SSE-KMS with its own CMK (bucket keys on). The bucket policy denies any PUT that is not aws:kms with that key. Object layout under tags/docs/<docId>/:

  • base.pdf
  • versions/<n>.pdf
  • ops/<seq12>-<opId>-<sha256 prefix>.json (offloaded op records; content-addressed, written If-None-Match: *, so nothing can overwrite one)
  • reports/<n>/verify.json

Rules the server enforces

These are DynamoDB conditions, so racing Lambdas cannot both win.

  • Append is one transaction with three parts:

    • a lease check: the caller’s session holds a lease that has not expired;
    • a META update, conditioned on headSeq = baseSeq and status = ready;
    • one conditional put per op (attribute_not_exists(sk)).

  • Append results:

    • A stale or skipped baseSeq gets 409 SEQ_CONFLICT with the current headSeq.
    • A retry of a batch that already landed (same opIds and the same payload hashes at the same seqs) gets 200 duplicate: true. A retry that changed any byte is 409 SEQ_CONFLICT.
    • Before the transaction runs, every record’s seq must equal its position and its baseHash must equal the document’s base.

  • Lease:

    • Acquiring is a conditional put. It succeeds only if there is no lease, the lease has expired (expiresAt < now), or the caller’s own user and session already hold it.
    • Acquiring again is the heartbeat: the client sends it every 15 s, and the lease expires 45 s after the last one.
    • A session whose lease was taken over can never append, not even at the right baseSeq.

  • Versions:

    • latestVersion advances under a condition on its old value, in one transaction with the lease check. Concurrent saves get 409 VERSION_CONFLICT.
    • Commit HEADs the object and requires the exact size and SHA-256. It then pins the S3 VersionId, so downloads and the verifier read exactly the committed bytes even if the key is written again.
    • Commit is resumable: committing a base that is stored while the document is still awaiting_base finishes opening it, and committing a version that is queued re-sends the verification message (the verifier drops duplicates by status).

  • Uploads are presigned for one key. The presign signs:

    • x-amz-checksum-sha256, so S3 rejects other bytes;
    • If-None-Match: *, so an upload never overwrites;
    • the SSE-KMS headers.

    URLs live for 5 minutes.

  • Ownership: every endpoint loads META and compares ownerId with the JWT subject. Not yours and not found both return 404. A malformed docId returns 404 before any key is built.

  • Delete:

    1. Set the document to deleting, which blocks edits.
    2. Book the unbilled write units.
    3. Purge every S3 object version under the prefix.
    4. Purge the items, META last.

    A failure leaves the document deleting, and the call can be retried.

Access flag (Phase 2)

TAGS_ALLOWLIST on the Lambda is a comma-separated list of Supabase user ids (auth.users.id). It is set from tagsAllowlist in infra/cdk/lib/env-config.ts.

  • Unset or empty means nobody gets in: 403 TAGS_NOT_ENABLED.
  • Emails are not accepted.
  • Missing storage env returns 503 TAGS_NOT_CONFIGURED.

Verification job

On commit of version n > 0, the API sends {type:'tags-verify', docId, version} to the tags-verify SQS queue. The tags-verifier container on the EC2 anchor then:

  1. Loads META and VER#n. It drops the message if the version is already settled or the document is gone.

  2. Sets the version to verifying and adds 1 to attempts.

  3. Reads base and version n at their pinned S3 versions, and the op records 1..upToSeq. The records must all be present and in order, and every offloaded payload must match its hash.

  4. Runs the oracles:

    CheckCommandpassfailcould not verify
    replaytag-engine replay base --oplog log --out r.pdf --json, then sha256(r.pdf) == sha256(uploaded)exit 0 and equal bytesbytes differ; stored bytes ≠ committed hash; exit 3 (conflict or drift); exit 1 (save not clean)missing binary, exit 2, signal, timeout, bad JSON
    qpdfqpdf --check uploadedexit 0, or 3 (warnings)exit 2anything else
    pixelDifftag-engine diff base uploaded --json with an explicit PDFIUM_LIB_PATH (150 dpi, tolerance 2, ≥ 99.95 % per page)exit 0 and report passexit 1 and report fail; waived when the version declared pixelChangesReasonlib path unset or missing, exit 2, report and exit disagree
    ua1verapdf --flavour ua1 --format jsoncompliantnot compliant: reported, does not fail integritymissing binary, parse exception, other profile, exit ≠ 0/1, exit and report disagree

  5. Sets the verdict:

    • failed if replay, qpdf or pixelDiff failed;
    • otherwise could_not_verify if any check could not run (veraPDF included);
    • otherwise passed.

    The UA-1 result is stored as ua1Compliant and ua1FailedRules. A document that is still being remediated may fail UA-1, but a missing veraPDF is never a pass.

  6. Writes reports/<n>/verify.json, sets the terminal status, and books tags-verify cost.

Retries. An unexpected error (S3, DynamoDB, a missing op) leaves the message for SQS to redeliver. On the 3rd receive the verifier records could_not_verify, with the error in each check’s detail. The queue’s DLQ (maxReceiveCount 3) should only ever see crashes mid-job.

Oracle versions are pinned to the engine CI image (engine/ci/Dockerfile): qpdf 12.3.2, libpdfium from pypdfium2 5.13.0, veraPDF 1.30.2. The image build checks every oracle.

Cost tracking

operation_typeBooked byWhenEstimate
tags-storageLambda (recordCost)each version commit, and deleteS3 PUT + KMS request + DynamoDB write units the log accrued since the last booking (META.writeUnits, decremented atomically after booking)
tags-verifyLambda (recordCost), from the durationMs the verifier storedthe first read of a finished version (GET doc or version), or delete; claimed once per version (verifyCostBooked)durationMs always; dollars only when TAGS_VERIFY_HOURLY_USD is set on the Lambda (same convention as flyer-tagging-host-compute)

The verifier books nothing itself, on purpose: it parses untrusted PDFs, so it holds no Supabase credentials. Write units are taken from META atomically (writeUnits >= :n condition), so racing commits or a retried delete never book them twice.

Storage-time cost (GB-month) is not booked per request. Read it from the AWS bill; the bucket carries the project=accessible-pdf tag.

Deploying (none of this has been applied)

Merging the PR changes nothing in AWS, because tagsEnabled is false everywhere: Deploy AWS synthesises identical stacks. Here is the order to switch it on.

  1. Migration 339. Apply supabase/migrations/20261001000001_339_cost_ledger_tags_optypes.sql to prod through the Supabase MCP. It is additive and safe before the code.

  2. Infra.

    1. Set tagsEnabled: true and tagsAllowlist: ['<Larry's auth.users.id>'] for production in infra/cdk/lib/env-config.ts.
    2. Run cdk deploy AccessiblePdfProd-Tags AccessiblePdfProd-Api AccessiblePdfProd-Anchor, or merge and let Deploy AWS run --all.
    3. Check the output for the new TagsStack (key, bucket, queue + DLQ), the TAGS_* env on the API Lambda, and the grants on the anchor role.

  3. Verifier image.

    1. Add { "name": "tags-verifier", "context": ".", "dockerfile": "services/tags-verifier/Dockerfile" } to infra/cdk/anchor-images.json.
    2. Add engine/** and services/tags-verifier/** to the path filters in .github/workflows/build-anchor-images.yml.
    3. Deploy the Anchor stack first, which creates the ECR repo; the image build fails until the repo exists.

  4. Anchor service. Add the service to generate-compose.py / the root docker-compose.yml and regenerate the compose file. The verifier parses untrusted PDFs, so it gets no env_file (no anchor secrets, no Supabase key) and a locked-down container:

    tags-verifier:
      image: ${ANCHOR_REGISTRY}/accessible-pdf-production-anchor-tags-verifier:${IMAGE_TAG}
      init: true
      # Deliberately no env_file: only these non-secret values.
      environment:
        - TAGS_VERIFY_QUEUE_URL=<TagsVerifyQueueUrl output>
        - TAGS_DOCS_BUCKET=<TagsDocsBucketName output>
        - TAGS_DOCS_KMS_KEY_ARN=<TagsDocsKeyArn output>
        - DYNAMODB_TABLE=accessible-pdf-production-data
        - AWS_REGION=us-east-1
      read_only: true
      tmpfs: [/tmp:size=2g]
      cap_drop: [ALL]
      security_opt: [no-new-privileges:true]
      pids_limit: 256
      restart: unless-stopped
      deploy: { resources: { limits: { memory: 3g } } }
      logging: { driver: awslogs, options: { awslogs-region: us-east-1, awslogs-group: /gevell/anchor, awslogs-stream: tags-verifier } }

    Known residual risk: the container still reaches the anchor’s instance role through IMDS (hop limit 2), and that role is broad. A dedicated, narrow role for the verifier is a follow-up (see the LOW-findings issue).

    Optionally set TAGS_VERIFY_HOURLY_USD (amortised anchor $/h) on the API Lambda to price tags-verify rows.

  5. Smoke test as an allowlisted user:

    1. Create a document, PUT the base, and commit v0.
    2. Acquire the lease and append ops (produced by the engine).
    3. Create v1, PUT, commit, then poll GET /versions/1 until the status is passed.
    4. Also check that a non-allowlisted user gets 403.

Rollback: set tagsEnabled: false and redeploy. The API returns 503, the verifier container can be stopped, and the bucket and key are RETAIN.

Runbook

  • Version stuck in queued. Check the verifier’s logs (/gevell/anchor, stream tags-verifier) and queue depth. The client can re-commit only from enqueue_failed, so a stuck queued version needs the verifier running.
  • DLQ has messages. The verifier crashed mid-job three times. The version is left verifying. Fix the cause, then redrive the DLQ: the job resumes from verifying.
  • Every version is could_not_verify. An oracle is missing or misconfigured in the image; see each check’s detail. PDFIUM_LIB_PATH must point at the bundled libpdfium.so.

Testing

  • npx vitest run src/__tests__/services/tag-docs src/__tests__/routes/tag-docs.test.ts src/__tests__/tags-verifier in workers/api. The DynamoDB fake evaluates the store’s real condition expressions, so removing a condition fails a test.

  • Real oracles (opt-in; the workers/api CI job has no oracle toolchain):

    Terminal window
    cargo build --release -p tag-engine-cli --manifest-path engine/Cargo.toml
    cd workers/api
    TAGS_VERIFIER_REAL_ORACLES=1 PDFIUM_LIB_PATH=<libpdfium file> \
      npx vitest run src/__tests__/tags-verifier/real-oracles.test.ts

    This produces a save with tag-engine apply and checks the following:

    • a genuine save passes;
    • one changed byte fails replay;
    • a dropped op fails;
    • removing each of tag-engine, qpdf, PDFium and veraPDF gives could_not_verify.