PDF Person-Photo Extraction (#4279)

When Gevell Org Chart builds a chart from an uploaded PDF (a PowerPoint or Word export is the common case), the leadership boxes often embed a small portrait JPEG next to each person’s name. The vision extraction (services/org-chart/vision-extractor.ts) reads names/titles/relationships from a rasterized screenshot of the page and has no notion of “this image belongs to this box” — photos were silently dropped.

This feature recovers them deterministically, with no AI call — see Cost below.

Where it runs

routes/org-chart/extract.ts, inside the /extract handler, after people (and their manager relationships) are inserted and before the initial version snapshot is taken — so a matched photo’s photo_url is already present in that first snapshot.

It only runs for PDF uploads (isPdfUpload, captured from chart.source_image_mime_type before the route overwrites mimeType to 'image/png' for the rasterized pages sent to the vision model). A plain image upload (PNG/JPG) has no embedded XObjects to recover and is unchanged.

A failure anywhere in this pass — a corrupt PDF, a pdf.js decode error, an R2/malware-scan failure on one photo — is caught at the call site and only logged (log.error('PDF photo extraction failed', err, { chartId })). It never fails chart extraction itself.

Pipeline

Implemented in services/org-chart/pdf-photo-extractor.ts. No headless browser is involved (unlike services/org-chart/pdf-converter.ts’s page-to-PNG rendering) — everything here is pure pdf.js operator-list / text-content parsing via unpdf (a Workers/Lambda-safe pdf.js build, already a workers/api dependency), which is why it can also be exercised directly in Node for local verification (see Verifying against a real PDF).

1. Geometry. Walk each page’s pdf.js operator list (paintImageXObject calls, tracking the CTM through save/restore/transform — the same technique utils/pdf-to-png.ts’s extractImageRectsFromPdfPages already uses for the region-crop pipeline, #1403) to get each embedded image’s placement rectangle in page points.

2. Filter. isLikelyPhotoRect keeps only geometry that could plausibly be a headshot:

   Constant	Value	Rejects
   MIN_PHOTO_DIM_PT	20pt	specks, bullets, rule lines
   MIN_PHOTO_AREA_PCT	0.05% of page area	same intent, belt-and-suspenders
   MAX_PHOTO_AREA_PCT	20% of page area	a figure too large to be a small headshot
   FULL_PAGE_AREA_PCT	60% of page area	a full-bleed background/page scan
   MAX_PHOTO_ASPECT_RATIO	2.4 (longer÷shorter side)	banner/logo images (ALDOT’s two logos are 521×115pt ≈ 4.5:1)

   No pixel inspection — this is pure geometry, so it’s cheap even on a dense multi-image page.

3. Match. Read the page’s text layer (page.getTextContent()). For each surviving photo, pickNameTextForPhoto tries two layouts, in order:

   • right of the photo (the common case) — text within MAX_MATCH_GAP_PT (50pt) of the photo’s right edge whose vertical span genuinely overlaps the photo’s own vertical span (±VERTICAL_SLACK_PT = 6pt slack) — a loose “starts somewhere in a y-range” check previously let a neighboring card’s name get grabbed when it sat close horizontally without actually being beside this photo;
   • below the photo (a caption-style layout), tried only if (1) found nothing — text within MAX_MATCH_GAP_PT below the photo’s bottom edge whose horizontal span overlaps the photo’s own (±HORIZONTAL_SLACK_PT).

   Within either set, the topmost (then leftmost) line is the name line — org-chart boxes always put the name first, title/department below it. That text is resolved against the chart’s extracted people list (matchTextToPerson):

   • an exact normalized-name match wins outright, even when another person’s name is a token-subset of it (ALDOT has both “William Patty” and ‘William F. “Bill” Patty” — the literal text next to the photo disambiguates them correctly);
   • otherwise, first+last significant-token containment is the fallback (tolerates a middle initial or a quoted nickname the other side doesn’t carry, e.g. “Kyle M. Leverette” vs. “Kyle Leverette”, or a “Jr.”/“Sr.” suffix present on one side only) — except a single-token person name (“Cooper”) must match the text in full, not merely appear as one of its words, or “Cooper” would equally “match” “John Cooper” and “Brian Cooper” text and silently pick whichever is nearer instead of refusing to guess;
   • zero or multiple candidates at either stage → no match, skipped, never guessed. One photo per person: if two candidate photos both resolve to the same person, only the nearer one is kept (duplicate-person).

4. Decode + store. Matched images are decoded via pdf.js’s own image object cache (page.objs.get / page.commonObjs.get, pure JS — no canvas), then re-encoded as JPEG with sharp (already a workers/api dependency — see services/pdf-pixel-contrast.ts) since there is no canvas available outside the headless-browser renderer that pdf-converter.ts uses for page screenshots. Stored via the same helper the manual per-person photo upload uses — services/org-chart/photo-storage.ts’s storePersonPhotoBytes — so both paths share one R2 key layout (R2_PATHS.personPhoto), the same malware-scan gate (#3262, storeScannedUpload), and the same PERSON_PHOTO_LIMITS resize/size enforcement (a sharp resize(1024, 1024, { fit: 'inside' }) pass before encoding, plus a max-bytes check — the manual-upload route already rejected an oversized image client-side, but an embedded PDF image has no such pre-check, so the limit has to live in the shared helper). routes/org-chart/people.ts’s manual-upload route was refactored to call the same helper. Storage is bounded to PHOTO_STORE_CONCURRENCY (4) photos in flight at once (storePersonPhotosWithConcurrency) — an unbounded Promise.all over a dense chart’s worth of matched photos could stall the request well past its own deadline waiting on the malware-scan gate.

Limits on untrusted input (review round 1, H2)

The PDF is untrusted and this runs on the request path, so:

• getDocumentProxy is called with maxImageSize: 4096×4096 (pdf.js’s own decoded-pixel-count cap) so a decompression-bomb image can’t blow memory/CPU decoding it.
• The whole extraction loop is bounded by PHOTO_EXTRACTION_DEADLINE_MS (15s default, overridable for tests), checked before starting each page and before decoding/encoding each matched photo — not true preemption (work already in flight still runs to completion), but it bounds how much new work a pathological PDF can start. routes/org-chart/extract.ts additionally races the whole extraction+storage call against the same deadline as a backstop (Promise.race, abandon-and-log on timeout).
• pdf.destroy() runs in a finally so a request against an untrusted PDF doesn’t leak pdf.js’s worker-side document resources.
• The PDF bytes are copied (.slice(0)) before handing them to getDocumentProxy, matching the convention other unpdf callers in this codebase use (e.g. unpdf-claude-converter.ts) — extract.ts hands the same underlying buffer to the page-rasterizer too, so the photo extractor must not assume it owns the live buffer for its document’s lifetime.
• A timed-out extraction must not still write (review round 2, M-1). extract.ts’s Promise.race moves the request on at the deadline, but the abandoned extraction keeps running in the background (JS can’t preempt it) and would otherwise reach R2/DB writes for a photo the request already reported as 0-matched. An abandoned flag (set when the timeout wins the race) is checked before storage and before each photo_url DB update, so late writes become no-ops.
• sharp is a native addon — routes/org-chart/people.ts (and this module) are also mounted on the legacy CF Worker fallback entry point (src/index.ts, see NEXT_PUBLIC_API_URL_FALLBACK), where sharp cannot run at all. resizeToLimits distinguishes “sharp unavailable” (the dynamic import('sharp') itself throws) from a genuine decode failure: the former degrades gracefully — resize skipped, original bytes kept, maxBytes still enforced — while the latter still fails closed.

Does NOT touch the show-photos setting

Per Larry’s decision (2026-10-06): extraction attaches photo_url to matched people, but never auto-enables photo display. org_charts grew a show_photos boolean not null default false column (migration supabase/migrations/20261006000000_349_org_chart_pdf_photo_extraction.sql). Every place that builds TemplateOptions from a request now resolves options.showPhotos ?? chart.show_photos ?? false — POST /generate (finalize/download), POST /generate-previews (the editor’s live preview), and the editor client no longer hardcodes showPhotos: true on its preview calls. This matters because generate-previews’ resolved options are cached and copied verbatim into org_chart_outputs.options at finalize time (regenerateOutputHtml replays that stored value on every later serve) — so a mismatch between the live preview’s default and the download/export default used to mean “Turn on photos” silently did nothing (review round 1, H1).

Backfill (migration 349): before this feature, the editor’s preview call hardcoded showPhotos: true unconditionally, so every chart with a manually-uploaded photo was already rendering with photos in its downloads/exports. The migration sets show_photos = true for exactly the charts that already have a person with a non-null photo_url at migration time, so existing users see no change; every other chart (including every chart with PDF-extracted photos) starts off.

chart.show_photos is authoritative for every chart-backed render (review round 2, H1-a). It isn’t enough to fix the DEFAULT resolution above — regenerateOutputHtml (the function every finalize/download/share/embed path, and the stale-output auto-regen path, renders through) used to reuse output.options verbatim, which baked in whatever showPhotos was true at finalize time. A PATCH + its stale-mark would therefore re-render with the old value forever. regenerateOutputHtml now overrides options.showPhotos with the live chart.show_photos unconditionally, regardless of what’s stored in output.options or passed as a caller override; every select that loads a chart row feeding it (org-chart-export.ts, org-chart-share.ts) was extended to include the column.

The regen-and-stamp itself has a privacy race, closed in review round 4, MEDIUM-A. regenerateStaleOutputInPlace renders using whatever chart.show_photos was when the call started, then writes the HTML to R2 and stamps render_version current. If the owner toggles photos ON (which is what triggered this regen, from public embed/share traffic hitting a chart whose output was just marked stale) and then OFF again before the render finishes, the render in hand reflects a setting that’s no longer current — stamping it would serve a privacy-sensitive with-photos artifact indefinitely even though the live setting is off. Before writing anything, the function re-reads org_charts.show_photos live and bails (discards the render, leaves the output stale for the next hit to retry) on a mismatch. It also stamps render_version with a conditional update (.eq('render_version', <the value this call started from>), which PostgREST already treats as IS NULL when that value is null — no separate .is() branch needed) and checks the update actually matched a row — a second, more general guard against any other concurrent change to the same output. A third, narrower check runs right after a successful stamp: re-read show_photos ONE more time, and if the owner flipped it again in the sliver of time between the pre-write check and the stamp, reset render_version back to 0 so the next hit re-renders instead of serving what was just committed.

render_version is NOT NULL DEFAULT 0 in prod — never write null (review round 5, HIGH). Migration 20260714120000_213_org_chart_output_render_version.sql made the column integer NOT NULL DEFAULT 0. An earlier round of the show_photos work (round 2) stale-marked outputs with render_version: null — in prod that violates the NOT NULL constraint and the whole UPDATE fails, so a photos-off toggle would never actually reach embeds/shares/downloads (the bug looked invisible locally because the test mocks accepted null unconditionally). invalidateShowPhotosCaches now writes render_version: 0 instead — 0 is falsy, so regenerateStaleOutputInPlace’s output.render_version && … >= RENDER_VERSION staleness guard already treats it as stale, exactly like null would have if the column allowed it. The regen test mocks now reject a render_version: null write the way real Postgres would, so this can’t regress silently again. (routes/org-chart/breaks.ts’s markAffectedOutputsStale has the identical bug — out of scope here, tracked/fixed separately in #4285; this PR does not touch breaks.ts to avoid conflicting with that fix.)

Changing it invalidates caches: PATCH /api/orgcharts/:id { showPhotos } (after validating it’s a boolean) and a manual photo upload that flips show_photos on both call invalidateShowPhotosCaches (services/org-chart/photo-storage.ts) — marks every stored org_chart_outputs row for the chart stale (render_version: null, the same pattern routes/org-chart/breaks.ts’s markAffectedOutputsStale uses for page breaks) and invalidates the KV preview cache (invalidatePreviewCache). Both best-effort, logged on failure, never blocking the write itself.

Two ways to turn photos on (review round 2, H1-b). A chart that already has people with manually-uploaded photos but show_photos off (every chart built after this migration that isn’t a PDF-extraction match) would otherwise show initials with no way to display them — the one-time “Found N photos” extraction banner doesn’t cover that case. So:

• A visible “Show photos” toggle in the preview page’s toolbar (next to Branding/Spacing), reflecting and PATCHing chart.show_photos directly — optimistic (flips immediately, reverts with an error toast on failure), re-generates previews on success. handleToggleShowPhotos backs both this and the banner’s CTA (which just calls it with true).
• A manual photo upload (POST /people/:pid/photo) auto-enables show_photos if it’s currently off — UPDATE ... SET show_photos = true WHERE id = :chartId AND show_photos = false, so it’s a no-op (no needless stale-marking) once already on. Uploading a photo is an explicit “I want this shown” action, unlike PDF extraction, which never auto-enables.

The editor also surfaces a dismissible “Found N photos” notice (apps/org-chart/src/app/preview/page.tsx, driven by a photosFound query param the wizard redirect — or its polling fallback — carries) with a “Turn on photos for downloads” button. The notice’s role="status" container stays mounted at all times (text is injected/cleared, not the element itself) so assistive tech reliably announces it, and it suppresses itself once the chart’s show_photos is already on (fetched via GET /extract/status, which now also returns photosExtracted/ showPhotos) — re-extracting a chart resets photos_extracted_count to null first, so a stale count from a prior extraction can’t resurface the notice for photos that no longer exist.

Removing a photo (manual or extracted) is a one-click “Remove” button next to each person’s photo in the editor table (apps/org-chart/src/app/editor/page.tsx), which clears photo_url through the same undo-tracked field-update path as the manager reparent control. This only clears the DB reference (same as the pre-existing manual-clear path via the photo-URL text field) — it does not delete the R2 object, matching that existing behavior; see R2 cleanup below for where objects actually do get deleted.

Public embeds and hosted pages also pick up the change (review round 3, M-1)

chart.show_photos being authoritative at render time (round 2, H1-a above) only helps if every SERVE path actually checks staleness before serving a stored artifact:

• GET /api/embed/:token (routes/org-chart/embed.ts) now resolves its output through resolveEmbedHtml, which checks render_version and calls regenerateStaleOutputInPlace first when stale — the exact same pattern org-chart-share.ts’s readShareHtml already used. Without this, turning photos OFF for privacy (e.g. a show_photos:false PATCH) would never reach an already-created public embed.
• POST /orgcharts/:id/publish (routes/org-chart/hosting.ts) is different: a hosted page is a published snapshot — publish copies the output’s HTML into its own independent hosted/{slug}/index.html R2 object once, and static hosting serves THAT object directly, never looking at org_chart_outputs again. There’s no request-time render to intercept. So publish instead regenerates the SOURCE output first if it’s stale, ensuring the snapshot it takes is current — but a show_photos change made AFTER a chart was already published does not retroactively reach that hosted page. A republish is required to pick up a show_photos (or any other) change on an already-hosted chart.

R2 cleanup (review round 1, M3 — narrowed in round 4, MEDIUM-B)

A person’s photo object is never referenced again once its row is gone, so DELETE /api/orgcharts/:id cleans it up (best-effort, logged on failure, never blocking the operation, alongside the existing source-image/output cleanup). It recovers the R2 key from the stored photo_url via r2KeyFromPersonPhotoUrl (the key is always the users/{userId}/photos/{chartId}/... substring of the public URL publicAssetUrl produced) rather than needing a list-by-prefix storage capability. “Remove photo” in the editor intentionally does not delete the R2 object — only chart delete does.

/re-extract does NOT delete R2 photo objects (review round 4, MEDIUM-B — reverted from round 2/3). An earlier round of this feature also cleaned up photos on re-extract (after the people-table clear succeeded, to avoid destroying photos for rows a failed clear left in place). That was a DATA- LOSS regression versus main: org_chart_versions snapshots (utils/org-chart-snapshot.ts) embed each person’s photoUrl verbatim, and restoring an older version (routes/org-chart/versions.ts) writes that photo_url straight back onto org_chart_people — against an R2 object re-extract had already deleted, since re-extraction does NOT delete a chart’s version history. Deleting on chart delete is safe because org_chart_versions.org_chart_id has ON DELETE CASCADE (supabase/migrations/20260402000001_056_org_chart_tables.sql) — no version can outlive the chart, so no version can end up pointing at a photo that chart-delete cleaned up. A version CAN and does outlive a re-extract, so photos must not be cleaned up there. Net effect: re-extracted-away photos are now orphaned in R2 until the chart itself is deleted — a storage-cost nit, not a correctness problem, and the trade this code deliberately makes in favor of not breaking version restore.

Forking copies photos too (review round 2, M-2). forkChart (services/org-chart/fork-chart.ts) used to copy photo_url verbatim, pointing straight at the SOURCE chart’s R2 key — once chart-delete started cleaning up photo objects (M3, above), deleting the source chart would silently break every photo in every chart forked from it. Each scoped person’s photo (when it’s one of ours — r2KeyFromPersonPhotoUrl returns null for a pasted external URL, which is left untouched) is now copied into the fork’s own key space (copyBrandingImage, the same helper branding images already use) and photo_url rewritten to the copy; a copy failure drops the photo (null) rather than leaving a reference to the source’s key. The fork also copies the source chart’s show_photos so it renders consistently with where it came from. Photos are copied with bounded concurrency (mapWithConcurrency, 4 at once) and, if the fork’s people insert fails and the chart is rolled back, the already-copied photo objects are cleaned up too (review round 3, LOW-4).

Pre-fix forks still need protecting (review round 3, M-2). The fix above only applies going forward — a fork made BEFORE it still has photo_urls pointing at its SOURCE chart’s R2 key. So before chart-delete’s cleanup actually deletes anything, filterOutCrossChartReferencedPhotoUrls (services/org-chart/photo-storage.ts, batches of 50 — Postgrest .in() URL-length limits) excludes any key still referenced by a person row in some OTHER chart (org_chart_people.photo_url lookup, .neq('org_chart_id', …)) — fails closed per batch (a lookup error means “treat as referenced, don’t delete”).

Known limitation (unchanged by the fixes above): a chart’s OWN version snapshots still embed whatever photoUrl a person had at snapshot time. Restoring an old version after a person’s photo was later replaced or removed (manually, or by a subsequent re-extraction’s matches) shows whatever that old photoUrl resolves to now — correct if the underlying object is untouched, a broken image only in the narrow case where that specific R2 key was itself overwritten/removed by an unrelated later action. Not something either round of this feature introduced or could fully close without snapshotting the photo bytes themselves; flagged for whoever next touches version restore.

Cost

$0. No AI/vision call — this is deterministic geometry and text-layer matching against data pdf.js already parses. The only incremental cost is the R2 write per matched photo, which is logged (not metered — org-chart monetizes via credits, not per-write cost rows; see the identical comment in routes/org-chart/people.ts’s manual photo-upload route).

Deploy order

Migration 349 (show_photos, photos_extracted_count) must be applied to prod before this code ships — /extract, PATCH /orgcharts/:id, and every render path reference both columns unconditionally. Larry applies migration 349 to prod before merging this PR.

Limitations

• Only ordinary embedded image XObjects are considered — inline images (paintInlineImageXObject) and image masks (paintImageMaskXObject) are not decoded (masks are compositing layers, not standalone photos; inline images are rare in slide-deck exports). A page where photos are inline images will extract zero matches, silently (logged at info, not warn — this is an expected shape for some PDFs, not a failure).
• Matching tries the name to the right of the photo first, then below it. A chart with the name above the photo will not match (future: widen pickNameTextForPhoto’s search if this turns out to be common).
• MAX_PHOTO_EXTRACTION_PAGES caps inspection at 10 pages, matching pdf-converter.ts’s MAX_PDF_PAGES — a photo on page 11+ of a very long PDF is not considered.
• Image uploads (PNG/JPG/scans) are explicitly out of scope — there is no embedded-object geometry to recover from a flat raster. The issue tracks this separately as a vision-model-bounding-box suggestion feature.

Verifying against a real PDF

Because the pipeline has no browser/canvas dependency, it can be exercised directly against a real PDF from a throwaway Vitest file (see the test suite’s own synthetic-PDF fixture in workers/api/src/__tests__/services/pdf-photo-extractor.test.ts for the pattern — build with pdf-lib, call extractPdfPersonPhotos directly, and write result.matches[].bytes to disk to inspect).

Verified against a real government org chart (ALDOT leadership, 1-page PowerPoint-exported PDF, 1296×1224pt, 8 portrait JPEGs + 2 logos + a full-page background): 8/8 people matched correctly, 0 false positives on the logos or background, 0 skipped.