From fb8ff612e8ba32f4a0c204d451b596e738ffa79e Mon Sep 17 00:00:00 2001 From: DEls Date: Fri, 18 Sep 2026 18:34:57 +0200 Subject: [PATCH] docs: design hybrid delivery note ingestion --- ...8-hybrid-delivery-note-ingestion-design.md | 428 ++++++++++++++++++ 1 file changed, 428 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-18-hybrid-delivery-note-ingestion-design.md diff --git a/docs/superpowers/specs/2026-09-18-hybrid-delivery-note-ingestion-design.md b/docs/superpowers/specs/2026-09-18-hybrid-delivery-note-ingestion-design.md new file mode 100644 index 0000000..3d99363 --- /dev/null +++ b/docs/superpowers/specs/2026-09-18-hybrid-delivery-note-ingestion-design.md @@ -0,0 +1,428 @@ +# Hybrid delivery-note ingestion and matching + +> A photo-first, staged ingestion pipeline for bulk PDFs and images. It +> preserves every uploaded source, uses vision once per page, assembles pages +> into documents, resolves supplier and purchase order within the selected +> site, and sends only genuine ambiguity to a human review queue. +> +> Date: 2026-09-18 + +## 1. Decision + +Extend the existing delivery-note inbox rather than replace it. The current +pipeline already has the right broad separation — uploaded batches, +independent page reads, candidate notes, checks, and promotion — but it runs +those stages as one operation and does not persist enough intermediate state +for reliable split review, bulk photographs, or supplier-centred exceptions. + +The new design is a **hybrid**: + +- a vision model reads raster pages and photographs; +- deterministic code normalises identifiers, assembles clear page sequences, + resolves exact relations, converts units, and enforces posting gates; +- a human handles only ambiguous page boundaries, suppliers, POs, or line + items; +- n8n schedules work and reports failures, while application code owns the + business rules and ledger writes. + +This is not a second OCR system. The existing MiniMax vision call is already +reading pixels rather than a PDF text layer. Classical OCR may later be added +as supporting searchable text, but it is not required for this increment and +must never overrule the structured vision result merely because it returned a +string. + +## 2. Scope and decomposition + +This design is intentionally split into three increments. Combining all three +would mix ingestion, mobile capture, and accounts-payable semantics in one +risky release. + +### Increment A — implement now + +1. Accept one or many PDF, JPEG, PNG, WebP, TIFF, HEIC, or HEIF sources in a + batch, subject to decoder support and clear rejection when a format cannot + be decoded. +2. Preserve every original source and produce normalised page images. +3. Persist one rich vision result per page and reuse it on retries. +4. Assemble pages into document candidates, with a split-review screen for + uncertain boundaries. +5. Resolve supplier and PO using site-scoped evidence. +6. Present a supplier/PO-grouped work queue and a two-column line-item linker. +7. Post only validated candidates into the existing delivery-note ledger. + +### Increment B — separate design and migration + +Introduce a physical `receipt` record and attach delivery notes, invoices, +collection notes, and weighbridge tickets as evidence of that receipt. This is +needed to prevent a delivery note and later invoice for the same load from +counting as two deliveries. It also becomes the basis of the month-end +three-way match and payment report. + +Increment A must not claim that the existing supplier ledger is a final amount +payable. It remains a delivered-value operational view until Increment B is +complete. + +### Increment C — mobile capture integration + +The separate site app may use GPS to suggest a site and may identify a +multi-photo document at capture time. Increment A defines the server contract +for that app, but does not implement the app itself. + +## 3. Success criteria + +1. A 20-page PDF containing 20 one-page notes produces 20 independently + traceable document candidates. +2. A two-page note produces one candidate with both pages attached. +3. Fifty JPEGs uploaded together can be treated as one batch rather than 50 + unrelated batches. +4. A JPEG and a PDF page are presented to the vision model in the same known, + correctly labelled image format. +5. An exact unique PO number resolves the PO, supplier, and site without + requiring the supplier name to be readable. +6. A weak fuzzy match never posts financial quantities automatically. +7. One ambiguous note does not prevent clear sibling notes from reaching the + ledger. +8. Every manual split, merge, relation change, dismissal, and posting action is + attributable and auditable. +9. Retrying a stage cannot create duplicate pages, candidates, notes, or ledger + lines. + +## 4. Intake and source model + +### 4.1 Batch hierarchy + +Use four levels: + +```text +dn_batches + -> dn_batch_sources original PDF or photograph + -> dn_batch_pages normalised page/region image + vision result + -> dn_batch_notes assembled candidate document + -> delivery_notes and delivery_note_items after posting +``` + +`dn_batches.file` remains readable for legacy rows, but new uploads write +originals to `dn_batch_sources`. This is necessary because PocketBase's current +batch file field accepts only one file, while the mobile and bulk flows need +many photographs in one batch. + +### 4.2 Upload contract + +`POST /api/delivery-notes/upload` continues to accept multipart form data and +gains a batch mode: + +- `file`: repeated source files; +- `site`: required authoritative site chosen or confirmed by the operator; +- `source`: `web`, `mobile`, or `email`; +- `capture_session_id`: optional idempotency key supplied by the mobile app; +- `delivery_session_id`: optional hint that several photographs belong to one + physical document; +- `captured_at`, `latitude`, `longitude`, `location_accuracy_m`: optional + capture evidence. + +Coordinates suggest a site; they do not silently override the confirmed site. +GPS can be unavailable, inaccurate, or captured later from the site office. + +The current endpoint's single-file response remains compatible. Multiple +files default to one batch unless the caller explicitly requests one batch per +file. This allows bulk photographs to be assembled across adjacent images. + +### 4.3 Image normalisation + +Every source is decoded before the model sees it: + +- PDFs are rendered page-by-page with Poppler. +- Raster images are auto-rotated using EXIF orientation, colour-normalised, + and converted to PNG. +- Large images are resized to a bounded long edge while retaining enough + detail for small printed numbers. +- MIME type is detected from decoded content, not trusted from the request + header or file extension. +- The original source is never overwritten. + +Use a direct image library such as `sharp` for raster decoding and +normalisation. HEIC/HEIF/TIFF support is accepted only when the deployed +decoder can prove it can read the file; otherwise the API returns a per-file +`unsupported_image_codec` error. The mobile app should upload JPEG for the +most predictable path. + +The current pass-through behaviour must be removed: JPEG bytes must not be +labelled `image/png`, and a JPEG candidate must not be uploaded to PocketBase +as an `application/pdf` blob. + +## 5. Persisted page reading + +Create `dn_batch_pages` with at least: + +- batch and source relations; +- source page number and optional region number; +- normalised image file and image hash; +- detected source MIME, dimensions, and applied rotation; +- read status: `pending`, `reading`, `read`, or `failed`; +- structured extraction JSON and extraction schema version; +- model name, prompt version, and raw response/error summary; +- optional manually assigned document group. + +The vision schema extends the existing `PageRead` with: + +- printed supplier name and VAT/registration identifiers; +- printed site/project/customer references; +- PO number candidates rather than one invented value; +- document type, primary number, alternate numbers, and page numbering; +- header/continuation/multiple-document signals; +- dates, vehicle registration, receiver, weights and units; +- complete line items, product codes, quantities, units, rates, and totals; +- raw page transcription for search and diagnosis; +- field-level confidence or evidence notes where the model supports them. + +One rich read is performed per normalised page. Retries reuse a successful +read unless an operator requests re-extraction or the prompt/schema version +changes. A model parse error is stored against that page rather than shrinking +the page count. + +## 6. Document assembly + +Assembly is its own persisted stage, not a side effect of extraction. + +### 6.1 Deterministic first pass + +Adjacent pages are joined when strong evidence agrees, including: + +- the same document number; +- explicit `page n of m` numbering; +- a continuation page directly following a header page; +- a shared mobile `delivery_session_id` that does not contradict printed + identifiers. + +A new document starts when a new header carries a different document number. +All pages must be accounted for as grouped, open, or dismissed. + +### 6.2 Vision-assisted ambiguity + +Only ambiguous adjacency receives a second, narrow model decision. The model +sees neighbouring thumbnails and their persisted reads and answers a grouping +question; it does not re-extract all commercial fields. + +### 6.3 Split review + +The batch detail screen shows page thumbnails in order and the proposed groups. +The operator can: + +- join an adjacent page to a document; +- split before or after a page; +- move a page to another candidate; +- dismiss a blank, cover, or unrelated page; +- re-run vision on one page; +- mark an image containing multiple physical documents for recapture or a + later crop workflow. + +Increment A does not attempt arbitrary automatic cropping of two documents in +one photograph. It detects and isolates that exception without silently +posting it. Automatic region cropping can be added after real examples exist. + +Reassembly creates a new assembly revision and reuses page reads. It does not +call vision again and cannot duplicate an already posted ledger note. + +## 7. Supplier and PO resolution + +Resolution is ordered evidence, not one fuzzy score. + +1. Search purchase orders belonging to the confirmed batch site. +2. Normalise PO references by case, whitespace, punctuation, and known OCR + confusions. +3. A unique exact normalised PO match is authoritative. It supplies the + internal supplier relation even when the printed supplier is unreadable. +4. Otherwise resolve supplier by VAT/registration identifier, then a learned + supplier alias, then a unique normalised legal/trading name. +5. Once a supplier is known, rank only that supplier's open POs on the site by + partial PO number, date, product codes, descriptions, units, quantities, + and printed rates. +6. Auto-select only deterministic unique matches. Fuzzy results are candidates + for review until production examples establish safe, measured thresholds + and a required winner margin. + +Add a `supplier_document_aliases` collection so an operator can teach the +system that a printed trading name belongs to an existing supplier. Store the +raw value, normalised key, supplier relation, source, and audit metadata. + +`dn_batch_notes` gains the resolved supplier relation, raw supplier value, +match method, candidate/evidence JSON, pages JSON, and an independent workflow +status. + +## 8. Note and line workflow + +The batch board answers whether source files were successfully prepared and +assembled. The note queue answers whether each resulting commercial document +has been resolved and posted. They must not share one overloaded status. + +### 8.1 Batch stages + +```text +Queued -> Preparing -> Reading -> Split review -> Complete + \-> Failed +``` + +`Complete` means every source page is accounted for and document candidates +exist. It does not mean every candidate is financially posted. + +### 8.2 Candidate stages + +```text +Matching -> Needs review -> Ready to post -> Posted + \-> Dismissed +``` + +Clear candidates can move through matching and posting automatically. An +exception never blocks clear sibling candidates. + +### 8.3 Grouped work queue + +The office queue groups candidate cards as: + +```text +Supplier + -> Purchase order + -> Candidate documents and exceptions +``` + +Special groups are `Unknown supplier`, `Supplier known / PO unknown`, and +`PO known / line items unresolved`. Filters include site, supplier, PO, +document type, exception reason, upload age, and source batch. + +The supplier page gains exception counts and links into the filtered queue so +recurring problem suppliers are visible. + +### 8.4 Line-item linker + +On desktop, show extracted lines on the left and PO lines on the right. Each +left line displays its best candidates and evidence. The reviewer links a line +with one action; the resulting supplier-specific alias is saved for later +documents. On narrow screens use stacked cards rather than crossing connector +lines. + +Unmatched lines remain recorded but contribute zero payable/delivered value +until linked. Manual line decisions store actor, previous value, new value, +reason, and timestamp. + +## 9. Posting, idempotency, and safety + +Promotion remains the only path from a candidate into `delivery_notes`. + +Posting gates require: + +- all candidate pages are assigned; +- document identity is sufficient for deduplication or explicitly approved; +- PO is uniquely resolved; +- every financial line is linked and unit-compatible, or the unmatched line is + retained with zero financial quantity; +- duplicate checks use supplier + document number where available, rather than + a document number globally; +- the operator has approved any manual exception required by policy. + +Add a unique candidate relation on the promoted ledger row so a retry updates +the same note. Use unique indexes for source, page, candidate revision, and +promotion identities. + +Replace the current read/update/re-read claim with an atomic lock collection. +A worker claims a batch by creating `dn_pipeline_locks` with a deterministic +record id derived from the batch id. PocketBase's unique record id makes the +create atomic. Locks carry an expiry and can be reclaimed; every stage remains +idempotent even if a worker dies after acquiring one. + +Do not delete and recreate previously posted ledger lines merely because an +upstream split review changed another candidate. Posted candidates require an +explicit correction action with an audit event. + +## 10. UI wording + +Site users and routine office users see plain outcomes: + +- `Reading photos` +- `Check page grouping` +- `Supplier or PO needs help` +- `Link 2 items` +- `Ready to post` +- `Posted` + +Internal check identifiers remain available in a technical details disclosure +and logs, not as the primary card text. + +Manual movement uses explicit actions — join pages, assign supplier, choose PO, +link item, approve, dismiss — rather than unrestricted Kanban dragging. This +keeps the workflow simple while preserving the meaning of every state. + +## 11. Mobile app boundary + +The photo app should: + +1. suggest the site from coordinates and require one-tap confirmation; +2. warn about blur/darkness while allowing an override; +3. upload JPEG with capture metadata; +4. use one `delivery_session_id` for multiple photographs of the same physical + document; +5. start a new delivery session for the next document; +6. work offline by queuing immutable capture sessions until connectivity + returns. + +The two currently untracked `src/lib/image-quality*` files appear to belong to +concurrent photo-app work and are not modified by this implementation. The +server accepts their resulting JPEGs and treats client quality signals as +advisory metadata, never as proof that the document was read correctly. + +## 12. Migration and compatibility + +Use additive PocketBase migrations/scripts first: + +- create sources, pages, aliases, audit events, and processing locks; +- extend batch and candidate status values; +- add new relations/evidence fields; +- backfill each legacy `dn_batches.file` as one source lazily or with a dry-run + migration; +- map current `candidate`, `blocked`, and `promoted` states to `ready_to_post`, + `needs_review`, and `posted` in the new UI without rewriting live rows until + verified. + +The existing single-file upload, inbox URLs, ledger records, and PO/supplier +screens remain functional during rollout. Feature flags select the staged +worker and new queue until a representative production corpus passes. + +## 13. Testing and rollout + +### Automated fixtures + +- 10-page PDF with ten one-page delivery notes; +- one two-page note followed by another note; +- mixed PDF, JPEG, PNG, and WebP sources in one batch; +- rotated JPEG with EXIF orientation; +- unreadable supplier plus exact unique PO; +- duplicate PO-like values on different sites; +- ambiguous fuzzy PO candidates; +- one failed page read among successful siblings; +- manual split revision followed by a retry; +- duplicate upload and duplicate posting attempts; +- unit conversion and unmatched-line zero-value behaviour. + +### Rollout + +1. Run the new pipeline in shadow mode on copies of existing batches and + compare document counts, page coverage, PO matches, item matches, and value. +2. Enable it for one site with automatic posting limited to unique exact PO + matches and existing deterministic item rules. +3. Review false-positive and false-negative examples weekly and add aliases or + deterministic rules from evidence. +4. Expand site by site only after the exception rate and incorrect-auto-match + rate are acceptable. + +The primary safety metric is **incorrect automatic financial matches**, whose +target is zero. Exception volume is secondary; a visible unresolved document +is safer than a confident wrong one. + +## 14. Out of scope for Increment A + +- the physical receipt/invoice pairing and final amount-payable report; +- automatic cropping of several notes photographed in one frame; +- replacing MiniMax with a different vision provider; +- a full classical OCR pipeline; +- implementation inside the separate mobile app; +- unrestricted workflow dragging or automatic learning from unapproved AI + guesses.