docs: design hybrid delivery note ingestion
This commit is contained in:
DEls
parent
1dfc734dbc
commit
fb8ff612e8
1 changed files with 428 additions and 0 deletions
|
|
@ -0,0 +1,428 @@
|
|||
# Hybrid delivery-note ingestion and matching
|
||||
|
||||
> A photo-first, staged ingestion pipeline for bulk PDFs and images. It
|
||||
> preserves every uploaded source, uses vision once per page, assembles pages
|
||||
> into documents, resolves supplier and purchase order within the selected
|
||||
> site, and sends only genuine ambiguity to a human review queue.
|
||||
>
|
||||
> Date: 2026-09-18
|
||||
|
||||
## 1. Decision
|
||||
|
||||
Extend the existing delivery-note inbox rather than replace it. The current
|
||||
pipeline already has the right broad separation — uploaded batches,
|
||||
independent page reads, candidate notes, checks, and promotion — but it runs
|
||||
those stages as one operation and does not persist enough intermediate state
|
||||
for reliable split review, bulk photographs, or supplier-centred exceptions.
|
||||
|
||||
The new design is a **hybrid**:
|
||||
|
||||
- a vision model reads raster pages and photographs;
|
||||
- deterministic code normalises identifiers, assembles clear page sequences,
|
||||
resolves exact relations, converts units, and enforces posting gates;
|
||||
- a human handles only ambiguous page boundaries, suppliers, POs, or line
|
||||
items;
|
||||
- n8n schedules work and reports failures, while application code owns the
|
||||
business rules and ledger writes.
|
||||
|
||||
This is not a second OCR system. The existing MiniMax vision call is already
|
||||
reading pixels rather than a PDF text layer. Classical OCR may later be added
|
||||
as supporting searchable text, but it is not required for this increment and
|
||||
must never overrule the structured vision result merely because it returned a
|
||||
string.
|
||||
|
||||
## 2. Scope and decomposition
|
||||
|
||||
This design is intentionally split into three increments. Combining all three
|
||||
would mix ingestion, mobile capture, and accounts-payable semantics in one
|
||||
risky release.
|
||||
|
||||
### Increment A — implement now
|
||||
|
||||
1. Accept one or many PDF, JPEG, PNG, WebP, TIFF, HEIC, or HEIF sources in a
|
||||
batch, subject to decoder support and clear rejection when a format cannot
|
||||
be decoded.
|
||||
2. Preserve every original source and produce normalised page images.
|
||||
3. Persist one rich vision result per page and reuse it on retries.
|
||||
4. Assemble pages into document candidates, with a split-review screen for
|
||||
uncertain boundaries.
|
||||
5. Resolve supplier and PO using site-scoped evidence.
|
||||
6. Present a supplier/PO-grouped work queue and a two-column line-item linker.
|
||||
7. Post only validated candidates into the existing delivery-note ledger.
|
||||
|
||||
### Increment B — separate design and migration
|
||||
|
||||
Introduce a physical `receipt` record and attach delivery notes, invoices,
|
||||
collection notes, and weighbridge tickets as evidence of that receipt. This is
|
||||
needed to prevent a delivery note and later invoice for the same load from
|
||||
counting as two deliveries. It also becomes the basis of the month-end
|
||||
three-way match and payment report.
|
||||
|
||||
Increment A must not claim that the existing supplier ledger is a final amount
|
||||
payable. It remains a delivered-value operational view until Increment B is
|
||||
complete.
|
||||
|
||||
### Increment C — mobile capture integration
|
||||
|
||||
The separate site app may use GPS to suggest a site and may identify a
|
||||
multi-photo document at capture time. Increment A defines the server contract
|
||||
for that app, but does not implement the app itself.
|
||||
|
||||
## 3. Success criteria
|
||||
|
||||
1. A 20-page PDF containing 20 one-page notes produces 20 independently
|
||||
traceable document candidates.
|
||||
2. A two-page note produces one candidate with both pages attached.
|
||||
3. Fifty JPEGs uploaded together can be treated as one batch rather than 50
|
||||
unrelated batches.
|
||||
4. A JPEG and a PDF page are presented to the vision model in the same known,
|
||||
correctly labelled image format.
|
||||
5. An exact unique PO number resolves the PO, supplier, and site without
|
||||
requiring the supplier name to be readable.
|
||||
6. A weak fuzzy match never posts financial quantities automatically.
|
||||
7. One ambiguous note does not prevent clear sibling notes from reaching the
|
||||
ledger.
|
||||
8. Every manual split, merge, relation change, dismissal, and posting action is
|
||||
attributable and auditable.
|
||||
9. Retrying a stage cannot create duplicate pages, candidates, notes, or ledger
|
||||
lines.
|
||||
|
||||
## 4. Intake and source model
|
||||
|
||||
### 4.1 Batch hierarchy
|
||||
|
||||
Use four levels:
|
||||
|
||||
```text
|
||||
dn_batches
|
||||
-> dn_batch_sources original PDF or photograph
|
||||
-> dn_batch_pages normalised page/region image + vision result
|
||||
-> dn_batch_notes assembled candidate document
|
||||
-> delivery_notes and delivery_note_items after posting
|
||||
```
|
||||
|
||||
`dn_batches.file` remains readable for legacy rows, but new uploads write
|
||||
originals to `dn_batch_sources`. This is necessary because PocketBase's current
|
||||
batch file field accepts only one file, while the mobile and bulk flows need
|
||||
many photographs in one batch.
|
||||
|
||||
### 4.2 Upload contract
|
||||
|
||||
`POST /api/delivery-notes/upload` continues to accept multipart form data and
|
||||
gains a batch mode:
|
||||
|
||||
- `file`: repeated source files;
|
||||
- `site`: required authoritative site chosen or confirmed by the operator;
|
||||
- `source`: `web`, `mobile`, or `email`;
|
||||
- `capture_session_id`: optional idempotency key supplied by the mobile app;
|
||||
- `delivery_session_id`: optional hint that several photographs belong to one
|
||||
physical document;
|
||||
- `captured_at`, `latitude`, `longitude`, `location_accuracy_m`: optional
|
||||
capture evidence.
|
||||
|
||||
Coordinates suggest a site; they do not silently override the confirmed site.
|
||||
GPS can be unavailable, inaccurate, or captured later from the site office.
|
||||
|
||||
The current endpoint's single-file response remains compatible. Multiple
|
||||
files default to one batch unless the caller explicitly requests one batch per
|
||||
file. This allows bulk photographs to be assembled across adjacent images.
|
||||
|
||||
### 4.3 Image normalisation
|
||||
|
||||
Every source is decoded before the model sees it:
|
||||
|
||||
- PDFs are rendered page-by-page with Poppler.
|
||||
- Raster images are auto-rotated using EXIF orientation, colour-normalised,
|
||||
and converted to PNG.
|
||||
- Large images are resized to a bounded long edge while retaining enough
|
||||
detail for small printed numbers.
|
||||
- MIME type is detected from decoded content, not trusted from the request
|
||||
header or file extension.
|
||||
- The original source is never overwritten.
|
||||
|
||||
Use a direct image library such as `sharp` for raster decoding and
|
||||
normalisation. HEIC/HEIF/TIFF support is accepted only when the deployed
|
||||
decoder can prove it can read the file; otherwise the API returns a per-file
|
||||
`unsupported_image_codec` error. The mobile app should upload JPEG for the
|
||||
most predictable path.
|
||||
|
||||
The current pass-through behaviour must be removed: JPEG bytes must not be
|
||||
labelled `image/png`, and a JPEG candidate must not be uploaded to PocketBase
|
||||
as an `application/pdf` blob.
|
||||
|
||||
## 5. Persisted page reading
|
||||
|
||||
Create `dn_batch_pages` with at least:
|
||||
|
||||
- batch and source relations;
|
||||
- source page number and optional region number;
|
||||
- normalised image file and image hash;
|
||||
- detected source MIME, dimensions, and applied rotation;
|
||||
- read status: `pending`, `reading`, `read`, or `failed`;
|
||||
- structured extraction JSON and extraction schema version;
|
||||
- model name, prompt version, and raw response/error summary;
|
||||
- optional manually assigned document group.
|
||||
|
||||
The vision schema extends the existing `PageRead` with:
|
||||
|
||||
- printed supplier name and VAT/registration identifiers;
|
||||
- printed site/project/customer references;
|
||||
- PO number candidates rather than one invented value;
|
||||
- document type, primary number, alternate numbers, and page numbering;
|
||||
- header/continuation/multiple-document signals;
|
||||
- dates, vehicle registration, receiver, weights and units;
|
||||
- complete line items, product codes, quantities, units, rates, and totals;
|
||||
- raw page transcription for search and diagnosis;
|
||||
- field-level confidence or evidence notes where the model supports them.
|
||||
|
||||
One rich read is performed per normalised page. Retries reuse a successful
|
||||
read unless an operator requests re-extraction or the prompt/schema version
|
||||
changes. A model parse error is stored against that page rather than shrinking
|
||||
the page count.
|
||||
|
||||
## 6. Document assembly
|
||||
|
||||
Assembly is its own persisted stage, not a side effect of extraction.
|
||||
|
||||
### 6.1 Deterministic first pass
|
||||
|
||||
Adjacent pages are joined when strong evidence agrees, including:
|
||||
|
||||
- the same document number;
|
||||
- explicit `page n of m` numbering;
|
||||
- a continuation page directly following a header page;
|
||||
- a shared mobile `delivery_session_id` that does not contradict printed
|
||||
identifiers.
|
||||
|
||||
A new document starts when a new header carries a different document number.
|
||||
All pages must be accounted for as grouped, open, or dismissed.
|
||||
|
||||
### 6.2 Vision-assisted ambiguity
|
||||
|
||||
Only ambiguous adjacency receives a second, narrow model decision. The model
|
||||
sees neighbouring thumbnails and their persisted reads and answers a grouping
|
||||
question; it does not re-extract all commercial fields.
|
||||
|
||||
### 6.3 Split review
|
||||
|
||||
The batch detail screen shows page thumbnails in order and the proposed groups.
|
||||
The operator can:
|
||||
|
||||
- join an adjacent page to a document;
|
||||
- split before or after a page;
|
||||
- move a page to another candidate;
|
||||
- dismiss a blank, cover, or unrelated page;
|
||||
- re-run vision on one page;
|
||||
- mark an image containing multiple physical documents for recapture or a
|
||||
later crop workflow.
|
||||
|
||||
Increment A does not attempt arbitrary automatic cropping of two documents in
|
||||
one photograph. It detects and isolates that exception without silently
|
||||
posting it. Automatic region cropping can be added after real examples exist.
|
||||
|
||||
Reassembly creates a new assembly revision and reuses page reads. It does not
|
||||
call vision again and cannot duplicate an already posted ledger note.
|
||||
|
||||
## 7. Supplier and PO resolution
|
||||
|
||||
Resolution is ordered evidence, not one fuzzy score.
|
||||
|
||||
1. Search purchase orders belonging to the confirmed batch site.
|
||||
2. Normalise PO references by case, whitespace, punctuation, and known OCR
|
||||
confusions.
|
||||
3. A unique exact normalised PO match is authoritative. It supplies the
|
||||
internal supplier relation even when the printed supplier is unreadable.
|
||||
4. Otherwise resolve supplier by VAT/registration identifier, then a learned
|
||||
supplier alias, then a unique normalised legal/trading name.
|
||||
5. Once a supplier is known, rank only that supplier's open POs on the site by
|
||||
partial PO number, date, product codes, descriptions, units, quantities,
|
||||
and printed rates.
|
||||
6. Auto-select only deterministic unique matches. Fuzzy results are candidates
|
||||
for review until production examples establish safe, measured thresholds
|
||||
and a required winner margin.
|
||||
|
||||
Add a `supplier_document_aliases` collection so an operator can teach the
|
||||
system that a printed trading name belongs to an existing supplier. Store the
|
||||
raw value, normalised key, supplier relation, source, and audit metadata.
|
||||
|
||||
`dn_batch_notes` gains the resolved supplier relation, raw supplier value,
|
||||
match method, candidate/evidence JSON, pages JSON, and an independent workflow
|
||||
status.
|
||||
|
||||
## 8. Note and line workflow
|
||||
|
||||
The batch board answers whether source files were successfully prepared and
|
||||
assembled. The note queue answers whether each resulting commercial document
|
||||
has been resolved and posted. They must not share one overloaded status.
|
||||
|
||||
### 8.1 Batch stages
|
||||
|
||||
```text
|
||||
Queued -> Preparing -> Reading -> Split review -> Complete
|
||||
\-> Failed
|
||||
```
|
||||
|
||||
`Complete` means every source page is accounted for and document candidates
|
||||
exist. It does not mean every candidate is financially posted.
|
||||
|
||||
### 8.2 Candidate stages
|
||||
|
||||
```text
|
||||
Matching -> Needs review -> Ready to post -> Posted
|
||||
\-> Dismissed
|
||||
```
|
||||
|
||||
Clear candidates can move through matching and posting automatically. An
|
||||
exception never blocks clear sibling candidates.
|
||||
|
||||
### 8.3 Grouped work queue
|
||||
|
||||
The office queue groups candidate cards as:
|
||||
|
||||
```text
|
||||
Supplier
|
||||
-> Purchase order
|
||||
-> Candidate documents and exceptions
|
||||
```
|
||||
|
||||
Special groups are `Unknown supplier`, `Supplier known / PO unknown`, and
|
||||
`PO known / line items unresolved`. Filters include site, supplier, PO,
|
||||
document type, exception reason, upload age, and source batch.
|
||||
|
||||
The supplier page gains exception counts and links into the filtered queue so
|
||||
recurring problem suppliers are visible.
|
||||
|
||||
### 8.4 Line-item linker
|
||||
|
||||
On desktop, show extracted lines on the left and PO lines on the right. Each
|
||||
left line displays its best candidates and evidence. The reviewer links a line
|
||||
with one action; the resulting supplier-specific alias is saved for later
|
||||
documents. On narrow screens use stacked cards rather than crossing connector
|
||||
lines.
|
||||
|
||||
Unmatched lines remain recorded but contribute zero payable/delivered value
|
||||
until linked. Manual line decisions store actor, previous value, new value,
|
||||
reason, and timestamp.
|
||||
|
||||
## 9. Posting, idempotency, and safety
|
||||
|
||||
Promotion remains the only path from a candidate into `delivery_notes`.
|
||||
|
||||
Posting gates require:
|
||||
|
||||
- all candidate pages are assigned;
|
||||
- document identity is sufficient for deduplication or explicitly approved;
|
||||
- PO is uniquely resolved;
|
||||
- every financial line is linked and unit-compatible, or the unmatched line is
|
||||
retained with zero financial quantity;
|
||||
- duplicate checks use supplier + document number where available, rather than
|
||||
a document number globally;
|
||||
- the operator has approved any manual exception required by policy.
|
||||
|
||||
Add a unique candidate relation on the promoted ledger row so a retry updates
|
||||
the same note. Use unique indexes for source, page, candidate revision, and
|
||||
promotion identities.
|
||||
|
||||
Replace the current read/update/re-read claim with an atomic lock collection.
|
||||
A worker claims a batch by creating `dn_pipeline_locks` with a deterministic
|
||||
record id derived from the batch id. PocketBase's unique record id makes the
|
||||
create atomic. Locks carry an expiry and can be reclaimed; every stage remains
|
||||
idempotent even if a worker dies after acquiring one.
|
||||
|
||||
Do not delete and recreate previously posted ledger lines merely because an
|
||||
upstream split review changed another candidate. Posted candidates require an
|
||||
explicit correction action with an audit event.
|
||||
|
||||
## 10. UI wording
|
||||
|
||||
Site users and routine office users see plain outcomes:
|
||||
|
||||
- `Reading photos`
|
||||
- `Check page grouping`
|
||||
- `Supplier or PO needs help`
|
||||
- `Link 2 items`
|
||||
- `Ready to post`
|
||||
- `Posted`
|
||||
|
||||
Internal check identifiers remain available in a technical details disclosure
|
||||
and logs, not as the primary card text.
|
||||
|
||||
Manual movement uses explicit actions — join pages, assign supplier, choose PO,
|
||||
link item, approve, dismiss — rather than unrestricted Kanban dragging. This
|
||||
keeps the workflow simple while preserving the meaning of every state.
|
||||
|
||||
## 11. Mobile app boundary
|
||||
|
||||
The photo app should:
|
||||
|
||||
1. suggest the site from coordinates and require one-tap confirmation;
|
||||
2. warn about blur/darkness while allowing an override;
|
||||
3. upload JPEG with capture metadata;
|
||||
4. use one `delivery_session_id` for multiple photographs of the same physical
|
||||
document;
|
||||
5. start a new delivery session for the next document;
|
||||
6. work offline by queuing immutable capture sessions until connectivity
|
||||
returns.
|
||||
|
||||
The two currently untracked `src/lib/image-quality*` files appear to belong to
|
||||
concurrent photo-app work and are not modified by this implementation. The
|
||||
server accepts their resulting JPEGs and treats client quality signals as
|
||||
advisory metadata, never as proof that the document was read correctly.
|
||||
|
||||
## 12. Migration and compatibility
|
||||
|
||||
Use additive PocketBase migrations/scripts first:
|
||||
|
||||
- create sources, pages, aliases, audit events, and processing locks;
|
||||
- extend batch and candidate status values;
|
||||
- add new relations/evidence fields;
|
||||
- backfill each legacy `dn_batches.file` as one source lazily or with a dry-run
|
||||
migration;
|
||||
- map current `candidate`, `blocked`, and `promoted` states to `ready_to_post`,
|
||||
`needs_review`, and `posted` in the new UI without rewriting live rows until
|
||||
verified.
|
||||
|
||||
The existing single-file upload, inbox URLs, ledger records, and PO/supplier
|
||||
screens remain functional during rollout. Feature flags select the staged
|
||||
worker and new queue until a representative production corpus passes.
|
||||
|
||||
## 13. Testing and rollout
|
||||
|
||||
### Automated fixtures
|
||||
|
||||
- 10-page PDF with ten one-page delivery notes;
|
||||
- one two-page note followed by another note;
|
||||
- mixed PDF, JPEG, PNG, and WebP sources in one batch;
|
||||
- rotated JPEG with EXIF orientation;
|
||||
- unreadable supplier plus exact unique PO;
|
||||
- duplicate PO-like values on different sites;
|
||||
- ambiguous fuzzy PO candidates;
|
||||
- one failed page read among successful siblings;
|
||||
- manual split revision followed by a retry;
|
||||
- duplicate upload and duplicate posting attempts;
|
||||
- unit conversion and unmatched-line zero-value behaviour.
|
||||
|
||||
### Rollout
|
||||
|
||||
1. Run the new pipeline in shadow mode on copies of existing batches and
|
||||
compare document counts, page coverage, PO matches, item matches, and value.
|
||||
2. Enable it for one site with automatic posting limited to unique exact PO
|
||||
matches and existing deterministic item rules.
|
||||
3. Review false-positive and false-negative examples weekly and add aliases or
|
||||
deterministic rules from evidence.
|
||||
4. Expand site by site only after the exception rate and incorrect-auto-match
|
||||
rate are acceptable.
|
||||
|
||||
The primary safety metric is **incorrect automatic financial matches**, whose
|
||||
target is zero. Exception volume is secondary; a visible unresolved document
|
||||
is safer than a confident wrong one.
|
||||
|
||||
## 14. Out of scope for Increment A
|
||||
|
||||
- the physical receipt/invoice pairing and final amount-payable report;
|
||||
- automatic cropping of several notes photographed in one frame;
|
||||
- replacing MiniMax with a different vision provider;
|
||||
- a full classical OCR pipeline;
|
||||
- implementation inside the separate mobile app;
|
||||
- unrestricted workflow dragging or automatic learning from unapproved AI
|
||||
guesses.
|
||||
Loading…
Reference in a new issue