From dd0ac47375877bf7e37bf94c749b3a67495514ad Mon Sep 17 00:00:00 2001 From: DEls Date: Thu, 17 Sep 2026 12:51:03 +0200 Subject: [PATCH] docs(delivery-notes): add the phase 2-3 inbox pipeline implementation plan Seventeen tasks covering the batch collections, the pure segmentation and check modules, page rendering via poppler, per-page extraction, promotion, the orchestrator, the tick route, the Kanban board and the batch detail view with note correction. The design point: pages are read independently and grouped into notes by the delivery note numbers printed on them, so a note spanning three pages is one note and a page belonging to no note blocks the batch. segmentPages throws rather than return a segmentation that loses a page, and runBatch cannot reach "processed" unless notes_promoted equals notes_detected -- the check whose absence let two notes vanish from scan_20260915 while the pipeline reported success at 95% confidence. Task 16 step 7 is the regression test for exactly that: force a page read to fail and confirm the batch lands in needs_attention rather than processed. Co-Authored-By: Claude Opus 5 --- ...2026-09-17-delivery-note-inbox-pipeline.md | 3600 +++++++++++++++++ 1 file changed, 3600 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-17-delivery-note-inbox-pipeline.md diff --git a/docs/superpowers/plans/2026-09-17-delivery-note-inbox-pipeline.md b/docs/superpowers/plans/2026-09-17-delivery-note-inbox-pipeline.md new file mode 100644 index 0000000..45dcba1 --- /dev/null +++ b/docs/superpowers/plans/2026-09-17-delivery-note-inbox-pipeline.md @@ -0,0 +1,3600 @@ +# Delivery Note Inbox Pipeline Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Give uploaded delivery-note documents a staging inbox with a deterministic validation gate, so a multi-note document cannot report success while notes from it are missing — and show the whole thing on a Kanban board. + +**Architecture:** A batch is the uploaded file. It moves `inbox → analysing → needs_attention | processed`. Analysis splits the document into pages, reads each page independently, then **groups pages into notes by the delivery-note numbers printed on them** — so a note spanning three pages is one note, and a page belonging to no note blocks the batch. Every note passes a set of named checks before anything is written to `delivery_notes`. The pure logic (segmentation, checks) lives in small tested modules; the I/O (page rendering, model calls, PocketBase writes) lives in thin wrappers around them. n8n shrinks to one HTTP call. + +**Tech Stack:** TypeScript, Next.js 16 App Router, Vitest, PocketBase, `pdf-lib` (npm), `poppler-utils` (`pdfinfo`/`pdftoppm`, Alpine package), MiniMax via the OpenAI-compatible `/v1` endpoint, Tailwind + the existing `src/components/ui` primitives. + +**Spec:** [docs/superpowers/specs/2026-09-17-delivery-note-inbox-design.md](../specs/2026-09-17-delivery-note-inbox-design.md) §4 and §5. + +**Depends on:** [2026-09-17-delivery-note-unit-correctness.md](./2026-09-17-delivery-note-unit-correctness.md) (phase 1). Task 12 here imports `buildDeliveryNoteItemRows` from `src/lib/dn-item-row.ts`, which that plan creates. Do not start Task 12 until phase 1 has landed. + +--- + +## Background you need + +The current n8n workflow makes **exactly two callbacks regardless of how many delivery notes a document contains**, because its `Match items to PO` Code node runs in *Run Once for All Items* mode and returns a single item. A four-note scan produced two populated records and two empty ones in a `processing_status` the dashboard cannot display, while reporting `auto-verified` at 95 % confidence. + +The fix is not a better prompt. It is **counting**: every page must be accounted for, and `notes_promoted` must equal `notes_detected`, or the batch cannot leave `analysing`. + +Your reference document is `scan_20260915_c5p7fkmzgv.pdf` (attached to `delivery_notes/u27fj2ddxgs138l`). Four pages, four Raumix weighbridge tickets, one note per page, all against PO `MEGCPO9165`: + +| Page | Note | Quantity | Gross | Tare | Nett | +| --- | --- | --- | --- | --- | --- | +| 1 | DQ0010107 | 29 960 kg | 48 760 | 18 800 | 29 960 | +| 2 | DQ0010099 | 34 820 kg | — | — | — | +| 3 | DQ0010239 | 34 200 kg | 56 440 | 22 240 | 34 200 | +| 4 | DQ0010217 | 34 520 kg | 56 640 | 22 120 | 34 520 | + +Note the arithmetic: **gross − tare = nett = quantity**. That is a deterministic correctness proof that needs no model confidence. Also note `Order No: #MEGCPO9165` — suppliers print a `#` prefix, which is where the phantom PO `#MEGCP09109` in your `purchase_orders` table came from. + +These PDFs have **no text layer**. They are pure scans, so vision extraction is genuinely required. + +## File structure + +| File | Responsibility | +| --- | --- | +| `src/lib/dn-pipeline/types.ts` | Shared types for the pipeline. No logic. | +| `src/lib/dn-pipeline/segment.ts` | Pages → notes. Pure. The heart of the fix. | +| `src/lib/dn-pipeline/checks.ts` | Named pass/fail checks. Pure. | +| `src/lib/dn-pipeline/split.ts` | PDF → page PNGs, page range → per-note PDF. I/O. | +| `src/lib/dn-pipeline/extract.ts` | One page PNG → one `PageRead`. I/O. | +| `src/lib/dn-pipeline/promote.ts` | Validated note → `delivery_notes` rows. I/O. | +| `src/lib/dn-pipeline/run.ts` | Orchestrates one batch through the stages. | +| `src/app/api/dn-batches/**` | HTTP surface. | +| `src/app/dashboard/financial/delivery-notes/inbox/**` | The board and the batch detail view. | + +--- + +## Task 1: Dependencies and the Docker image + +`split.ts` shells out to `pdfinfo` and `pdftoppm`, which are not in `node:20-alpine`. Get this in before anything depends on it. + +**Files:** +- Modify: `Dockerfile` +- Modify: `package.json` + +- [ ] **Step 1: Install pdf-lib** + +```bash +npm install pdf-lib +``` + +Expected: `package.json` gains `"pdf-lib": "^1.17.1"` (or newer) under `dependencies`. + +- [ ] **Step 2: Add poppler to the builder stage** + +In `Dockerfile`, the builder stage currently begins: + +```dockerfile +FROM node:20-alpine AS builder +WORKDIR /app +``` + +Change it to: + +```dockerfile +FROM node:20-alpine AS builder +# poppler-utils provides pdfinfo + pdftoppm, used by +# src/lib/dn-pipeline/split.ts to rasterise delivery note pages. The +# MiniMax OpenAI-compatible endpoint does not accept PDF documents, so +# pages must become images before extraction. +RUN apk add --no-cache poppler-utils +WORKDIR /app +``` + +- [ ] **Step 3: Add poppler to the runner stage** + +The runner stage currently begins: + +```dockerfile +FROM node:20-alpine AS runner +WORKDIR /app +``` + +Change it to: + +```dockerfile +FROM node:20-alpine AS runner +# Same reason as the builder stage — the running server shells out to +# pdftoppm on every batch. +RUN apk add --no-cache poppler-utils +WORKDIR /app +``` + +- [ ] **Step 4: Verify locally** + +```bash +which pdfinfo pdftoppm +``` + +Expected: two paths. If absent on macOS: `brew install poppler`. + +- [ ] **Step 5: Verify in the image** + +```bash +docker build -t mega-poppler-check . && \ + docker run --rm --user nextjs mega-poppler-check pdftoppm -v +``` + +Expected: a version banner such as `pdftoppm version 24.x`. It must work **as the `nextjs` user**, not just as root — the server does not run as root. + +- [ ] **Step 6: Commit** + +```bash +git add package.json package-lock.json Dockerfile +git commit -m "build: add poppler-utils and pdf-lib for delivery note page splitting" +``` + +--- + +## Task 2: PocketBase collections + +**Files:** +- Create: `scripts/pb-add-dn-batches.py` +- Modify: `src/lib/pb-types.ts` + +- [ ] **Step 1: Write the schema script** + +Create `scripts/pb-add-dn-batches.py`: + +```python +#!/usr/bin/env python3 +""" +Creates the DN_BATCHES and DN_BATCH_NOTES collections, and adds the +batch back-references to DELIVERY_NOTES. + +A batch is the uploaded file. Notes found inside it live in +DN_BATCH_NOTES as candidates until they pass every check — only then is +a DELIVERY_NOTES row created. That separation is the point: before it, +delivery_notes was both the inbox and the ledger, so a half-read +document sat inside the figures used to pay suppliers. + +Idempotent — existing collections and fields are left alone. +""" +import sys + +from pb_env import PB, http, superuser_token + + +def get_collection(token, name): + code, resp = http("GET", f"/api/collections/{name}", token=token) + return (resp if code == 200 else None) + + +def collection_id(token, name): + coll = get_collection(token, name) + if not coll: + sys.exit(f"collection {name} not found — cannot resolve its id") + return coll["id"] + + +def create_collection(token, payload): + existing = get_collection(token, payload["name"]) + if existing: + print(f" • {payload['name']} already exists — skipping create") + return existing["id"] + code, resp = http("POST", "/api/collections", payload, token) + if code != 200: + sys.exit(f"CREATE {payload['name']} failed: {code} {resp}") + print(f" ✓ created {payload['name']}") + return resp["id"] + + +def add_fields(token, name, wanted): + coll = get_collection(token, name) + if not coll: + sys.exit(f"collection {name} not found") + fields = list(coll.get("fields", [])) + existing = {f.get("name") for f in fields} + additions = [w for w in wanted if w["name"] not in existing] + if not additions: + print(f" • {name}: no new fields needed") + return + code, resp = http("PATCH", f"/api/collections/{coll['id']}", {"fields": fields + additions}, token) + if code != 200: + sys.exit(f"PATCH {name} failed: {code} {resp}") + for a in additions: + print(f" ✓ {name}: added {a['name']}") + + +def main(): + token = superuser_token() + print(f"\n→ {PB}") + + sites_id = collection_id(token, "sites") + pos_id = collection_id(token, "purchase_orders") + dns_id = collection_id(token, "delivery_notes") + + batches_id = create_collection(token, { + "name": "DN_BATCHES", + "type": "base", + "fields": [ + {"name": "file", "type": "file", "required": False, "maxSelect": 1, "maxSize": 20971520}, + {"name": "file_hash", "type": "text", "required": False}, + {"name": "site", "type": "relation", "required": True, + "collectionId": sites_id, "maxSelect": 1, "cascadeDelete": False}, + {"name": "source", "type": "select", "required": False, "maxSelect": 1, + "values": ["web", "mobile", "email"]}, + {"name": "uploaded_by", "type": "text", "required": False}, + {"name": "status", "type": "select", "required": False, "maxSelect": 1, + "values": ["inbox", "analysing", "needs_attention", "processed"]}, + {"name": "page_count", "type": "number", "required": False}, + {"name": "notes_detected", "type": "number", "required": False}, + {"name": "notes_promoted", "type": "number", "required": False}, + {"name": "segmentation", "type": "json", "required": False, "maxSize": 2000000}, + {"name": "checks", "type": "json", "required": False, "maxSize": 2000000}, + {"name": "attempts", "type": "number", "required": False}, + {"name": "claimed_at", "type": "date", "required": False}, + {"name": "last_error", "type": "text", "required": False}, + ], + }) + + create_collection(token, { + "name": "DN_BATCH_NOTES", + "type": "base", + "fields": [ + {"name": "batch", "type": "relation", "required": True, + "collectionId": batches_id, "maxSelect": 1, "cascadeDelete": True}, + {"name": "note_index", "type": "number", "required": False}, + {"name": "page_from", "type": "number", "required": False}, + {"name": "page_to", "type": "number", "required": False}, + {"name": "page_file", "type": "file", "required": False, "maxSelect": 1, "maxSize": 20971520}, + {"name": "delivery_note_number", "type": "text", "required": False}, + {"name": "po_number_raw", "type": "text", "required": False}, + {"name": "po_number_normalised", "type": "text", "required": False}, + {"name": "purchase_order", "type": "relation", "required": False, + "collectionId": pos_id, "maxSelect": 1, "cascadeDelete": False}, + {"name": "gross_mass", "type": "number", "required": False}, + {"name": "tare_mass", "type": "number", "required": False}, + {"name": "nett_mass", "type": "number", "required": False}, + {"name": "quantity", "type": "number", "required": False}, + {"name": "unit", "type": "text", "required": False}, + {"name": "items", "type": "json", "required": False, "maxSize": 2000000}, + {"name": "checks", "type": "json", "required": False, "maxSize": 2000000}, + {"name": "status", "type": "select", "required": False, "maxSelect": 1, + "values": ["candidate", "blocked", "promoted"]}, + {"name": "delivery_note", "type": "relation", "required": False, + "collectionId": dns_id, "maxSelect": 1, "cascadeDelete": False}, + ], + }) + + add_fields(token, "delivery_notes", [ + {"name": "batch", "type": "relation", "required": False, + "collectionId": batches_id, "maxSelect": 1, "cascadeDelete": False}, + {"name": "page_from", "type": "number", "required": False}, + {"name": "page_to", "type": "number", "required": False}, + ]) + + +if __name__ == "__main__": + main() + print("\nDone.") +``` + +- [ ] **Step 2: Run it** + +```bash +python3 scripts/pb-add-dn-batches.py +``` + +Expected: + +``` +auth ok on https://pb.mega.infra.apphub.co.za + +→ https://pb.mega.infra.apphub.co.za + ✓ created DN_BATCHES + ✓ created DN_BATCH_NOTES + ✓ delivery_notes: added batch + ✓ delivery_notes: added page_from + ✓ delivery_notes: added page_to + +Done. +``` + +- [ ] **Step 3: Verify idempotency** + +Run the same command again. Expected: `already exists — skipping create` twice and `no new fields needed`. + +- [ ] **Step 4: Add the record types** + +Append to `src/lib/pb-types.ts`: + +```ts +export type DnBatchStatus = + | "inbox" + | "analysing" + | "needs_attention" + | "processed"; + +export interface DnBatchRecord extends RecordModel { + collectionName: "dn_batches"; + file: string; + file_hash: string; + site: string; + source: "web" | "mobile" | "email" | ""; + uploaded_by: string; + status: DnBatchStatus; + page_count: number; + notes_detected: number; + notes_promoted: number; + segmentation: unknown; + checks: unknown; + attempts: number; + claimed_at: string; + last_error: string; + expand?: { site?: SiteRecord }; +} + +export type DnBatchNoteStatus = "candidate" | "blocked" | "promoted"; + +export interface DnBatchNoteRecord extends RecordModel { + collectionName: "dn_batch_notes"; + batch: string; + note_index: number; + page_from: number; + page_to: number; + page_file: string; + delivery_note_number: string; + po_number_raw: string; + po_number_normalised: string; + purchase_order: string; + gross_mass: number; + tare_mass: number; + nett_mass: number; + quantity: number; + unit: string; + items: unknown; + checks: unknown; + status: DnBatchNoteStatus; + delivery_note: string; +} +``` + +If `SiteRecord` is not the exact name of the sites interface in that file, use whatever it is called — grep for `collectionName: "sites"`. + +- [ ] **Step 5: Typecheck and commit** + +```bash +npm run typecheck +git add scripts/pb-add-dn-batches.py src/lib/pb-types.ts +git commit -m "feat(delivery-notes): add dn_batches and dn_batch_notes collections" +``` + +--- + +## Task 3: Pipeline types + +**Files:** +- Create: `src/lib/dn-pipeline/types.ts` + +- [ ] **Step 1: Write the types** + +Create `src/lib/dn-pipeline/types.ts`: + +```ts +/** + * Shared shapes for the delivery note inbox pipeline. No logic here — + * segment.ts and checks.ts are pure functions over these types, and the + * I/O modules produce and consume them. + */ + +export interface PageItem { + description: string; + qty: number; + unit: string | null; + unit_price: number | null; +} + +/** + * What one page of a document says, read in isolation. The extractor is + * instructed never to infer anything from a neighbouring page — the + * grouping is segment.ts's job, and it can only do it honestly if each + * page reports only what is printed on it. + */ +export interface PageRead { + page: number; + /** Every delivery note number printed on this page. Often exactly one. */ + delivery_note_numbers: string[]; + /** True when the page carries a supplier letterhead / document title. */ + has_header: boolean; + /** True when the page is line items continuing from a previous page. */ + is_continuation: boolean; + po_number_raw: string | null; + customer_vat_no: string | null; + date: string | null; + received_by_name: string | null; + gross_mass: number | null; + tare_mass: number | null; + nett_mass: number | null; + quantity: number | null; + unit: string | null; + items: PageItem[]; +} + +export interface NoteSegment { + note_index: number; + delivery_note_number: string; + page_from: number; + page_to: number; + /** Page numbers belonging to this note, ascending. */ + pages: number[]; + po_number_raw: string | null; + customer_vat_no: string | null; + date: string | null; + received_by_name: string | null; + gross_mass: number | null; + tare_mass: number | null; + nett_mass: number | null; + quantity: number | null; + unit: string | null; + items: PageItem[]; +} + +export interface Segmentation { + notes: NoteSegment[]; + /** Pages belonging to no note. Any entry here blocks the batch. */ + unassigned: number[]; + /** Pages showing more than one note number. Any entry blocks the batch. */ + needsSubsplit: number[]; +} + +export interface CheckResult { + id: string; + /** null means "not applicable to this document" — neither pass nor fail. */ + passed: boolean | null; + reason: string; +} +``` + +- [ ] **Step 2: Typecheck and commit** + +```bash +npm run typecheck +git add src/lib/dn-pipeline/types.ts +git commit -m "feat(dn-pipeline): add shared pipeline types" +``` + +--- + +## Task 4: Segmentation + +This is the module that would have caught the original bug. Every page must end up in exactly one note, in `unassigned`, or in `needsSubsplit` — and the function asserts that before returning. + +**Files:** +- Create: `src/lib/dn-pipeline/segment.ts` +- Test: `src/lib/dn-pipeline/segment.test.ts` + +- [ ] **Step 1: Write the failing test** + +Create `src/lib/dn-pipeline/segment.test.ts`: + +```ts +import { describe, it, expect } from "vitest"; +import { segmentPages } from "./segment"; +import type { PageRead } from "./types"; + +function page(n: number, over: Partial = {}): PageRead { + return { + page: n, + delivery_note_numbers: [], + has_header: true, + is_continuation: false, + po_number_raw: null, + customer_vat_no: null, + date: null, + received_by_name: null, + gross_mass: null, + tare_mass: null, + nett_mass: null, + quantity: null, + unit: null, + items: [], + ...over, + }; +} + +/** Every page must be accounted for exactly once. */ +function assertTotalCoverage(pages: PageRead[], seg: ReturnType) { + const seen = new Set(); + for (const n of seg.notes) for (const p of n.pages) { + expect(seen.has(p)).toBe(false); + seen.add(p); + } + for (const p of [...seg.unassigned, ...seg.needsSubsplit]) { + expect(seen.has(p)).toBe(false); + seen.add(p); + } + expect([...seen].sort((a, b) => a - b)).toEqual(pages.map((p) => p.page)); +} + +describe("segmentPages", () => { + it("treats four single-page notes as four notes", () => { + const pages = [ + page(1, { delivery_note_numbers: ["DQ0010107"], quantity: 29960, unit: "kg" }), + page(2, { delivery_note_numbers: ["DQ0010099"], quantity: 34820, unit: "kg" }), + page(3, { delivery_note_numbers: ["DQ0010239"], quantity: 34200, unit: "kg" }), + page(4, { delivery_note_numbers: ["DQ0010217"], quantity: 34520, unit: "kg" }), + ]; + const seg = segmentPages(pages); + expect(seg.notes).toHaveLength(4); + expect(seg.notes.map((n) => n.delivery_note_number)).toEqual([ + "DQ0010107", "DQ0010099", "DQ0010239", "DQ0010217", + ]); + expect(seg.notes[0]).toMatchObject({ page_from: 1, page_to: 1, note_index: 0 }); + expect(seg.unassigned).toEqual([]); + assertTotalCoverage(pages, seg); + }); + + it("merges consecutive pages carrying the same note number into one note", () => { + const pages = [ + page(1, { delivery_note_numbers: ["DQ0010107"], items: [ + { description: "G6", qty: 10, unit: "kg", unit_price: null }, + ] }), + page(2, { delivery_note_numbers: ["DQ0010107"], items: [ + { description: "G1", qty: 5, unit: "kg", unit_price: null }, + ] }), + page(3, { delivery_note_numbers: ["DQ0010099"] }), + ]; + const seg = segmentPages(pages); + expect(seg.notes).toHaveLength(2); + expect(seg.notes[0]).toMatchObject({ + delivery_note_number: "DQ0010107", page_from: 1, page_to: 2, pages: [1, 2], + }); + expect(seg.notes[0].items).toHaveLength(2); + assertTotalCoverage(pages, seg); + }); + + it("attaches a header-less continuation page to the note before it", () => { + const pages = [ + page(1, { delivery_note_numbers: ["DQ0010107"], items: [ + { description: "G6", qty: 10, unit: "kg", unit_price: null }, + ] }), + page(2, { + delivery_note_numbers: [], + has_header: false, + is_continuation: true, + items: [{ description: "G1", qty: 5, unit: "kg", unit_price: null }], + }), + ]; + const seg = segmentPages(pages); + expect(seg.notes).toHaveLength(1); + expect(seg.notes[0]).toMatchObject({ page_from: 1, page_to: 2, pages: [1, 2] }); + expect(seg.notes[0].items).toHaveLength(2); + assertTotalCoverage(pages, seg); + }); + + it("flags a page showing two note numbers rather than guessing", () => { + const pages = [ + page(1, { delivery_note_numbers: ["DQ0010107", "DQ0010099"] }), + page(2, { delivery_note_numbers: ["DQ0010239"] }), + ]; + const seg = segmentPages(pages); + expect(seg.needsSubsplit).toEqual([1]); + expect(seg.notes).toHaveLength(1); + assertTotalCoverage(pages, seg); + }); + + it("marks a leading continuation page unassigned rather than inventing a note", () => { + const pages = [ + page(1, { delivery_note_numbers: [], has_header: false, is_continuation: true }), + page(2, { delivery_note_numbers: ["DQ0010107"] }), + ]; + const seg = segmentPages(pages); + expect(seg.unassigned).toEqual([1]); + expect(seg.notes).toHaveLength(1); + assertTotalCoverage(pages, seg); + }); + + it("marks a blank page with no header and no items unassigned", () => { + const pages = [ + page(1, { delivery_note_numbers: ["DQ0010107"] }), + page(2, { delivery_note_numbers: [], has_header: true, is_continuation: false }), + ]; + const seg = segmentPages(pages); + expect(seg.unassigned).toEqual([2]); + assertTotalCoverage(pages, seg); + }); + + it("does not merge the same note number when another note interrupts it", () => { + const pages = [ + page(1, { delivery_note_numbers: ["DQ0010107"] }), + page(2, { delivery_note_numbers: ["DQ0010099"] }), + page(3, { delivery_note_numbers: ["DQ0010107"] }), + ]; + const seg = segmentPages(pages); + expect(seg.notes).toHaveLength(3); + assertTotalCoverage(pages, seg); + }); + + it("takes header fields from the first page of a note", () => { + const pages = [ + page(1, { + delivery_note_numbers: ["DQ0010107"], + po_number_raw: "#MEGCPO9165", + gross_mass: 48760, tare_mass: 18800, nett_mass: 29960, + quantity: 29960, unit: "kg", + }), + page(2, { delivery_note_numbers: ["DQ0010107"], po_number_raw: null }), + ]; + const seg = segmentPages(pages); + expect(seg.notes[0]).toMatchObject({ + po_number_raw: "#MEGCPO9165", gross_mass: 48760, quantity: 29960, unit: "kg", + }); + }); + + it("returns an empty segmentation for no pages", () => { + const seg = segmentPages([]); + expect(seg).toEqual({ notes: [], unassigned: [], needsSubsplit: [] }); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run src/lib/dn-pipeline/segment.test.ts` +Expected: FAIL — `Failed to resolve import "./segment"`. + +- [ ] **Step 3: Write minimal implementation** + +Create `src/lib/dn-pipeline/segment.ts`: + +```ts +/** + * Groups independently-read pages into delivery notes. + * + * The unit of reading is the page; the unit of output is the note. A + * note spanning three pages is one note, and a page belonging to no + * note blocks the whole batch rather than being silently dropped — + * that silent drop is exactly how two notes disappeared from + * scan_20260915_c5p7fkmzgv.pdf while the pipeline reported success. + * + * Pure. No I/O, no model calls, no PocketBase. + */ +import type { NoteSegment, PageRead, Segmentation } from "./types"; + +export function segmentPages(reads: readonly PageRead[]): Segmentation { + const notes: NoteSegment[] = []; + const unassigned: number[] = []; + const needsSubsplit: number[] = []; + + for (const read of reads) { + const numbers = read.delivery_note_numbers.filter((n) => n && n.trim()); + + // Rule 4: ambiguous page. We will not guess which items belong to + // which note — a human or the agent escalation resolves it. + if (numbers.length > 1) { + needsSubsplit.push(read.page); + continue; + } + + if (numbers.length === 1) { + const number = numbers[0].trim(); + const open = notes[notes.length - 1]; + // Rule 2: the SAME number on the immediately preceding page means + // one note spanning both. A number seen earlier but interrupted by + // another note is a different delivery, not a continuation. + if (open && open.delivery_note_number === number && open.page_to === read.page - 1) { + appendPage(open, read); + } else { + notes.push(startNote(notes.length, number, read)); + } + continue; + } + + // Rule 3: no number and no header — line items continuing from the + // page before. Requires an open note directly above it. + const open = notes[notes.length - 1]; + const continues = + (read.is_continuation || !read.has_header) && + open !== undefined && + open.page_to === read.page - 1; + if (continues) { + appendPage(open, read); + continue; + } + + // Rule 5: belongs nowhere. + unassigned.push(read.page); + } + + assertTotalCoverage(reads, notes, unassigned, needsSubsplit); + return { notes, unassigned, needsSubsplit }; +} + +function startNote(index: number, number: string, read: PageRead): NoteSegment { + return { + note_index: index, + delivery_note_number: number, + page_from: read.page, + page_to: read.page, + pages: [read.page], + // Header fields come from the note's first page and are never + // overwritten by a later one — a continuation page repeating a + // half-legible header must not clobber a clean read. + po_number_raw: read.po_number_raw, + customer_vat_no: read.customer_vat_no, + date: read.date, + received_by_name: read.received_by_name, + gross_mass: read.gross_mass, + tare_mass: read.tare_mass, + nett_mass: read.nett_mass, + quantity: read.quantity, + unit: read.unit, + items: [...read.items], + }; +} + +function appendPage(note: NoteSegment, read: PageRead): void { + note.pages.push(read.page); + note.page_to = read.page; + note.items.push(...read.items); + // Fill header fields only where the first page left a gap. + note.po_number_raw ??= read.po_number_raw; + note.customer_vat_no ??= read.customer_vat_no; + note.date ??= read.date; + note.received_by_name ??= read.received_by_name; + note.gross_mass ??= read.gross_mass; + note.tare_mass ??= read.tare_mass; + note.nett_mass ??= read.nett_mass; + note.quantity ??= read.quantity; + note.unit ??= read.unit; +} + +/** + * Every page appears exactly once across notes, unassigned and + * needsSubsplit. A violation is a programming error in the rules above, + * not a data problem — throwing is correct, because the alternative is + * a batch that quietly reports fewer notes than the document holds. + */ +function assertTotalCoverage( + reads: readonly PageRead[], + notes: NoteSegment[], + unassigned: number[], + needsSubsplit: number[], +): void { + const seen = new Set(); + const claim = (p: number) => { + if (seen.has(p)) throw new Error(`segmentPages: page ${p} claimed twice`); + seen.add(p); + }; + for (const n of notes) for (const p of n.pages) claim(p); + for (const p of unassigned) claim(p); + for (const p of needsSubsplit) claim(p); + for (const r of reads) { + if (!seen.has(r.page)) { + throw new Error(`segmentPages: page ${r.page} unaccounted for`); + } + } + if (seen.size !== reads.length) { + throw new Error( + `segmentPages: accounted for ${seen.size} pages but read ${reads.length}`, + ); + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run src/lib/dn-pipeline/segment.test.ts` +Expected: PASS, 9 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/dn-pipeline/segment.ts src/lib/dn-pipeline/segment.test.ts +git commit -m "feat(dn-pipeline): group pages into notes with total page coverage + +Every page ends in exactly one note, unassigned, or needs-subsplit, and +the function throws rather than return a segmentation that loses one." +``` + +--- + +## Task 5: Checks + +**Files:** +- Create: `src/lib/dn-pipeline/checks.ts` +- Test: `src/lib/dn-pipeline/checks.test.ts` + +- [ ] **Step 1: Write the failing test** + +Create `src/lib/dn-pipeline/checks.test.ts`: + +```ts +import { describe, it, expect } from "vitest"; +import { runNoteChecks, runBatchChecks } from "./checks"; +import type { NoteSegment } from "./types"; + +function note(over: Partial = {}): NoteSegment { + return { + note_index: 0, + delivery_note_number: "DQ0010107", + page_from: 1, + page_to: 1, + pages: [1], + po_number_raw: "#MEGCPO9165", + customer_vat_no: "4280307796", + date: "2026-09-08", + received_by_name: "Peter Khabo", + gross_mass: 48760, + tare_mass: 18800, + nett_mass: 29960, + quantity: 29960, + unit: "kg", + items: [{ description: "G6 Sub Base", qty: 29960, unit: "kg", unit_price: null }], + ...over, + }; +} + +const CTX = { + customerVatNo: "4280307796", + purchaseOrderId: "y35epf1bpx9vai8", + poItems: [{ id: "bxqbo8rs04pjqwc", description: "G6 Sub Base", unit: "ton" }], + existingNoteNumbers: new Set(), +}; + +function byId(results: ReturnType, id: string) { + const r = results.find((x) => x.id === id); + if (!r) throw new Error(`no check named ${id}`); + return r; +} + +describe("runNoteChecks", () => { + it("passes a clean Raumix weighbridge ticket", () => { + const results = runNoteChecks(note(), CTX); + expect(results.every((r) => r.passed !== false)).toBe(true); + }); + + it("fails when gross minus tare does not equal nett", () => { + const r = byId(runNoteChecks(note({ tare_mass: 18000 }), CTX), "weighbridge_arithmetic"); + expect(r.passed).toBe(false); + expect(r.reason).toContain("48760"); + }); + + it("fails when nett does not equal the stated quantity", () => { + const r = byId(runNoteChecks(note({ quantity: 29000 }), CTX), "weighbridge_arithmetic"); + expect(r.passed).toBe(false); + }); + + it("tolerates a rounding difference of one unit", () => { + const r = byId(runNoteChecks(note({ quantity: 29961 }), CTX), "weighbridge_arithmetic"); + expect(r.passed).toBe(true); + }); + + it("skips the arithmetic when the slip has no weighbridge section", () => { + const r = byId( + runNoteChecks(note({ gross_mass: null, tare_mass: null, nett_mass: null }), CTX), + "weighbridge_arithmetic", + ); + expect(r.passed).toBeNull(); + }); + + it("fails when the customer VAT number is not ours", () => { + const r = byId(runNoteChecks(note({ customer_vat_no: "9999999999" }), CTX), "customer_identity"); + expect(r.passed).toBe(false); + }); + + it("skips customer identity when the slip shows no VAT number", () => { + const r = byId(runNoteChecks(note({ customer_vat_no: null }), CTX), "customer_identity"); + expect(r.passed).toBeNull(); + }); + + it("fails when no purchase order resolved", () => { + const r = byId(runNoteChecks(note(), { ...CTX, purchaseOrderId: null }), "po_resolved"); + expect(r.passed).toBe(false); + expect(r.reason).toContain("MEGCPO9165"); + }); + + it("fails when a line item matches nothing on the PO", () => { + const n = note({ items: [{ description: "Unobtainium", qty: 1, unit: "kg", unit_price: null }] }); + const r = byId(runNoteChecks(n, CTX), "product_matched"); + expect(r.passed).toBe(false); + expect(r.reason).toContain("Unobtainium"); + }); + + it("passes units_convertible for kg against a ton PO line", () => { + expect(byId(runNoteChecks(note(), CTX), "units_convertible").passed).toBe(true); + }); + + it("fails units_convertible for an incompatible pair", () => { + const n = note({ items: [{ description: "G6 Sub Base", qty: 3, unit: "m3", unit_price: null }] }); + const r = byId(runNoteChecks(n, CTX), "units_convertible"); + expect(r.passed).toBe(false); + }); + + it("fails when the note number is already in delivery_notes", () => { + const ctx = { ...CTX, existingNoteNumbers: new Set(["DQ0010107"]) }; + const r = byId(runNoteChecks(note(), ctx), "not_duplicate"); + expect(r.passed).toBe(false); + }); + + it("fails when there is no note number at all", () => { + const r = byId(runNoteChecks(note({ delivery_note_number: "" }), CTX), "not_duplicate"); + expect(r.passed).toBe(false); + }); +}); + +describe("runBatchChecks", () => { + it("passes when every page is accounted for and every note promoted", () => { + const results = runBatchChecks({ + unassigned: [], needsSubsplit: [], notesDetected: 4, notesPromoted: 4, + }); + expect(results.every((r) => r.passed)).toBe(true); + }); + + it("fails pages_accounted when a page belongs nowhere", () => { + const r = runBatchChecks({ + unassigned: [3], needsSubsplit: [], notesDetected: 3, notesPromoted: 3, + }).find((x) => x.id === "pages_accounted")!; + expect(r.passed).toBe(false); + expect(r.reason).toContain("3"); + }); + + it("fails notes_reconciled when fewer notes promoted than detected", () => { + const r = runBatchChecks({ + unassigned: [], needsSubsplit: [], notesDetected: 4, notesPromoted: 2, + }).find((x) => x.id === "notes_reconciled")!; + expect(r.passed).toBe(false); + expect(r.reason).toContain("2 of 4"); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run src/lib/dn-pipeline/checks.test.ts` +Expected: FAIL — `Failed to resolve import "./checks"`. + +- [ ] **Step 3: Write minimal implementation** + +Create `src/lib/dn-pipeline/checks.ts`: + +```ts +/** + * Named, deterministic checks over a segmented delivery note. + * + * A check returns `passed: null` when it does not apply to the document + * — not every supplier issues weighbridge tickets, and a skipped check + * must not read as a failure. Only `false` blocks a note. + * + * Nothing here consults a model's self-reported confidence. Confidence + * was what let a four-note document report success with two notes + * recorded; arithmetic cannot do that. + */ +import { convertQty, normaliseUnit } from "../units"; +import { normalisePoNumber } from "../po-matching"; +import type { CheckResult, NoteSegment } from "./types"; + +/** Mass tolerance, in the slip's own unit. Scales print to the kilogram. */ +const MASS_TOLERANCE = 1; + +export interface NoteCheckContext { + /** Our own VAT number, from MEGA_CUSTOMER_VAT_NO. */ + customerVatNo: string | null; + /** Resolved PO id, or null when the number matched nothing we hold. */ + purchaseOrderId: string | null; + poItems: ReadonlyArray<{ id: string; description: string; unit?: string }>; + /** Note numbers already present in delivery_notes. */ + existingNoteNumbers: ReadonlySet; +} + +export function runNoteChecks( + note: NoteSegment, + ctx: NoteCheckContext, +): CheckResult[] { + return [ + weighbridgeArithmetic(note), + customerIdentity(note, ctx), + poResolved(note, ctx), + productMatched(note, ctx), + unitsConvertible(note, ctx), + notDuplicate(note, ctx), + ]; +} + +function weighbridgeArithmetic(note: NoteSegment): CheckResult { + const { gross_mass: g, tare_mass: t, nett_mass: n, quantity: q } = note; + if (g === null && t === null && n === null) { + return { id: "weighbridge_arithmetic", passed: null, reason: "no weighbridge section on this slip" }; + } + if (g === null || t === null || n === null) { + return { + id: "weighbridge_arithmetic", + passed: false, + reason: `incomplete weigh section (gross ${fmt(g)}, tare ${fmt(t)}, nett ${fmt(n)})`, + }; + } + if (Math.abs(g - t - n) > MASS_TOLERANCE) { + return { + id: "weighbridge_arithmetic", + passed: false, + reason: `${fmt(g)} − ${fmt(t)} = ${fmt(g - t)}, but nett reads ${fmt(n)}`, + }; + } + if (q !== null && Math.abs(n - q) > MASS_TOLERANCE) { + return { + id: "weighbridge_arithmetic", + passed: false, + reason: `nett ${fmt(n)} does not match quantity ${fmt(q)}`, + }; + } + return { id: "weighbridge_arithmetic", passed: true, reason: `${fmt(g)} − ${fmt(t)} = ${fmt(n)}` }; +} + +function customerIdentity(note: NoteSegment, ctx: NoteCheckContext): CheckResult { + if (!note.customer_vat_no || !ctx.customerVatNo) { + return { id: "customer_identity", passed: null, reason: "no customer VAT number to compare" }; + } + const a = note.customer_vat_no.replace(/\D/g, ""); + const b = ctx.customerVatNo.replace(/\D/g, ""); + return a === b + ? { id: "customer_identity", passed: true, reason: `VAT ${a}` } + : { id: "customer_identity", passed: false, reason: `slip shows VAT ${a}, ours is ${b}` }; +} + +function poResolved(note: NoteSegment, ctx: NoteCheckContext): CheckResult { + if (ctx.purchaseOrderId) { + return { id: "po_resolved", passed: true, reason: normalisePoNumber(note.po_number_raw ?? "") }; + } + const raw = note.po_number_raw?.trim(); + return { + id: "po_resolved", + passed: false, + reason: raw + ? `no purchase order matches "${raw}"` + : "no order number found on the slip", + }; +} + +function productMatched(note: NoteSegment, ctx: NoteCheckContext): CheckResult { + if (note.items.length === 0) { + return { id: "product_matched", passed: false, reason: "no line items read from the slip" }; + } + if (ctx.poItems.length === 0) { + return { id: "product_matched", passed: null, reason: "no purchase order lines to match against" }; + } + const unmatched = note.items.filter((it) => matchPoItem(it.description, ctx.poItems) === null); + return unmatched.length === 0 + ? { id: "product_matched", passed: true, reason: `${note.items.length} line(s) matched` } + : { + id: "product_matched", + passed: false, + reason: `no PO line for: ${unmatched.map((u) => u.description).join(", ")}`, + }; +} + +function unitsConvertible(note: NoteSegment, ctx: NoteCheckContext): CheckResult { + if (ctx.poItems.length === 0) { + return { id: "units_convertible", passed: null, reason: "no purchase order lines to convert into" }; + } + const bad: string[] = []; + for (const it of note.items) { + const poItem = matchPoItem(it.description, ctx.poItems); + if (!poItem) continue; // product_matched already reports this + const conv = convertQty(it.qty, normaliseUnit(it.unit), normaliseUnit(poItem.unit ?? null)); + if (!conv.ok) bad.push(`${it.description} (${it.unit ?? "no unit"} → ${poItem.unit || "no unit"})`); + } + return bad.length === 0 + ? { id: "units_convertible", passed: true, reason: "all units convert" } + : { id: "units_convertible", passed: false, reason: `cannot convert: ${bad.join(", ")}` }; +} + +function notDuplicate(note: NoteSegment, ctx: NoteCheckContext): CheckResult { + const number = note.delivery_note_number?.trim(); + if (!number) { + return { id: "not_duplicate", passed: false, reason: "no delivery note number read" }; + } + return ctx.existingNoteNumbers.has(number) + ? { id: "not_duplicate", passed: false, reason: `${number} is already in delivery notes` } + : { id: "not_duplicate", passed: true, reason: number }; +} + +/** + * Description matching, same three-tier strategy the n8n workflow used: + * exact after normalisation, then substring either way, then first-token. + * Deliberately conservative — a wrong match links a delivery to the + * wrong PO line and the wrong rate. + */ +export function matchPoItem( + description: string, + poItems: readonly T[], +): T | null { + const target = norm(description); + if (!target) return null; + for (const p of poItems) if (norm(p.description) === target) return p; + for (const p of poItems) { + const d = norm(p.description); + if (d.length >= 3 && target.includes(d)) return p; + if (target.length >= 3 && d.includes(target)) return p; + } + for (const p of poItems) { + const first = norm(p.description).split(" ")[0]; + if (first && first.length >= 3 && target.includes(first)) return p; + } + return null; +} + +function norm(s: string): string { + return (s || "").toLowerCase().replace(/[^a-z0-9 ]/g, " ").replace(/\s+/g, " ").trim(); +} + +function fmt(n: number | null): string { + return n === null ? "—" : String(n); +} + +export interface BatchCheckInput { + unassigned: readonly number[]; + needsSubsplit: readonly number[]; + notesDetected: number; + notesPromoted: number; +} + +export function runBatchChecks(input: BatchCheckInput): CheckResult[] { + const stray = [...input.unassigned, ...input.needsSubsplit].sort((a, b) => a - b); + return [ + stray.length === 0 + ? { id: "pages_accounted", passed: true, reason: "every page belongs to a note" } + : { + id: "pages_accounted", + passed: false, + reason: `page${stray.length > 1 ? "s" : ""} ${stray.join(", ")} belong${stray.length > 1 ? "" : "s"} to no note`, + }, + input.notesPromoted === input.notesDetected + ? { id: "notes_reconciled", passed: true, reason: `${input.notesPromoted} of ${input.notesDetected} promoted` } + : { + id: "notes_reconciled", + passed: false, + reason: `only ${input.notesPromoted} of ${input.notesDetected} notes promoted`, + }, + ]; +} +``` + +- [ ] **Step 4: Confirm the po-matching export name** + +`checks.ts` imports `normalisePoNumber` from `../po-matching`. Verify the spelling: + +```bash +grep -n "export function normalis\|export function normaliz" src/lib/po-matching.ts +``` + +If the export is spelled `normalizePoNumber`, change the import to match. Do **not** add a second spelling. + +- [ ] **Step 5: Run test to verify it passes** + +Run: `npx vitest run src/lib/dn-pipeline/checks.test.ts` +Expected: PASS, 16 tests. + +- [ ] **Step 6: Commit** + +```bash +git add src/lib/dn-pipeline/checks.ts src/lib/dn-pipeline/checks.test.ts +git commit -m "feat(dn-pipeline): add deterministic per-note and per-batch checks" +``` + +--- + +## Task 6: Page splitting + +**Files:** +- Create: `src/lib/dn-pipeline/split.ts` +- Test: `src/lib/dn-pipeline/split.test.ts` + +- [ ] **Step 1: Write the implementation** + +This module shells out, so it is verified with a real PDF fixture rather than unit-tested in isolation. + +Create `src/lib/dn-pipeline/split.ts`: + +```ts +/** + * Document → page images, and page range → per-note PDF. + * + * MiniMax's OpenAI-compatible endpoint takes images, not PDF documents, + * so every page is rasterised before extraction. The per-note PDF is + * assembled separately with pdf-lib so each promoted note carries an + * attachment showing exactly the pages it was read from. + */ +import { execFile } from "node:child_process"; +import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { PDFDocument } from "pdf-lib"; + +const run = promisify(execFile); + +/** Rendering resolution. 150 dpi is legible for scanned slips without bloating the payload. */ +const DPI = 150; + +export interface PageImage { + page: number; + png: Buffer; +} + +/** Page count of a PDF. Images are always one page. */ +export async function pageCount(buffer: Buffer, mime: string): Promise { + if (mime !== "application/pdf") return 1; + const dir = await mkdtemp(join(tmpdir(), "dn-count-")); + try { + const path = join(dir, "in.pdf"); + await writeFile(path, buffer); + const { stdout } = await run("pdfinfo", [path]); + const m = stdout.match(/^Pages:\s+(\d+)/m); + if (!m) throw new Error("pdfinfo did not report a page count"); + return Number(m[1]); + } finally { + await rm(dir, { recursive: true, force: true }); + } +} + +/** + * Render every page to PNG. An image upload passes through untouched as + * a single page. + */ +export async function splitPages(buffer: Buffer, mime: string): Promise { + if (mime !== "application/pdf") return [{ page: 1, png: buffer }]; + + const dir = await mkdtemp(join(tmpdir(), "dn-split-")); + try { + const path = join(dir, "in.pdf"); + await writeFile(path, buffer); + // -r DPI, -png, prefix "pg" → pg-1.png, pg-2.png, … + await run("pdftoppm", ["-r", String(DPI), "-png", path, join(dir, "pg")]); + + const files = (await readdir(dir)) + .filter((f) => f.startsWith("pg-") && f.endsWith(".png")) + .map((f) => ({ f, n: Number(f.replace(/^pg-|\.png$/g, "")) })) + .filter((e) => Number.isFinite(e.n)) + .sort((a, b) => a.n - b.n); + + if (files.length === 0) throw new Error("pdftoppm produced no pages"); + + return Promise.all( + files.map(async (e) => ({ page: e.n, png: await readFile(join(dir, e.f)) })), + ); + } finally { + await rm(dir, { recursive: true, force: true }); + } +} + +/** + * A PDF containing only `pageFrom..pageTo` (1-indexed, inclusive) of the + * source. Non-PDF sources return their own bytes — a photo of a single + * slip is already the note's own file. + */ +export async function extractPageRange( + buffer: Buffer, + mime: string, + pageFrom: number, + pageTo: number, +): Promise { + if (mime !== "application/pdf") return buffer; + + const src = await PDFDocument.load(buffer); + const out = await PDFDocument.create(); + const indices: number[] = []; + for (let p = pageFrom; p <= pageTo; p++) { + const i = p - 1; + if (i < 0 || i >= src.getPageCount()) { + throw new Error(`page ${p} out of range (document has ${src.getPageCount()})`); + } + indices.push(i); + } + const copied = await out.copyPages(src, indices); + for (const page of copied) out.addPage(page); + return Buffer.from(await out.save()); +} +``` + +- [ ] **Step 2: Write the fixture test** + +Create `src/lib/dn-pipeline/split.test.ts`: + +```ts +import { describe, it, expect, beforeAll } from "vitest"; +import { existsSync } from "node:fs"; +import { readFile } from "node:fs/promises"; +import { extractPageRange, pageCount, splitPages } from "./split"; + +const FIXTURE = "src/lib/dn-pipeline/__fixtures__/four-note-scan.pdf"; +const PDF = "application/pdf"; + +describe.skipIf(!existsSync(FIXTURE))("split (requires poppler + the fixture)", () => { + let buf: Buffer; + beforeAll(async () => { + buf = await readFile(FIXTURE); + }); + + it("counts the pages", async () => { + expect(await pageCount(buf, PDF)).toBe(4); + }); + + it("renders one PNG per page, in order", async () => { + const pages = await splitPages(buf, PDF); + expect(pages.map((p) => p.page)).toEqual([1, 2, 3, 4]); + for (const p of pages) { + // PNG magic number. + expect(p.png.subarray(0, 4)).toEqual(Buffer.from([0x89, 0x50, 0x4e, 0x47])); + expect(p.png.length).toBeGreaterThan(1000); + } + }); + + it("extracts a page range into its own PDF", async () => { + const out = await extractPageRange(buf, PDF, 2, 3); + expect(await pageCount(out, PDF)).toBe(2); + }); + + it("rejects a range outside the document", async () => { + await expect(extractPageRange(buf, PDF, 3, 9)).rejects.toThrow("out of range"); + }); + + it("passes an image through as a single page", async () => { + const fake = Buffer.from([0xff, 0xd8, 0xff]); + expect(await pageCount(fake, "image/jpeg")).toBe(1); + const pages = await splitPages(fake, "image/jpeg"); + expect(pages).toEqual([{ page: 1, png: fake }]); + }); +}); +``` + +- [ ] **Step 3: Fetch the fixture** + +```bash +mkdir -p src/lib/dn-pipeline/__fixtures__ +python3 - <<'EOF' +import sys; sys.path.insert(0, "scripts") +from pb_env import PB, http, superuser_token +import urllib.request +t = superuser_token() +url = f"{PB}/api/files/pbc_3665775234/u27fj2ddxgs138l/scan_20260915_c5p7fkmzgv.pdf" +req = urllib.request.Request(url, headers={"Authorization": t}) +with urllib.request.urlopen(req) as r, open("src/lib/dn-pipeline/__fixtures__/four-note-scan.pdf", "wb") as f: + f.write(r.read()) +print("fixture saved") +EOF +``` + +Expected: `fixture saved`, and the file is about 220 KB. + +Add it to git — it is a small, stable fixture and the tests are worth far more with it than with a synthetic PDF. + +- [ ] **Step 4: Run the test** + +Run: `npx vitest run src/lib/dn-pipeline/split.test.ts` +Expected: PASS, 5 tests. If it reports *skipped*, the fixture path is wrong — check step 3 wrote to the right place. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/dn-pipeline/split.ts src/lib/dn-pipeline/split.test.ts \ + src/lib/dn-pipeline/__fixtures__/four-note-scan.pdf +git commit -m "feat(dn-pipeline): render document pages and extract per-note page ranges" +``` + +--- + +## Task 7: Page extraction + +**Files:** +- Create: `src/lib/dn-pipeline/extract.ts` +- Test: `src/lib/dn-pipeline/extract.test.ts` + +- [ ] **Step 1: Write the implementation** + +Create `src/lib/dn-pipeline/extract.ts`: + +```ts +/** + * One page image → one PageRead. + * + * The prompt's single most important instruction is that the model must + * report ONLY what is printed on the page in front of it. Grouping + * pages into notes is segment.ts's job, and it can only do that + * honestly if each page's read is independent — a model that "helpfully" + * carries a note number across a page break destroys the page + * accounting that stops notes going missing. + */ +import OpenAI from "openai"; +import { z } from "zod"; +import type { PageRead } from "./types"; + +const nullableString = z.string().nullish().transform((v) => v ?? null); +const nullableNumber = z.coerce.number().nullish().transform((v) => (v === null || v === undefined || Number.isNaN(v) ? null : v)); + +const PageReadSchema = z.object({ + delivery_note_numbers: z.array(z.string()).default([]), + has_header: z.boolean().default(true), + is_continuation: z.boolean().default(false), + po_number_raw: nullableString, + customer_vat_no: nullableString, + date: nullableString, + received_by_name: nullableString, + gross_mass: nullableNumber, + tare_mass: nullableNumber, + nett_mass: nullableNumber, + quantity: nullableNumber, + unit: nullableString, + items: z + .array( + z.object({ + description: z.string().default(""), + qty: z.coerce.number().default(0), + unit: nullableString, + unit_price: nullableNumber, + }), + ) + .default([]), +}); + +const PROMPT = `You are reading ONE page of a scanned delivery note document. + +Report ONLY what is printed on this page. Never infer a delivery note +number, purchase order number or quantity from another page you have +seen - you are shown one page at a time and the pages are assembled +elsewhere. + +Return ONLY valid JSON, no markdown fences and no commentary: + +{ + "delivery_note_numbers": string[], + "has_header": boolean, + "is_continuation": boolean, + "po_number_raw": string|null, + "customer_vat_no": string|null, + "date": string|null, + "received_by_name": string|null, + "gross_mass": number|null, + "tare_mass": number|null, + "nett_mass": number|null, + "quantity": number|null, + "unit": string|null, + "items": [{ "description": string, "qty": number, "unit": string|null, "unit_price": number|null }] +} + +Rules: +- delivery_note_numbers: every distinct delivery note number printed on + THIS page. Usually one. Empty array if none is printed. +- has_header: true if this page shows a supplier letterhead or a + document title such as "Delivery Note". +- is_continuation: true if this page is line items carrying on from a + previous page, with no header of its own. +- po_number_raw: the customer's order number VERBATIM, including any + leading "#". Do not clean it up. +- gross_mass / tare_mass / nett_mass: from the weigh section, numbers + only, no unit suffix. Null if the page has no weigh section. +- quantity / unit: the delivered quantity and its unit as printed. +- date: ISO YYYY-MM-DD, or null. +- Numbers must be numbers, not strings. Strip thousands separators.`; + +let client: OpenAI | null = null; +function getClient(): OpenAI { + if (!client) { + const apiKey = process.env.MINIMAX_API_KEY; + if (!apiKey) throw new Error("MINIMAX_API_KEY is not set"); + client = new OpenAI({ + apiKey, + baseURL: process.env.MINIMAX_BASE_URL || "https://api.minimax.io/v1", + }); + } + return client; +} + +export async function extractPage(png: Buffer, page: number): Promise { + const res = await getClient().chat.completions.create({ + model: process.env.MINIMAX_MODEL || "MiniMax-M3", + messages: [ + { + role: "user", + content: [ + { type: "image_url", image_url: { url: `data:image/png;base64,${png.toString("base64")}` } }, + { type: "text", text: PROMPT }, + ], + }, + ], + }); + + const raw = res.choices[0]?.message?.content; + if (!raw) throw new Error(`page ${page}: model returned no content`); + return { page, ...parsePageRead(raw) }; +} + +/** + * Exported for testing. A parse failure throws — it must surface as a + * page-level error the batch can report, never as a silently empty read + * that shrinks the note count. + */ +export function parsePageRead(raw: string): Omit { + const text = raw.trim().replace(/^```(?:json)?\s*/i, "").replace(/\s*```$/, ""); + let json: unknown; + try { + json = JSON.parse(text); + } catch { + throw new Error(`model did not return JSON: ${text.slice(0, 200)}`); + } + const parsed = PageReadSchema.safeParse(json); + if (!parsed.success) { + throw new Error(`model JSON failed validation: ${parsed.error.issues.map((i) => i.path.join(".") + " " + i.message).join("; ")}`); + } + return parsed.data; +} +``` + +- [ ] **Step 2: Write the parser test** + +Create `src/lib/dn-pipeline/extract.test.ts`: + +```ts +import { describe, it, expect } from "vitest"; +import { parsePageRead } from "./extract"; + +describe("parsePageRead", () => { + it("parses a clean weighbridge page", () => { + const r = parsePageRead(JSON.stringify({ + delivery_note_numbers: ["DQ0010107"], + has_header: true, + is_continuation: false, + po_number_raw: "#MEGCPO9165", + customer_vat_no: "4280307796", + date: "2026-09-08", + received_by_name: "Peter Khabo", + gross_mass: 48760, tare_mass: 18800, nett_mass: 29960, + quantity: 29960, unit: "kg", + items: [{ description: "G6 Sub Base", qty: 29960, unit: "kg", unit_price: null }], + })); + expect(r.delivery_note_numbers).toEqual(["DQ0010107"]); + expect(r.gross_mass).toBe(48760); + expect(r.items).toHaveLength(1); + }); + + it("strips markdown fences the model sometimes adds", () => { + const r = parsePageRead('```json\n{"delivery_note_numbers":["DQ1"]}\n```'); + expect(r.delivery_note_numbers).toEqual(["DQ1"]); + }); + + it("defaults absent fields rather than dropping the page", () => { + const r = parsePageRead("{}"); + expect(r.delivery_note_numbers).toEqual([]); + expect(r.items).toEqual([]); + expect(r.gross_mass).toBeNull(); + expect(r.has_header).toBe(true); + }); + + it("coerces numbers the model returned as strings", () => { + const r = parsePageRead('{"gross_mass":"48760","quantity":"29960"}'); + expect(r.gross_mass).toBe(48760); + expect(r.quantity).toBe(29960); + }); + + it("throws on non-JSON rather than returning an empty read", () => { + expect(() => parsePageRead("I could not read this page")).toThrow("did not return JSON"); + }); + + it("throws when a field has the wrong shape", () => { + expect(() => parsePageRead('{"delivery_note_numbers":"DQ1"}')).toThrow("failed validation"); + }); +}); +``` + +- [ ] **Step 3: Run test to verify it passes** + +Run: `npx vitest run src/lib/dn-pipeline/extract.test.ts` +Expected: PASS, 6 tests. + +- [ ] **Step 4: Smoke-test against the real model** + +```bash +npx tsx --env-file=.env.local -e ' +import { readFile } from "node:fs/promises"; +import { splitPages } from "./src/lib/dn-pipeline/split"; +import { extractPage } from "./src/lib/dn-pipeline/extract"; +const buf = await readFile("src/lib/dn-pipeline/__fixtures__/four-note-scan.pdf"); +const pages = await splitPages(buf, "application/pdf"); +for (const p of pages) { + const r = await extractPage(p.png, p.page); + console.log(p.page, r.delivery_note_numbers, r.quantity, r.unit, r.gross_mass, r.tare_mass, r.nett_mass); +} +' +``` + +Expected, matching the table in the Background section: + +``` +1 [ 'DQ0010107' ] 29960 kg 48760 18800 29960 +2 [ 'DQ0010099' ] 34820 kg ... +3 [ 'DQ0010239' ] 34200 kg 56440 22240 34200 +4 [ 'DQ0010217' ] 34520 kg 56640 22120 34520 +``` + +**This is the gate on the whole approach.** If the model cannot read these four pages reliably, stop and tune the prompt before building anything on top. Iterate here until all four are right. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/dn-pipeline/extract.ts src/lib/dn-pipeline/extract.test.ts +git commit -m "feat(dn-pipeline): extract one page at a time with strict JSON validation" +``` + +--- + +## Task 8: Promotion + +Depends on phase 1's `src/lib/dn-item-row.ts`. + +**Files:** +- Create: `src/lib/dn-pipeline/promote.ts` + +- [ ] **Step 1: Write the implementation** + +Create `src/lib/dn-pipeline/promote.ts`: + +```ts +/** + * A validated candidate note → a real delivery_notes row. + * + * Nothing else in the codebase writes to delivery_notes from the + * pipeline. Everything that reaches here has already passed every check + * in checks.ts; this module's only job is to write it down faithfully + * and idempotently. + */ +import type PocketBase from "pocketbase"; +import { buildDeliveryNoteItemRows, type PoItemLite } from "../dn-item-row"; +import { matchPoItem } from "./checks"; +import type { NoteSegment } from "./types"; + +/** A PO line with enough on it to both match and convert against. */ +export type PromotePoItem = PoItemLite & { description: string }; + +export interface PromoteInput { + pb: PocketBase; + note: NoteSegment; + /** dn_batch_notes record id. */ + batchNoteId: string; + batchId: string; + siteId: string; + purchaseOrderId: string | null; + poItems: readonly PromotePoItem[]; + /** Per-note PDF assembled from this note's own pages. */ + pageFile: Buffer; + pageFileName: string; +} + +export interface PromoteResult { + deliveryNoteId: string; + itemsCreated: number; +} + +export async function promoteNote(input: PromoteInput): Promise { + const { pb, note, batchNoteId, batchId, siteId, purchaseOrderId, poItems } = input; + + // Idempotency: a re-run must update the row it already made, never + // mint a second one. + const existing = await pb + .collection("dn_batch_notes") + .getOne(batchNoteId, { fields: "id,delivery_note" }) + .catch(() => null); + const existingDnId: string | null = existing?.delivery_note || null; + + const payload = { + delivery_note_number: note.delivery_note_number, + site: siteId, + purchase_order: purchaseOrderId, + date_received: note.date ? new Date(note.date).toISOString() : new Date().toISOString(), + received_by: "dn-pipeline", + received_by_name: note.received_by_name ?? "", + batch: batchId, + page_from: note.page_from, + page_to: note.page_to, + is_verified: true, + processing_status: "auto-verified", + ocr_completed_at: new Date().toISOString(), + raw_ocr_data: { + source: "dn-pipeline", + po_number_raw: note.po_number_raw, + gross_mass: note.gross_mass, + tare_mass: note.tare_mass, + nett_mass: note.nett_mass, + quantity: note.quantity, + unit: note.unit, + pages: note.pages, + items: note.items, + }, + }; + + const form = new FormData(); + for (const [k, v] of Object.entries(payload)) { + form.append(k, typeof v === "object" && v !== null ? JSON.stringify(v) : String(v ?? "")); + } + form.append( + "slip_image", + new Blob([new Uint8Array(input.pageFile)], { type: "application/pdf" }), + input.pageFileName, + ); + + const dn = existingDnId + ? await pb.collection("delivery_notes").update(existingDnId, form) + : await pb.collection("delivery_notes").create(form); + + // Replace the item set wholesale so a re-run cannot leave stale lines. + const stale = await pb + .collection("delivery_note_items") + .getFullList({ filter: `delivery_note='${dn.id}'`, fields: "id" }); + for (const s of stale) { + await pb.collection("delivery_note_items").delete(s.id).catch(() => undefined); + } + + const { rows } = buildDeliveryNoteItemRows({ + items: note.items.map((it) => ({ + description: it.description, + qty: it.qty, + unit: it.unit, + unit_price: it.unit_price, + matched_po_item_id: matchId(it.description, poItems), + })), + poItems, + confidence: 1, + }); + + let created = 0; + for (const row of rows) { + await pb.collection("delivery_note_items").create({ + delivery_note: dn.id, + purchase_order_item: row.purchase_order_item, + qty_received: row.qty_received, + qty_raw: row.qty_raw, + unit_raw: row.unit_raw, + unit_converted: row.unit_converted, + condition_notes: row.condition_notes, + }); + created++; + } + + await pb.collection("dn_batch_notes").update(batchNoteId, { + status: "promoted", + delivery_note: dn.id, + }); + + return { deliveryNoteId: dn.id, itemsCreated: created }; +} + +/** + * checks.ts already proved every line matches a PO line — reuse the + * same matcher here so the link written down is the one the check + * validated. Two different matchers would let a note pass its check and + * then be linked to something else. + */ +function matchId(description: string, poItems: readonly PromotePoItem[]): string | null { + const hit = matchPoItem( + description, + poItems.map((p) => ({ id: p.id, description: p.description })), + ); + return hit?.id ?? null; +} +``` + +- [ ] **Step 2: Typecheck** + +Run: `npm run typecheck` +Expected: no errors. If `PoItemLite` is not exported from `src/lib/dn-item-row.ts`, export it there — phase 1 declares it as an exported interface. + +- [ ] **Step 3: Commit** + +```bash +git add src/lib/dn-pipeline/promote.ts +git commit -m "feat(dn-pipeline): promote a validated note into delivery_notes" +``` + +--- + +## Task 9: The orchestrator + +**Files:** +- Create: `src/lib/dn-pipeline/run.ts` + +- [ ] **Step 1: Write the implementation** + +Create `src/lib/dn-pipeline/run.ts`: + +```ts +/** + * Runs one batch from `analysing` to its terminal state. + * + * The invariant this file exists to hold: a batch reaches `processed` + * only when every page belongs to a note AND every detected note was + * promoted. Anything else is `needs_attention` with the failing checks + * named. There is no path that reports success while a note is missing. + */ +import type PocketBase from "pocketbase"; +import { findBestPoMatch, normalisePoNumber } from "../po-matching"; +import { runBatchChecks, runNoteChecks } from "./checks"; +import { extractPage } from "./extract"; +import { promoteNote } from "./promote"; +import { extractPageRange, splitPages } from "./split"; +import { segmentPages } from "./segment"; +import type { CheckResult, PageRead } from "./types"; + +export interface RunBatchResult { + status: "processed" | "needs_attention"; + notesDetected: number; + notesPromoted: number; + checks: CheckResult[]; +} + +export async function runBatch(pb: PocketBase, batchId: string): Promise { + const batch = await pb.collection("dn_batches").getOne(batchId); + const { buffer, mime, filename } = await fetchBatchFile(pb, batch); + + // 1. Read every page independently. + const images = await splitPages(buffer, mime); + const reads: PageRead[] = []; + const pageErrors: CheckResult[] = []; + for (const img of images) { + try { + reads.push(await extractPage(img.png, img.page)); + } catch (err) { + pageErrors.push({ + id: `page_${img.page}_read`, + passed: false, + reason: err instanceof Error ? err.message : String(err), + }); + } + } + + // 2. Group pages into notes. + const seg = segmentPages(reads); + + await pb.collection("dn_batches").update(batchId, { + page_count: images.length, + notes_detected: seg.notes.length, + segmentation: seg, + }); + + // 3. Resolve each note's PO, run its checks, create its candidate row. + const existingNumbers = await loadExistingNoteNumbers(pb); + const customerVatNo = process.env.MEGA_CUSTOMER_VAT_NO || null; + const allChecks: CheckResult[] = [...pageErrors]; + let promoted = 0; + + for (const note of seg.notes) { + const po = await resolvePo(pb, note.po_number_raw); + const poItems = po + ? ((await pb.collection("purchase_order_items").getFullList({ + filter: `purchase_order='${po.id}'`, + fields: "id,description,unit", + })) as unknown as { id: string; description: string; unit?: string }[]) + : []; + + const checks = runNoteChecks(note, { + customerVatNo, + purchaseOrderId: po?.id ?? null, + poItems, + existingNoteNumbers: existingNumbers, + }); + const blocked = checks.some((c) => c.passed === false); + + const pageFile = await extractPageRange(buffer, mime, note.page_from, note.page_to); + const pageFileName = `${filename.replace(/\.[^.]+$/, "")}-note-${note.note_index + 1}.pdf`; + + const candidate = await upsertCandidate(pb, batchId, note, { + poId: po?.id ?? null, + poNumberNormalised: normalisePoNumber(note.po_number_raw ?? ""), + checks, + status: blocked ? "blocked" : "candidate", + pageFile, + pageFileName, + }); + + allChecks.push(...checks.map((c) => ({ ...c, id: `note${note.note_index + 1}.${c.id}` }))); + + if (!blocked) { + await promoteNote({ + pb, + note, + batchNoteId: candidate.id, + batchId, + siteId: batch.site, + purchaseOrderId: po?.id ?? null, + poItems: poItems.map((p) => ({ id: p.id, unit: p.unit, description: p.description })), + pageFile, + pageFileName, + }); + promoted++; + // A promoted number must block a duplicate later in the same batch. + existingNumbers.add(note.delivery_note_number); + } + } + + // 4. Reconcile. This is the check whose absence lost two notes. + const batchChecks = runBatchChecks({ + unassigned: seg.unassigned, + needsSubsplit: seg.needsSubsplit, + notesDetected: seg.notes.length, + notesPromoted: promoted, + }); + allChecks.push(...batchChecks); + + const ok = allChecks.every((c) => c.passed !== false); + const status = ok ? "processed" : "needs_attention"; + + await pb.collection("dn_batches").update(batchId, { + status, + notes_promoted: promoted, + checks: allChecks, + claimed_at: "", + last_error: "", + }); + + return { status, notesDetected: seg.notes.length, notesPromoted: promoted, checks: allChecks }; +} + +async function fetchBatchFile( + pb: PocketBase, + batch: { id: string; file: string }, +): Promise<{ buffer: Buffer; mime: string; filename: string }> { + const base = process.env.NEXT_PUBLIC_POCKETBASE_URL; + if (!base) throw new Error("NEXT_PUBLIC_POCKETBASE_URL is not set"); + const res = await fetch(`${base}/api/files/dn_batches/${batch.id}/${batch.file}`, { + headers: { Authorization: pb.authStore.token }, + }); + if (!res.ok) throw new Error(`batch file fetch failed: ${res.status}`); + return { + buffer: Buffer.from(await res.arrayBuffer()), + mime: res.headers.get("content-type") ?? "application/pdf", + filename: batch.file, + }; +} + +async function loadExistingNoteNumbers(pb: PocketBase): Promise> { + const rows = await pb + .collection("delivery_notes") + .getFullList({ fields: "delivery_note_number" }); + return new Set( + rows + .map((r) => (r as { delivery_note_number?: string }).delivery_note_number?.trim()) + .filter((v): v is string => Boolean(v)), + ); +} + +async function resolvePo( + pb: PocketBase, + raw: string | null, +): Promise<{ id: string; purchase_order_number: string } | null> { + if (!raw) return null; + // Suppliers print "#MEGCPO9165". normalisePoNumber strips the + // decoration so the lookup sees what we actually store. + const target = normalisePoNumber(raw); + if (!target) return null; + const candidates = (await pb.collection("purchase_orders").getFullList({ + fields: "id,purchase_order_number", + })) as unknown as { id: string; purchase_order_number: string }[]; + + const exact = candidates.find( + (c) => normalisePoNumber(c.purchase_order_number ?? "") === target, + ); + if (exact) return exact; + return findBestPoMatch(target, candidates, { threshold: 0.85 }) ?? null; +} + +async function upsertCandidate( + pb: PocketBase, + batchId: string, + note: import("./types").NoteSegment, + extra: { + poId: string | null; + poNumberNormalised: string; + checks: CheckResult[]; + status: "candidate" | "blocked"; + pageFile: Buffer; + pageFileName: string; + }, +) { + const existing = await pb + .collection("dn_batch_notes") + .getFullList({ filter: `batch='${batchId}' && note_index=${note.note_index}`, fields: "id" }); + + const form = new FormData(); + const fields: Record = { + batch: batchId, + note_index: note.note_index, + page_from: note.page_from, + page_to: note.page_to, + delivery_note_number: note.delivery_note_number, + po_number_raw: note.po_number_raw ?? "", + po_number_normalised: extra.poNumberNormalised, + purchase_order: extra.poId, + gross_mass: note.gross_mass, + tare_mass: note.tare_mass, + nett_mass: note.nett_mass, + quantity: note.quantity, + unit: note.unit ?? "", + items: note.items, + checks: extra.checks, + status: extra.status, + }; + for (const [k, v] of Object.entries(fields)) { + form.append(k, typeof v === "object" && v !== null ? JSON.stringify(v) : String(v ?? "")); + } + form.append( + "page_file", + new Blob([new Uint8Array(extra.pageFile)], { type: "application/pdf" }), + extra.pageFileName, + ); + + return existing.length > 0 + ? pb.collection("dn_batch_notes").update(existing[0].id, form) + : pb.collection("dn_batch_notes").create(form); +} +``` + +- [ ] **Step 2: Confirm findBestPoMatch's return shape** + +```bash +grep -n "export function findBestPoMatch" -A 8 src/lib/po-matching.ts +``` + +If it returns a wrapper such as `{ id, purchase_order_number, score }` rather than the candidate itself, adjust the `resolvePo` return accordingly. Do not guess. + +- [ ] **Step 3: Typecheck and commit** + +```bash +npm run typecheck +git add src/lib/dn-pipeline/run.ts +git commit -m "feat(dn-pipeline): orchestrate a batch through extraction, checks and promotion" +``` + +--- + +## Task 10: The tick route + +**Files:** +- Create: `src/app/api/dn-batches/tick/route.ts` + +- [ ] **Step 1: Write the route** + +Create `src/app/api/dn-batches/tick/route.ts`: + +```ts +import { NextResponse } from "next/server"; +import { pbAdmin } from "@/lib/pb"; +import { runBatch } from "@/lib/dn-pipeline/run"; + +export const dynamic = "force-dynamic"; +export const runtime = "nodejs"; +export const maxDuration = 300; + +/** A batch claimed longer ago than this is assumed dead and reclaimed. */ +const STALE_CLAIM_MS = 10 * 60 * 1000; +const MAX_ATTEMPTS = 3; + +/** + * POST /api/dn-batches/tick + * + * The only thing n8n calls. Claims up to DN_TICK_BATCH_SIZE inbox + * batches and runs each through the pipeline. + * + * Claiming is compare-and-set on `status`: we PATCH inbox → analysing + * and re-read. PocketBase has no transactions, so a losing racer sees a + * status that is no longer `inbox` and skips the batch rather than + * double-processing it. + */ +export async function POST(request: Request) { + const secret = request.headers.get("x-ingest-secret"); + if (!secret || secret !== process.env.N8N_INGEST_SECRET) { + return NextResponse.json({ error: "unauthorized" }, { status: 401 }); + } + + const pb = await pbAdmin(); + const limit = Number(process.env.DN_TICK_BATCH_SIZE || 3); + const results: Record[] = []; + + await reclaimStale(pb); + + let candidates: { id: string }[] = []; + try { + candidates = await pb.collection("dn_batches").getFullList({ + filter: `status='inbox'`, + fields: "id", + sort: "created", + }); + } catch (err) { + console.error("[dn-tick] list failed", err); + return NextResponse.json({ error: "list_failed" }, { status: 500 }); + } + + for (const c of candidates.slice(0, limit)) { + const claimed = await claim(pb, c.id); + if (!claimed) { + results.push({ id: c.id, skipped: "not_claimed" }); + continue; + } + try { + const r = await runBatch(pb, c.id); + results.push({ id: c.id, ...r, checks: undefined }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error("[dn-tick] run failed", { id: c.id, message }); + const current = await pb + .collection("dn_batches") + .getOne(c.id, { fields: "attempts" }) + .catch(() => ({ attempts: 0 })); + const attempts = ((current as { attempts?: number }).attempts ?? 0) + 1; + await pb.collection("dn_batches").update(c.id, { + attempts, + last_error: message.slice(0, 2000), + claimed_at: "", + // Give up after MAX_ATTEMPTS rather than looping forever and + // spending model calls on a document that will never parse. + status: attempts >= MAX_ATTEMPTS ? "needs_attention" : "inbox", + }); + results.push({ id: c.id, error: message, attempts }); + } + } + + return NextResponse.json({ ok: true, claimed: results.length, results }); +} + +async function claim(pb: Awaited>, id: string): Promise { + try { + const before = await pb.collection("dn_batches").getOne(id, { fields: "status" }); + if ((before as { status?: string }).status !== "inbox") return false; + await pb.collection("dn_batches").update(id, { + status: "analysing", + claimed_at: new Date().toISOString(), + }); + const after = await pb.collection("dn_batches").getOne(id, { fields: "status,claimed_at" }); + return (after as { status?: string }).status === "analysing"; + } catch { + return false; + } +} + +async function reclaimStale(pb: Awaited>): Promise { + const cutoff = new Date(Date.now() - STALE_CLAIM_MS).toISOString(); + try { + const stuck = await pb.collection("dn_batches").getFullList({ + filter: `status='analysing' && claimed_at < "${cutoff}"`, + fields: "id,attempts", + }); + for (const s of stuck) { + const attempts = ((s as { attempts?: number }).attempts ?? 0) + 1; + await pb.collection("dn_batches").update(s.id, { + attempts, + status: attempts >= MAX_ATTEMPTS ? "needs_attention" : "inbox", + claimed_at: "", + last_error: "reclaimed after a stalled run", + }); + } + } catch (err) { + console.warn("[dn-tick] reclaim failed", err); + } +} +``` + +- [ ] **Step 2: Add the environment variables** + +Add to `.env.local` and `.env.local.example`, and to the Coolify environment: + +``` +MEGA_CUSTOMER_VAT_NO=4280307796 +DN_TICK_BATCH_SIZE=3 +``` + +- [ ] **Step 3: Typecheck and commit** + +```bash +npm run typecheck +git add src/app/api/dn-batches/tick/route.ts .env.local.example +git commit -m "feat(dn-batches): add the tick route with compare-and-set claiming" +``` + +--- + +## Task 11: Upload creates a batch + +**Files:** +- Modify: `src/app/api/delivery-notes/upload/route.ts` +- Modify: `src/app/api/delivery-notes/upload/route.test.ts` + +- [ ] **Step 1: Point the dedup query at batches** + +In `src/app/api/delivery-notes/upload/route.ts`, the dedup block currently queries `delivery_notes`: + +```ts + existing = await pb.collection("delivery_notes").getFullList({ + filter: + `file_hash = "${file_hash}" && ` + + `processing_status != "failed" && ` + + `date_received >= "${cutoff}"`, + sort: "-date_received", + fields: "id,delivery_note_number", + }); +``` + +Replace it with: + +```ts + existing = await pb.collection("dn_batches").getFullList({ + filter: `file_hash = "${file_hash}" && status != "needs_attention"`, + fields: "id,status", + }); +``` + +and change the duplicate response body to report the batch: + +```ts + return NextResponse.json( + { + duplicate: true, + status: "duplicate", + existing_batch_id: prior.id, + filename: file.name, + }, + { status: 200 }, + ); +``` + +Update the `existing` declaration's type to `Array<{ id: string; status?: string }>`. + +- [ ] **Step 2: Create a batch instead of a delivery note** + +Replace the `pb.collection("delivery_notes").create({...})` block with: + +```ts + try { + const batch = await pb.collection("dn_batches").create({ + file, + file_hash, + site: siteId, + source: "web", + uploaded_by: "web-upload", + status: "inbox", + page_count: 0, + notes_detected: 0, + notes_promoted: 0, + attempts: 0, + }); + + return NextResponse.json( + { + batch_id: batch.id, + siteId, + projectCode: resolvedProject, + status: "inbox", + created: batch.created, + }, + { status: 201 }, + ); + } catch (err) { + return NextResponse.json( + { error: err instanceof Error ? err.message : "Failed to create batch" }, + { status: 500 }, + ); + } +``` + +The `delivery_note_number` form field is no longer used — the pipeline reads it off the slip. Remove its parsing. + +- [ ] **Step 3: Update the tests** + +Three edits to `src/app/api/delivery-notes/upload/route.test.ts`. The six validation tests (invalid form data, missing file, empty file, oversize, unsupported type, missing site) are unchanged — that behaviour is identical. + +**a.** In the `vi.mock("@/lib/pb", …)` block near the top, the collection switch names `delivery_notes`. Change that branch to `dn_batches`: + +```ts + if (name === "dn_batches") { + return { create: createMock, getFullList: getFullListMock }; + } +``` + +Leave the `sites` branch and the `throw new Error(...)` default exactly as they are — that guard is what proves no other collection is touched. + +**b.** Replace the body of the test named `"creates a DN with file_hash set, returns 201 with the new id"`. Rename it and rewrite its assertions: + +```ts + it("creates an inbox batch with file_hash set, returns 201", async () => { + createMock.mockResolvedValueOnce({ + id: "btchnew12345678", + created: "2026-09-07T10:00:00Z", + }); + + const res = await POST(makeRequest(makeValidForm())); + + expect(res.status).toBe(201); + expect(await res.json()).toEqual({ + batch_id: "btchnew12345678", + siteId: "siteabc", + projectCode: "P-001", + status: "inbox", + created: "2026-09-07T10:00:00Z", + }); + + expect(createMock).toHaveBeenCalledTimes(1); + + const payload = createMock.mock.calls[0][0]; + expect(payload.site).toBe("siteabc"); + expect(payload.status).toBe("inbox"); + expect(payload.source).toBe("web"); + expect(payload.file).toBeInstanceOf(File); + expect(payload.file_hash).toMatch(/^[a-f0-9]{64}$/); + // The note number now comes off the slip, not the upload form. + expect(payload.delivery_note_number).toBeUndefined(); + }); +``` + +**c.** In `"dedups against delivery_notes, never against import batches"`, rename it to `"dedups against dn_batches"` and change the mocked row to a batch shape: + +```ts + getFullListMock.mockResolvedValue([{ id: "btch000000000001", status: "processed" }]); +``` + +then assert the duplicate body uses `existing_batch_id`: + +```ts + expect(body.existing_batch_id).toBe("btch000000000001"); +``` + +The `"computes the same hash for byte-identical files"` test needs no change beyond its mocked create id — `file_hash` is still computed the same way. + +- [ ] **Step 4: Run the tests** + +Run: `npx vitest run src/app/api/delivery-notes/upload/route.test.ts` +Expected: PASS. + +- [ ] **Step 5: Update the upload button** + +`src/components/upload-delivery-note-button.tsx` reads `id` from the 201 response. Change it to `batch_id`, and change any success copy from "delivery note uploaded" to "added to the inbox". Grep for `existing_dn_id` and update it to `existing_batch_id`. + +- [ ] **Step 6: Commit** + +```bash +npm run typecheck && npm run test:run +git add src/app/api/delivery-notes/upload/route.ts \ + src/app/api/delivery-notes/upload/route.test.ts \ + src/components/upload-delivery-note-button.tsx +git commit -m "feat(delivery-notes): upload creates an inbox batch, not a delivery note" +``` + +--- + +## Task 12: The page image route + +Spec §4.8 lists `GET /api/dn-batches` and `GET /api/dn-batches/[id]`. **Do not build those.** The board and the detail view are server components that read PocketBase directly through `pbAdmin()`, which is how every other dashboard page in this codebase works — an HTTP hop to your own server to fetch data the server already has is pure latency. The only read that genuinely needs a route is the page image, because a browser `` tag cannot carry the admin token. + +**Files:** +- Create: `src/app/api/dn-batches/[id]/pages/[n]/route.ts` + +- [ ] **Step 1: Write the page proxy** + +The board needs page images without exposing the PocketBase admin token. Mirror the existing pattern in `src/app/api/delivery-notes/[id]/slip/route.ts`. + +Create `src/app/api/dn-batches/[id]/pages/[n]/route.ts`: + +```ts +import { NextResponse } from "next/server"; +import { pbAdmin } from "@/lib/pb"; +import { splitPages } from "@/lib/dn-pipeline/split"; + +export const dynamic = "force-dynamic"; +export const runtime = "nodejs"; + +/** + * GET /api/dn-batches/[id]/pages/[n] + * + * One rendered page of a batch document as a PNG. Rendering on demand + * rather than storing every page keeps PocketBase's file store to one + * copy of the original plus one per-note PDF. + * + * Session-authenticated like the rest of the dashboard — the admin + * token never leaves the server. + */ +export async function GET( + _request: Request, + { params }: { params: Promise<{ id: string; n: string }> }, +) { + const { id, n } = await params; + if (!/^[a-z0-9]{15}$/i.test(id)) { + return NextResponse.json({ error: "invalid_batch_id" }, { status: 400 }); + } + const pageNo = Number(n); + if (!Number.isInteger(pageNo) || pageNo < 1) { + return NextResponse.json({ error: "invalid_page" }, { status: 400 }); + } + + const pb = await pbAdmin(); + let batch: { id: string; file: string }; + try { + batch = await pb.collection("dn_batches").getOne(id, { fields: "id,file" }); + } catch { + return NextResponse.json({ error: "batch_not_found" }, { status: 404 }); + } + if (!batch.file) { + return NextResponse.json({ error: "no_file" }, { status: 404 }); + } + + const base = process.env.NEXT_PUBLIC_POCKETBASE_URL; + if (!base) { + return NextResponse.json({ error: "pb_url_not_configured" }, { status: 500 }); + } + + const fileRes = await fetch(`${base}/api/files/dn_batches/${id}/${batch.file}`, { + headers: { Authorization: pb.authStore.token }, + }); + if (!fileRes.ok) { + return NextResponse.json({ error: "file_fetch_failed" }, { status: 502 }); + } + + const mime = fileRes.headers.get("content-type") ?? "application/pdf"; + const buffer = Buffer.from(await fileRes.arrayBuffer()); + + let pages; + try { + pages = await splitPages(buffer, mime); + } catch (err) { + console.error("[dn-page] render failed", err); + return NextResponse.json({ error: "render_failed" }, { status: 500 }); + } + + const hit = pages.find((p) => p.page === pageNo); + if (!hit) { + return NextResponse.json({ error: "page_out_of_range", pages: pages.length }, { status: 404 }); + } + + return new NextResponse(new Uint8Array(hit.png), { + status: 200, + headers: { + "Content-Type": "image/png", + // Pages of an uploaded document never change. + "Cache-Control": "private, max-age=86400, immutable", + }, + }); +} +``` + +- [ ] **Step 2: Typecheck and commit** + +```bash +npm run typecheck +git add src/app/api/dn-batches/\[id\]/pages/\[n\]/route.ts +git commit -m "feat(dn-batches): serve rendered document pages" +``` + +--- + +## Task 13: The board + +**Files:** +- Create: `src/app/dashboard/financial/delivery-notes/inbox/page.tsx` +- Create: `src/app/dashboard/financial/delivery-notes/inbox/inbox-board.tsx` + +- [ ] **Step 1: Write the server page** + +Create `src/app/dashboard/financial/delivery-notes/inbox/page.tsx`: + +```tsx +import { pbAdmin } from "@/lib/pb"; +import { Topbar } from "@/components/layout/sidebar"; +import { filterOpt, siteFilter, type SearchParams } from "@/lib/site-scope"; +import { getSiteScope } from "@/lib/site-scope-server"; +import { InboxBoard, type BatchCard } from "./inbox-board"; +import type { CheckResult } from "@/lib/dn-pipeline/types"; + +export const dynamic = "force-dynamic"; + +export default async function InboxPage({ + searchParams, +}: { + searchParams: Promise; +}) { + const scope = await getSiteScope(await searchParams); + const pb = await pbAdmin(); + + const batches = await pb.collection("dn_batches").getFullList({ + expand: "site", + sort: "-created", + ...filterOpt(siteFilter(scope)), + }); + + const noteRows = await pb.collection("dn_batch_notes").getFullList({ + fields: "batch,quantity,unit,status", + }); + + const totals = new Map(); + for (const n of noteRows) { + const row = n as unknown as { batch: string; quantity?: number; unit?: string }; + const cur = totals.get(row.batch) ?? { qty: 0, unit: row.unit ?? "" }; + cur.qty += row.quantity ?? 0; + if (!cur.unit && row.unit) cur.unit = row.unit; + totals.set(row.batch, cur); + } + + const cards: BatchCard[] = batches.map((b) => { + const site = b.expand?.site as { name?: string } | undefined; + const total = totals.get(b.id); + const checks = Array.isArray(b.checks) ? (b.checks as CheckResult[]) : []; + return { + id: b.id, + filename: b.file ?? "(no file)", + siteName: site?.name ?? "(unknown site)", + status: b.status ?? "inbox", + pageCount: b.page_count ?? 0, + notesDetected: b.notes_detected ?? 0, + notesPromoted: b.notes_promoted ?? 0, + totalQty: total?.qty ?? 0, + totalUnit: total?.unit ?? "", + failedChecks: checks + .filter((c) => c.passed === false) + .map((c) => c.reason) + .slice(0, 3), + lastError: b.last_error ?? "", + }; + }); + + return ( + <> + +
+ +
+ + ); +} +``` + +- [ ] **Step 2: Write the board component** + +Create `src/app/dashboard/financial/delivery-notes/inbox/inbox-board.tsx`: + +```tsx +"use client"; + +import Link from "next/link"; +import { Badge } from "@/components/ui/badge"; +import { cn, formatNumber } from "@/lib/utils"; + +export interface BatchCard { + id: string; + filename: string; + siteName: string; + status: string; + pageCount: number; + notesDetected: number; + notesPromoted: number; + totalQty: number; + totalUnit: string; + failedChecks: string[]; + lastError: string; +} + +const COLUMNS: { key: string; label: string }[] = [ + { key: "inbox", label: "Inbox" }, + { key: "analysing", label: "Analysing" }, + { key: "needs_attention", label: "Needs attention" }, + { key: "processed", label: "Processed" }, +]; + +/** Processed batches are historical — cap the column and link to the full list. */ +const PROCESSED_LIMIT = 20; + +export function InboxBoard({ cards }: { cards: BatchCard[] }) { + return ( +
+ {COLUMNS.map((col) => { + const all = cards.filter((c) => c.status === col.key); + const shown = col.key === "processed" ? all.slice(0, PROCESSED_LIMIT) : all; + return ( +
+
+ {col.label} + {all.length} +
+
+ {shown.map((c) => ( + + ))} + {all.length === 0 && ( +

+ Nothing here +

+ )} + {all.length > shown.length && ( +

+ + {all.length - shown.length} older +

+ )} +
+
+ ); + })} +
+ ); +} + +function Card({ card }: { card: BatchCard }) { + const complete = + card.notesDetected > 0 && card.notesPromoted === card.notesDetected; + return ( + +

{card.filename}

+

+ {card.siteName} + {card.pageCount > 0 && ` · ${card.pageCount} page${card.pageCount === 1 ? "" : "s"}`} +

+ {card.notesDetected > 0 && ( +

+ {card.notesPromoted} of {card.notesDetected} notes +

+ )} + {card.totalQty > 0 && ( +

+ {formatNumber(card.totalQty)} {card.totalUnit} +

+ )} +
+ {card.status === "processed" && complete && ( + All checks passed + )} + {card.failedChecks.map((reason) => ( + + {reason} + + ))} + {card.lastError && {card.lastError.slice(0, 60)}} +
+ + ); +} +``` + +- [ ] **Step 3: Link it from the delivery notes topbar** + +In `src/app/dashboard/financial/delivery-notes/page.tsx`, add a second button beside the existing "Upload status" one: + +```tsx + +``` + +and add `Inbox` to the `lucide-react` import at the top of that file. + +- [ ] **Step 4: Verify in the browser** + +```bash +npm run dev +``` + +Open http://localhost:6004/dashboard/financial/delivery-notes/inbox + +Expected: four columns render. With no batches yet, each shows "Nothing here". + +(`src/components/ui/badge.tsx` defines `default`, `secondary`, `destructive` and `outline` — the two used here both exist.) + +- [ ] **Step 5: Commit** + +```bash +npm run typecheck +git add src/app/dashboard/financial/delivery-notes/inbox/ \ + src/app/dashboard/financial/delivery-notes/page.tsx +git commit -m "feat(delivery-notes): add the inbox kanban board" +``` + +--- + +## Task 14: Batch detail + +**Files:** +- Create: `src/app/dashboard/financial/delivery-notes/inbox/[id]/page.tsx` +- Create: `src/app/dashboard/financial/delivery-notes/inbox/[id]/batch-detail-view.tsx` + +- [ ] **Step 1: Write the server page** + +Create `src/app/dashboard/financial/delivery-notes/inbox/[id]/page.tsx`: + +```tsx +import { notFound } from "next/navigation"; +import { pbAdmin } from "@/lib/pb"; +import { Topbar } from "@/components/layout/sidebar"; +import { BatchDetailView, type BatchNoteView } from "./batch-detail-view"; +import type { CheckResult, PageItem } from "@/lib/dn-pipeline/types"; + +export const dynamic = "force-dynamic"; + +export default async function BatchDetailPage({ + params, +}: { + params: Promise<{ id: string }>; +}) { + const { id } = await params; + const pb = await pbAdmin(); + + let batch; + try { + batch = await pb.collection("dn_batches").getOne(id, { expand: "site" }); + } catch { + notFound(); + } + + const notes = await pb.collection("dn_batch_notes").getFullList({ + filter: `batch='${id}'`, + sort: "note_index", + expand: "purchase_order", + }); + + const noteViews: BatchNoteView[] = notes.map((n) => ({ + id: n.id, + noteIndex: n.note_index ?? 0, + pageFrom: n.page_from ?? 0, + pageTo: n.page_to ?? 0, + deliveryNoteNumber: n.delivery_note_number ?? "", + poNumberRaw: n.po_number_raw ?? "", + poNumber: + (n.expand?.purchase_order as { purchase_order_number?: string } | undefined) + ?.purchase_order_number ?? "", + grossMass: n.gross_mass ?? null, + tareMass: n.tare_mass ?? null, + nettMass: n.nett_mass ?? null, + quantity: n.quantity ?? null, + unit: n.unit ?? "", + status: n.status ?? "candidate", + items: (Array.isArray(n.items) ? n.items : []) as PageItem[], + checks: (Array.isArray(n.checks) ? n.checks : []) as CheckResult[], + deliveryNoteId: n.delivery_note ?? "", + })); + + const site = batch.expand?.site as { name?: string } | undefined; + + return ( + <> + +
+ +
+ + ); +} +``` + +- [ ] **Step 2: Write the detail view** + +Create `src/app/dashboard/financial/delivery-notes/inbox/[id]/batch-detail-view.tsx`: + +```tsx +"use client"; + +import * as React from "react"; +import { Badge } from "@/components/ui/badge"; +import { Button } from "@/components/ui/button"; +import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; +import { formatNumber } from "@/lib/utils"; +import type { CheckResult, PageItem } from "@/lib/dn-pipeline/types"; + +export interface BatchNoteView { + id: string; + noteIndex: number; + pageFrom: number; + pageTo: number; + deliveryNoteNumber: string; + poNumberRaw: string; + poNumber: string; + grossMass: number | null; + tareMass: number | null; + nettMass: number | null; + quantity: number | null; + unit: string; + status: string; + items: PageItem[]; + checks: CheckResult[]; + deliveryNoteId: string; +} + +export function BatchDetailView({ + batchId, + status, + pageCount, + batchChecks, + notes, +}: { + batchId: string; + status: string; + pageCount: number; + batchChecks: CheckResult[]; + notes: BatchNoteView[]; +}) { + const [busy, setBusy] = React.useState(false); + + async function rerun() { + setBusy(true); + try { + await fetch(`/api/dn-batches/${batchId}/rerun`, { method: "POST" }); + window.location.reload(); + } finally { + setBusy(false); + } + } + + async function dismiss() { + if (!confirm("Mark this batch as already ingested? No delivery notes will be created from it.")) return; + setBusy(true); + try { + await fetch(`/api/dn-batches/${batchId}/dismiss`, { method: "POST" }); + window.location.reload(); + } finally { + setBusy(false); + } + } + + // Pages belonging to no note — the failure that used to lose notes + // silently. Show them first; they are the thing to act on. + const claimed = new Set(); + for (const n of notes) { + for (let p = n.pageFrom; p <= n.pageTo; p++) claimed.add(p); + } + const orphanPages = Array.from({ length: pageCount }, (_, i) => i + 1).filter( + (p) => !claimed.has(p), + ); + + return ( +
+
+ {status} + + +
+ + {batchChecks.filter((c) => c.passed === false).length > 0 && ( + + + Batch problems + + + {batchChecks + .filter((c) => c.passed === false) + .map((c) => ( +

+ {c.id}: {c.reason} +

+ ))} + + + )} + + {orphanPages.length > 0 && ( + + + + Pages belonging to no note: {orphanPages.join(", ")} + + + + {orphanPages.map((p) => ( + + ))} + + + )} + + {notes.map((n) => ( + + + + {n.deliveryNoteNumber || "(no note number)"} + + page{n.pageFrom === n.pageTo ? "" : "s"} {n.pageFrom} + {n.pageFrom === n.pageTo ? "" : `–${n.pageTo}`} + + + {n.status} + + + + +
+ {Array.from({ length: n.pageTo - n.pageFrom + 1 }, (_, i) => n.pageFrom + i).map( + (p) => ( + + ), + )} +
+
+
+ + + + + + +
+ +
    + {n.items.map((it, i) => ( +
  • + {it.description} + + {formatNumber(it.qty)} {it.unit ?? ""} + +
    • + ))} +
+ +
    + {n.checks.map((c) => ( +
  • + {c.passed === false ? "✗" : c.passed === null ? "–" : "✓"} {c.id}: {c.reason} +
    • + ))} +
+
+ + + ))} +
+ ); +} + +function Field({ label, value }: { label: string; value: string }) { + return ( + <> +
{label}
+
{value}
+ + ); +} + +function PageImage({ batchId, page }: { batchId: string; page: number }) { + return ( + + {/* eslint-disable-next-line @next/next/no-img-element */} + {`Page + p{page} + + ); +} +``` + +- [ ] **Step 3: Add the note editor** + +The detail view renders fields read-only. Add an editor so a misread value or a wrong page grouping can be corrected, then re-checked. + +Append to `batch-detail-view.tsx`: + +```tsx +const EDITABLE_FIELDS: { key: string; label: string; numeric: boolean }[] = [ + { key: "delivery_note_number", label: "Note number", numeric: false }, + { key: "po_number_raw", label: "Order no", numeric: false }, + { key: "quantity", label: "Quantity", numeric: true }, + { key: "unit", label: "Unit", numeric: false }, + { key: "gross_mass", label: "Gross", numeric: true }, + { key: "tare_mass", label: "Tare", numeric: true }, + { key: "nett_mass", label: "Nett", numeric: true }, + { key: "page_from", label: "First page", numeric: true }, + { key: "page_to", label: "Last page", numeric: true }, +]; + +function NoteEditor({ + batchId, + note, +}: { + batchId: string; + note: BatchNoteView; +}) { + const [open, setOpen] = React.useState(false); + const [saving, setSaving] = React.useState(false); + const [error, setError] = React.useState(""); + + const initial: Record = { + delivery_note_number: note.deliveryNoteNumber, + po_number_raw: note.poNumberRaw, + quantity: note.quantity === null ? "" : String(note.quantity), + unit: note.unit, + gross_mass: note.grossMass === null ? "" : String(note.grossMass), + tare_mass: note.tareMass === null ? "" : String(note.tareMass), + nett_mass: note.nettMass === null ? "" : String(note.nettMass), + page_from: String(note.pageFrom), + page_to: String(note.pageTo), + }; + const [values, setValues] = React.useState(initial); + + if (note.status === "promoted") { + return ( +

+ Promoted — delete the delivery note before editing this candidate. +

+ ); + } + + if (!open) { + return ( + + ); + } + + async function save() { + setError(""); + const from = Number(values.page_from); + const to = Number(values.page_to); + if (!Number.isInteger(from) || !Number.isInteger(to) || from < 1 || from > to) { + setError("Page range must be whole numbers with the first page no later than the last."); + return; + } + setSaving(true); + try { + const res = await fetch(`/api/dn-batches/${batchId}/notes/${note.id}`, { + method: "PATCH", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(values), + }); + if (!res.ok) { + const body = await res.json().catch(() => ({})); + setError(body.detail || body.error || `Save failed (${res.status})`); + return; + } + window.location.reload(); + } finally { + setSaving(false); + } + } + + return ( +
+
+ {EDITABLE_FIELDS.map((f) => ( + + ))} +
+ {error &&

{error}

} +
+ + +
+

+ Saving only updates the candidate. Re-run the checks to promote it. +

+
+ ); +} +``` + +Add `import { Input } from "@/components/ui/input";` to the file's imports, and render the editor inside each note card — directly after the `
    ` of check results: + +```tsx + +``` + +- [ ] **Step 4: Note the ordering** + +This view calls three routes that **Task 15 creates**: `/rerun`, `/dismiss` and `/notes/[noteId]`. Until Task 15 lands, those buttons will 404. That is expected — the view is committed first because Task 15's routes have no UI without it, and one of the two has to go first. + +Do not verify the buttons here. The correction loop is verified in Task 15 Step 5, once both halves exist. + +- [ ] **Step 5: Typecheck and commit** + +```bash +npm run typecheck +git add src/app/dashboard/financial/delivery-notes/inbox/\[id\]/ +git commit -m "feat(delivery-notes): add the batch detail view with note correction" +``` + +--- + +## Task 15: Correction actions + +Spec §5.4 requires four actions beyond viewing: edit an extracted field, re-run the gate, reassign a page to a different note, and promote a single note. + +Three routes cover all four. **"Promote a single note" needs no route of its own** — `runBatch` promotes every note that passes and leaves every note that does not, so correcting one note and re-running promotes exactly that note. A separate promote action would be a second path into `delivery_notes` that bypasses the checks, which is the thing this whole pipeline exists to prevent. + +**Files:** +- Create: `src/app/api/dn-batches/[id]/rerun/route.ts` +- Create: `src/app/api/dn-batches/[id]/dismiss/route.ts` +- Create: `src/app/api/dn-batches/[id]/notes/[noteId]/route.ts` + +- [ ] **Step 1: Write the re-run route** + +Create `src/app/api/dn-batches/[id]/rerun/route.ts`: + +```ts +import { NextResponse } from "next/server"; +import { pbAdmin } from "@/lib/pb"; +import { runBatch } from "@/lib/dn-pipeline/run"; + +export const dynamic = "force-dynamic"; +export const runtime = "nodejs"; +export const maxDuration = 300; + +/** + * POST /api/dn-batches/[id]/rerun + * + * Re-runs the pipeline over a batch after a manual correction. Safe to + * call repeatedly: runBatch upserts its candidate notes by note_index + * and promoteNote updates the delivery note it already created. + */ +export async function POST( + _request: Request, + { params }: { params: Promise<{ id: string }> }, +) { + const { id } = await params; + if (!/^[a-z0-9]{15}$/i.test(id)) { + return NextResponse.json({ error: "invalid_batch_id" }, { status: 400 }); + } + + const pb = await pbAdmin(); + try { + await pb.collection("dn_batches").update(id, { + status: "analysing", + claimed_at: new Date().toISOString(), + last_error: "", + }); + const result = await runBatch(pb, id); + return NextResponse.json({ ok: true, ...result }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + await pb + .collection("dn_batches") + .update(id, { status: "needs_attention", claimed_at: "", last_error: message.slice(0, 2000) }) + .catch(() => undefined); + return NextResponse.json({ error: message }, { status: 500 }); + } +} +``` + +- [ ] **Step 2: Write the dismiss route** + +Create `src/app/api/dn-batches/[id]/dismiss/route.ts`: + +```ts +import { NextResponse } from "next/server"; +import { pbAdmin } from "@/lib/pb"; + +export const dynamic = "force-dynamic"; +export const runtime = "nodejs"; + +/** + * POST /api/dn-batches/[id]/dismiss + * + * Marks a batch processed without promoting anything. + * + * Needed because the four notes of scan_20260915_c5p7fkmzgv.pdf were + * captured by hand during phase 1, so when this pipeline reads that + * file the not_duplicate check will correctly block every note. Without + * this action that card would sit in Needs attention forever. + */ +export async function POST( + _request: Request, + { params }: { params: Promise<{ id: string }> }, +) { + const { id } = await params; + if (!/^[a-z0-9]{15}$/i.test(id)) { + return NextResponse.json({ error: "invalid_batch_id" }, { status: 400 }); + } + + const pb = await pbAdmin(); + try { + await pb.collection("dn_batches").update(id, { + status: "processed", + notes_promoted: 0, + claimed_at: "", + last_error: "", + checks: [ + { + id: "dismissed", + passed: true, + reason: "marked as already ingested by an operator", + }, + ], + }); + return NextResponse.json({ ok: true }); + } catch (err) { + return NextResponse.json( + { error: err instanceof Error ? err.message : "dismiss_failed" }, + { status: 500 }, + ); + } +} +``` + +- [ ] **Step 3: Write the note correction route** + +Create `src/app/api/dn-batches/[id]/notes/[noteId]/route.ts`: + +```ts +import { NextResponse } from "next/server"; +import { pbAdmin } from "@/lib/pb"; + +export const dynamic = "force-dynamic"; +export const runtime = "nodejs"; + +/** + * PATCH /api/dn-batches/[id]/notes/[noteId] + * + * Operator correction of one candidate note: a misread field, or a page + * range that grouped wrongly. + * + * This route only edits the CANDIDATE. It never touches delivery_notes + * and never promotes — the caller re-runs afterwards, and the correction + * goes through the same checks as everything else. Letting an edit write + * straight to the ledger would reintroduce exactly the unvalidated path + * this pipeline replaced. + * + * Editing page_from/page_to is how a page is reassigned: widen one + * note's range or narrow another's, then re-run. + */ +const EDITABLE = new Set([ + "delivery_note_number", + "po_number_raw", + "quantity", + "unit", + "gross_mass", + "tare_mass", + "nett_mass", + "page_from", + "page_to", +]); + +const NUMERIC = new Set(["quantity", "gross_mass", "tare_mass", "nett_mass", "page_from", "page_to"]); + +export async function PATCH( + request: Request, + { params }: { params: Promise<{ id: string; noteId: string }> }, +) { + const { id, noteId } = await params; + if (!/^[a-z0-9]{15}$/i.test(id) || !/^[a-z0-9]{15}$/i.test(noteId)) { + return NextResponse.json({ error: "invalid_id" }, { status: 400 }); + } + + let body: Record; + try { + body = await request.json(); + } catch { + return NextResponse.json({ error: "invalid_json" }, { status: 400 }); + } + + const patch: Record = {}; + const rejected: string[] = []; + for (const [key, value] of Object.entries(body)) { + if (!EDITABLE.has(key)) { + rejected.push(key); + continue; + } + if (NUMERIC.has(key)) { + if (value === null || value === "") { + patch[key] = null; + continue; + } + const n = Number(value); + if (!Number.isFinite(n)) { + return NextResponse.json({ error: "not_a_number", field: key }, { status: 400 }); + } + patch[key] = n; + } else { + patch[key] = String(value ?? ""); + } + } + + if (Object.keys(patch).length === 0) { + return NextResponse.json({ error: "nothing_editable", rejected }, { status: 400 }); + } + + if ( + typeof patch.page_from === "number" && + typeof patch.page_to === "number" && + patch.page_from > patch.page_to + ) { + return NextResponse.json({ error: "page_range_inverted" }, { status: 400 }); + } + + const pb = await pbAdmin(); + try { + const note = await pb + .collection("dn_batch_notes") + .getOne(noteId, { fields: "id,batch,status" }); + if (note.batch !== id) { + return NextResponse.json({ error: "note_not_in_batch" }, { status: 400 }); + } + if (note.status === "promoted") { + // A promoted note already has a delivery_notes row behind it. + // Editing the candidate would silently desync the two. + return NextResponse.json( + { error: "already_promoted", detail: "delete the delivery note first, then re-run" }, + { status: 409 }, + ); + } + await pb.collection("dn_batch_notes").update(noteId, patch); + return NextResponse.json({ ok: true, updated: Object.keys(patch), rejected }); + } catch (err) { + return NextResponse.json( + { error: err instanceof Error ? err.message : "update_failed" }, + { status: 500 }, + ); + } +} +``` + +- [ ] **Step 4: Typecheck and commit** + +```bash +npm run typecheck +git add src/app/api/dn-batches/\[id\]/rerun/ src/app/api/dn-batches/\[id\]/dismiss/ \ + src/app/api/dn-batches/\[id\]/notes/ +git commit -m "feat(dn-batches): add re-run, dismiss and note correction actions" +``` + +- [ ] **Step 5: Verify the correction loop** + +The buttons added in Task 14 now have routes behind them. With a batch sitting in **needs attention**, open it and exercise each one: + +1. Click "Correct this note", change the quantity, click Save. Expected: the page reloads showing the new value. +2. Enter an inverted page range (first page later than last) and Save. Expected: the inline error appears and nothing is submitted. +3. Click "Re-run checks". Expected: the checks re-evaluate against the corrected values; a note that now passes everything is promoted and its badge turns to `promoted`. +4. On a promoted note, confirm the editor shows "Promoted — delete the delivery note before editing this candidate" rather than input fields. +5. Click "Already ingested" on a blocked batch and confirm. Expected: the batch moves to the **Processed** column with `0` notes promoted. + +--- + +## Task 16: End-to-end verification + +- [ ] **Step 1: Upload the reference document** + +Start the dev server, open `/dashboard/financial/delivery-notes`, click Upload, pick the site **The Hills**, and upload `src/lib/dn-pipeline/__fixtures__/four-note-scan.pdf`. + +Expected: a card appears in the **Inbox** column. + +- [ ] **Step 2: Run the tick** + +```bash +curl -s -X POST http://localhost:6004/api/dn-batches/tick \ + -H "x-ingest-secret: $(grep '^N8N_INGEST_SECRET=' .env.local | cut -d= -f2-)" | python3 -m json.tool +``` + +Expected: `"notesDetected": 4`, `"notesPromoted": 4`, `"status": "processed"`. + +Because phase 1 captured these four notes by hand, `not_duplicate` will fail on all four and you will instead see `"notesPromoted": 0` and `"status": "needs_attention"` with `DQ0010107 is already in delivery notes`. **That is the correct result.** To exercise the promotion path, delete the four existing rows first: + +```bash +python3 - <<'EOF' +import sys; sys.path.insert(0, "scripts") +from pb_env import http, superuser_token +t = superuser_token() +for dn in ("u27fj2ddxgs138l", "bfhopa5zp73fbi1", "jn154emn5c1h2ho", "2dkwx357mnexrk8"): + print(dn, http("DELETE", f"/api/collections/delivery_notes/records/{dn}", token=t)[0]) +EOF +``` + +- [ ] **Step 3: Check the board** + +Reload `/dashboard/financial/delivery-notes/inbox`. + +Expected: the card sits in **Processed**, reads `4 of 4 notes`, and shows a combined quantity of **133 500 kg**. + +- [ ] **Step 4: Check the detail view** + +Open the card. + +Expected: four note cards, each with its own page image, each showing `weighbridge_arithmetic ✓` with the actual sum (`48760 − 18800 = 29960`), and `DQ0010239` present with 34 200 kg — the note the old pipeline lost. + +- [ ] **Step 5: Verify the ledger** + +```bash +python3 - <<'EOF' +import sys; sys.path.insert(0, "scripts") +from pb_env import http, superuser_token +t = superuser_token() +_, r = http("GET", "/api/collections/delivery_note_items/records" + "?perPage=200&filter=purchase_order_item='bxqbo8rs04pjqwc'" + "&fields=qty_received,qty_raw,unit_raw", token=t) +tot = sum(i["qty_received"] for i in r["items"]) +print(f"lines {len(r['items'])} delivered {tot} ton value R{tot * 168.50:,.2f}") +EOF +``` + +Expected: `lines 4 delivered 133.5 ton value R22,494.75`. + +- [ ] **Step 6: Verify idempotency** + +Run the tick again, then re-run step 5. + +Expected: identical figures. No duplicated rows. + +- [ ] **Step 7: Verify the missing-page guard** + +Temporarily make `extract.ts` throw for page 3: + +```ts +export async function extractPage(png: Buffer, page: number): Promise { + if (page === 3) throw new Error("simulated page read failure"); +``` + +Delete the four delivery notes again, re-run the tick. + +Expected: the batch lands in **needs_attention**, `notesPromoted` is 3, and the checks name both `page_3_read` and `notes_reconciled: only 3 of 4 notes promoted`. **It must not reach `processed`.** This is the regression the whole pipeline exists to prevent — confirm it, then revert the edit. + +--- + +## Task 17: Reduce n8n and reconcile the old view + +- [ ] **Step 1: Replace the workflow** + +In n8n, open `Delivery Note OCR` (`wVbLw4paISkbrIpE`). Delete every node except the schedule trigger. Set the trigger to 1 minute. Add one HTTP Request node: + +- Method: `POST` +- URL: `https://mega.dash.apphub.co.za/api/dn-batches/tick` +- Header: `x-ingest-secret` — **from a credential**, not a pasted literal + +Connect trigger → HTTP request. Save, keep active. + +- [ ] **Step 2: Rotate the secret** + +The old value was pasted as a literal into four nodes and is in every export of that workflow. Generate a new one, set `N8N_INGEST_SECRET` in Coolify and `.env.local`, and put it in the n8n credential. This is a production credential change — do it yourself in the Coolify and n8n UIs. + +- [ ] **Step 3: Reconcile the delivery notes filter** + +In `src/app/dashboard/financial/delivery-notes/delivery-notes-view.tsx:70`, the `StatusFilter` union omits `ocr-complete`, which is a real value in the PocketBase select — which is why the two lost notes were invisible. Add it: + +```ts +type StatusFilter = + | "all" + | "needs-review" + | "auto-verified" + | "manually-verified" + | "ocr-complete" + | "pending" + | "failed"; +``` + +and add a matching `OCR complete` to the status select. + +- [ ] **Step 4: Add the guard test** + +Create `src/app/dashboard/financial/delivery-notes/status-filter.test.ts`: + +```ts +import { describe, it, expect } from "vitest"; +import { STATUS_FILTER_VALUES } from "./delivery-notes-view"; + +/** + * Every value in the PocketBase DELIVERY_NOTES.processing_status select. + * A value the filter does not know about is a row nobody can find — + * that is how DQ0010239 and DQ0010217 stayed invisible. + */ +const PB_STATUSES = [ + "pending", + "ocr-complete", + "auto-verified", + "needs-review", + "manual-verified", + "failed", +]; + +describe("delivery note status filter", () => { + it("covers every processing_status the collection allows", () => { + for (const s of PB_STATUSES) { + expect(STATUS_FILTER_VALUES).toContain(s); + } + }); +}); +``` + +For this to work, export the values from `delivery-notes-view.tsx`: + +```ts +export const STATUS_FILTER_VALUES = [ + "all", + "needs-review", + "auto-verified", + "manually-verified", + "manual-verified", + "ocr-complete", + "pending", + "failed", +] as const; + +type StatusFilter = (typeof STATUS_FILTER_VALUES)[number]; +``` + +Note `manual-verified` (PocketBase's spelling) and `manually-verified` (the view's existing label) are both listed — the view uses the latter as a pseudo-filter meaning "is_verified is true". Keep both rather than renaming one and breaking the existing behaviour. + +- [ ] **Step 5: Run everything and commit** + +```bash +npm run test:run && npm run typecheck +git add src/app/dashboard/financial/delivery-notes/ +git commit -m "fix(delivery-notes): surface every processing_status in the filter" +``` + +- [ ] **Step 6: Update the n8n doc** + +Rewrite `docs/n8n/delivery-note-ocr.md` to describe the two-node workflow, with a link to the spec and a note that all pipeline logic now lives in `src/lib/dn-pipeline/`. Delete the sections describing the old polling / split behaviour so nobody rebuilds it. + +```bash +git add docs/n8n/delivery-note-ocr.md +git commit -m "docs(n8n): rewrite for the two-node tick workflow" +``` + +--- + +## Done when + +- [ ] `npm run test:run` and `npm run typecheck` pass. +- [ ] `docker build` succeeds and `pdftoppm -v` runs as the `nextjs` user. +- [ ] Uploading the four-note scan produces `4 of 4 notes` and `133 500 kg`. +- [ ] Forcing a page-read failure leaves the batch in **needs attention** with `notes_reconciled` failed — it never reaches processed. +- [ ] Running the tick twice produces identical figures and no duplicate rows. +- [ ] PO `MEGCPO9165` line `bxqbo8rs04pjqwc` shows 133.5 ton delivered, R22 494.75. +- [ ] The n8n workflow has exactly two nodes and reads its secret from a credential. +- [ ] No value of `processing_status` is missing from the delivery notes filter. + +--- + +## Not in this plan + +**Agent escalation.** When segmentation is ambiguous (`unassigned` or `needsSubsplit` non-empty), the batch currently stops for a human. Handing that case to a reasoning agent that proposes a different segmentation — which then passes back through the same `runNoteChecks` / `runBatchChecks` gate — is the natural next step once the board shows which documents actually need it. Build it from evidence, not in advance. + +**Phase 4, purchase order cleanup.** The dead `pending → processing → processed` state machine (all 24 POs sit at `processing_status: ""`), `#` prefix normalisation at PO ingest, and the bad rows `#MEGCP09109` and `IN132201`. Not yet specified.