diff --git a/docs/superpowers/plans/2026-09-17-delivery-note-unit-correctness.md b/docs/superpowers/plans/2026-09-17-delivery-note-unit-correctness.md new file mode 100644 index 0000000..3345ff0 --- /dev/null +++ b/docs/superpowers/plans/2026-09-17-delivery-note-unit-correctness.md @@ -0,0 +1,1586 @@ +# Delivery Note Unit Correctness Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop delivery note quantities entering payment figures in the wrong unit, and fix the off-by-one that links a line to the wrong purchase order item. + +**Architecture:** A pure `units` module owns unit normalisation and conversion. A second pure module, `dn-item-row`, turns raw OCR line items into the exact rows written to `delivery_note_items` — filtering once, so the matched-id pairing cannot drift, and converting quantities into the PO line's unit. The OCR callback route becomes a thin caller of both. `qty_received` keeps its name and becomes strictly "in the PO line's unit", so `po-fulfilment.ts` and `delivery-note-value.ts` become correct without their logic changing. + +**Tech Stack:** TypeScript, Next.js 16 App Router, Vitest (`src/**/*.test.ts`, colocated), PocketBase via the JS SDK, Python 3 for schema and data migrations (`scripts/`, using `scripts/pb_env.py`). + +**Spec:** [docs/superpowers/specs/2026-09-17-delivery-note-inbox-design.md](../specs/2026-09-17-delivery-note-inbox-design.md) §3. + +--- + +## Background you need + +Delivery notes are weighbridge tickets quoting **kg** (`29 960.00 kg`). The purchase order lines they belong to are priced per **ton** (`4 273.89 ton @ R168.50`). Today `/api/delivery-notes/[id]/ocr-result` writes `qty_received: 29960` straight against a ton-priced line and nothing converts, so: + +- `src/lib/po-fulfilment.ts:53` computes `4273.89 − 64780` and reports that PO line as 1 515 % delivered. +- `src/lib/metrics/delivery-note-value.ts:22` computes `64780 × 168.50 = R10 915 430` of received value where the true figure is **R10 915.43**. + +`delivery_note_items` has no unit column at all — its fields are `delivery_note`, `purchase_order_item`, `qty_received`, `condition_notes`. The unit the model read is discarded into `condition_notes` as free text, in this format: + +``` +OCR 95% · desc: G6 Sub Base · unit: kg · match: 100% +``` + +That format is load-bearing for Task 7, which recovers the unit from it. + +**Never guess a conversion.** If a unit pair is not in the table in Task 2, the line contributes zero and surfaces for manual completion. A guessed conversion is what produced the current 1000× error. + +## File structure + +| File | Responsibility | +| --- | --- | +| Create `src/lib/units.ts` | Unit aliases and conversion. Pure, no I/O. | +| Create `src/lib/units.test.ts` | Tests for the above. | +| Create `src/lib/dn-item-row.ts` | OCR line items → `delivery_note_items` row payloads. Pure. | +| Create `src/lib/dn-item-row.test.ts` | Tests for the above. | +| Create `scripts/pb-add-dn-item-units.py` | Adds `qty_raw`, `unit_raw`, `unit_converted` to the collection. | +| Create `scripts/migrate-dn-item-units.py` | Backfills the 57 existing rows. | +| Modify `src/app/api/delivery-notes/[id]/ocr-result/route.ts` | Call the two new modules instead of inlining the logic. | +| Modify `src/lib/pb-types.ts:319-330` | Add the three new fields to `DeliveryNoteItemRecord`. | +| Modify `src/lib/utils.ts` | Add `formatQtyDual`. | +| Modify `src/app/dashboard/financial/purchase-orders/[poId]/po-detail-view.tsx` | Render both units. | + +--- + +## Task 1: Unit normalisation + +**Files:** +- Create: `src/lib/units.ts` +- Test: `src/lib/units.test.ts` + +- [ ] **Step 1: Write the failing test** + +Create `src/lib/units.test.ts`: + +```ts +import { describe, it, expect } from "vitest"; +import { normaliseUnit } from "./units"; + +describe("normaliseUnit", () => { + it("returns null for absent input", () => { + expect(normaliseUnit(null)).toBeNull(); + expect(normaliseUnit(undefined)).toBeNull(); + expect(normaliseUnit("")).toBeNull(); + expect(normaliseUnit(" ")).toBeNull(); + }); + + it("maps mass aliases", () => { + for (const a of ["kg", "KG", " Kgs ", "kilo", "kilos", "kilogram", "kilograms"]) { + expect(normaliseUnit(a)).toBe("kg"); + } + for (const a of ["t", "T", "ton", "tons", "tonne", "tonnes", "mt", "metric ton"]) { + expect(normaliseUnit(a)).toBe("ton"); + } + }); + + it("maps volume and area aliases", () => { + for (const a of ["m3", "m³", "cube", "cubes", "cubic metre", "cubic meter"]) { + expect(normaliseUnit(a)).toBe("m3"); + } + for (const a of ["m2", "m²", "square metre", "square meter"]) { + expect(normaliseUnit(a)).toBe("m2"); + } + for (const a of ["l", "litre", "litres", "liter", "liters"]) { + expect(normaliseUnit(a)).toBe("l"); + } + }); + + it("maps count and length aliases", () => { + for (const a of ["ea", "each", "unit", "units", "no", "nr", "item", "items"]) { + expect(normaliseUnit(a)).toBe("ea"); + } + for (const a of ["m", "metre", "meter", "metres", "meters", "lm", "linear metre"]) { + expect(normaliseUnit(a)).toBe("m"); + } + for (const a of ["bag", "bags", "pocket", "pockets"]) { + expect(normaliseUnit(a)).toBe("bag"); + } + for (const a of ["load", "loads", "trip", "trips"]) { + expect(normaliseUnit(a)).toBe("load"); + } + }); + + it("strips trailing punctuation the OCR leaves behind", () => { + expect(normaliseUnit("kg.")).toBe("kg"); + expect(normaliseUnit("(ton)")).toBe("ton"); + }); + + it("returns null for an unrecognised unit rather than guessing", () => { + expect(normaliseUnit("furlong")).toBeNull(); + expect(normaliseUnit("??")).toBeNull(); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run src/lib/units.test.ts` +Expected: FAIL — `Failed to resolve import "./units"`. + +- [ ] **Step 3: Write minimal implementation** + +Create `src/lib/units.ts`: + +```ts +/** + * Unit normalisation and conversion for delivery note quantities. + * + * Delivery notes are weighbridge tickets quoting kg; the purchase order + * lines they belong to are priced per ton. Writing the slip's number + * against a ton-priced line inflates received value 1000x, which is + * exactly what happened before this module existed. + * + * The conversion table is deliberately closed. An unrecognised unit or + * an unlisted pair returns a failure — callers must surface that as + * work for a human, never fall back to "assume they match". + */ + +const ALIASES: Record = { + kg: ["kg", "kgs", "kilo", "kilos", "kilogram", "kilograms"], + ton: ["t", "ton", "tons", "tonne", "tonnes", "mt", "metric ton"], + m3: ["m3", "m³", "cube", "cubes", "cubic metre", "cubic meter"], + m2: ["m2", "m²", "square metre", "square meter"], + m: ["m", "metre", "meter", "metres", "meters", "lm", "linear metre"], + ea: ["ea", "each", "unit", "units", "no", "nr", "item", "items"], + bag: ["bag", "bags", "pocket", "pockets"], + l: ["l", "litre", "litres", "liter", "liters"], + load: ["load", "loads", "trip", "trips"], +}; + +const LOOKUP: Map = new Map(); +for (const [canonical, aliases] of Object.entries(ALIASES)) { + for (const alias of aliases) LOOKUP.set(alias, canonical); +} + +/** + * Canonical token for a unit as written on a slip or PO line, or null + * when we do not recognise it. Null is a real answer, not an error — + * callers must treat it as "cannot convert". + */ +export function normaliseUnit(raw: string | null | undefined): string | null { + if (typeof raw !== "string") return null; + // Keep the superscript characters in m² / m³; strip the punctuation + // OCR sprinkles around units ("kg.", "(ton)"). + const cleaned = raw + .toLowerCase() + .replace(/[^a-z0-9²³ ]/g, " ") + .replace(/\s+/g, " ") + .trim(); + if (!cleaned) return null; + return LOOKUP.get(cleaned) ?? null; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run src/lib/units.test.ts` +Expected: PASS, 6 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/units.ts src/lib/units.test.ts +git commit -m "feat(units): add unit alias normalisation" +``` + +--- + +## Task 2: Unit conversion + +**Files:** +- Modify: `src/lib/units.ts` +- Test: `src/lib/units.test.ts` + +- [ ] **Step 1: Write the failing test** + +Append to `src/lib/units.test.ts`: + +```ts +import { convertQty } from "./units"; + +describe("convertQty", () => { + it("passes through when both units are the same", () => { + expect(convertQty(29960, "kg", "kg")).toEqual({ + ok: true, + qty: 29960, + converted: false, + }); + }); + + it("converts kg to ton", () => { + expect(convertQty(29960, "kg", "ton")).toEqual({ + ok: true, + qty: 29.96, + converted: true, + }); + }); + + it("converts ton to kg", () => { + expect(convertQty(29.96, "ton", "kg")).toEqual({ + ok: true, + qty: 29960, + converted: true, + }); + }); + + it("converts litres and cubic metres both ways", () => { + expect(convertQty(2500, "l", "m3")).toEqual({ ok: true, qty: 2.5, converted: true }); + expect(convertQty(2.5, "m3", "l")).toEqual({ ok: true, qty: 2500, converted: true }); + }); + + it("rounds to four decimal places", () => { + expect(convertQty(1, "kg", "ton")).toEqual({ ok: true, qty: 0.001, converted: true }); + expect(convertQty(12345, "kg", "ton")).toEqual({ ok: true, qty: 12.345, converted: true }); + }); + + it("fails on an unknown unit rather than assuming a match", () => { + expect(convertQty(10, null, "ton")).toEqual({ ok: false, reason: "unknown_unit" }); + expect(convertQty(10, "kg", null)).toEqual({ ok: false, reason: "unknown_unit" }); + expect(convertQty(10, null, null)).toEqual({ ok: false, reason: "unknown_unit" }); + }); + + it("fails on a pair that is not in the table", () => { + expect(convertQty(10, "kg", "m3")).toEqual({ ok: false, reason: "incompatible_units" }); + expect(convertQty(10, "ea", "bag")).toEqual({ ok: false, reason: "incompatible_units" }); + expect(convertQty(10, "m2", "m")).toEqual({ ok: false, reason: "incompatible_units" }); + }); + + it("accepts raw aliases, not just canonical tokens", () => { + expect(convertQty(1000, "Kgs", "tonnes")).toEqual({ + ok: true, + qty: 1, + converted: true, + }); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run src/lib/units.test.ts` +Expected: FAIL — `convertQty is not a function` / no export named `convertQty`. + +- [ ] **Step 3: Write minimal implementation** + +Append to `src/lib/units.ts`: + +```ts +export type UnitConversion = + | { ok: true; qty: number; converted: boolean } + | { ok: false; reason: "unknown_unit" | "incompatible_units" }; + +/** + * The complete set of conversions. Anything absent here fails. + * Keyed ">>" on canonical tokens. + */ +const FACTORS: Record = { + "kg>>>ton": 1 / 1000, + "ton>>>kg": 1000, + "l>>>m3": 1 / 1000, + "m3>>>l": 1000, +}; + +/** + * Convert `qty` from one unit into another. Accepts raw slip/PO text on + * either side — it normalises both before looking anything up. + * + * A null on either side is `unknown_unit`, never "assume they match": + * assuming is what made every received value 1000x too large. + */ +export function convertQty( + qty: number, + fromUnit: string | null, + toUnit: string | null, +): UnitConversion { + const from = normaliseUnit(fromUnit); + const to = normaliseUnit(toUnit); + if (from === null || to === null) return { ok: false, reason: "unknown_unit" }; + if (from === to) return { ok: true, qty, converted: false }; + + const factor = FACTORS[`${from}>>>${to}`]; + if (factor === undefined) return { ok: false, reason: "incompatible_units" }; + + return { + ok: true, + qty: Math.round(qty * factor * 10000) / 10000, + converted: true, + }; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run src/lib/units.test.ts` +Expected: PASS, 14 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/units.ts src/lib/units.test.ts +git commit -m "feat(units): add closed-table quantity conversion" +``` + +--- + +## Task 3: Add the schema fields + +**Files:** +- Create: `scripts/pb-add-dn-item-units.py` +- Modify: `src/lib/pb-types.ts:319-330` + +- [ ] **Step 1: Write the migration script** + +Create `scripts/pb-add-dn-item-units.py`: + +```python +#!/usr/bin/env python3 +""" +Adds qty_raw / unit_raw / unit_converted to the delivery_note_items +collection. + +qty_received keeps its name but changes meaning: it becomes strictly +"in the purchase order line's unit". qty_raw and unit_raw preserve what +the slip actually said, so the audit trail survives the conversion and +the UI can show both. + +Idempotent — re-running on a collection that already has the fields is +a no-op. + +Credentials come from .env.local via pb_env. +""" +import sys + +from pb_env import PB, http, superuser_token + + +def get_collection(token: str, name: str): + code, resp = http("GET", f"/api/collections/{name}", token=token) + if code != 200: + sys.exit(f"GET collection {name} failed: {code} {resp}") + return resp + + +def main() -> None: + token = superuser_token() + print(f"\n→ delivery_note_items on {PB}") + coll = get_collection(token, "delivery_note_items") + fields = list(coll.get("fields", [])) + existing = {f.get("name") for f in fields} + + wanted = [ + {"name": "qty_raw", "type": "number", "required": False}, + {"name": "unit_raw", "type": "text", "required": False}, + {"name": "unit_converted", "type": "bool", "required": False}, + ] + + additions = [] + for want in wanted: + if want["name"] in existing: + print(f" • {want['name']:15s} already exists — skipping") + else: + additions.append(want) + + if not additions: + print(" • no changes needed") + return + + code, resp = http( + "PATCH", + f"/api/collections/{coll['id']}", + {"fields": fields + additions}, + token, + ) + if code != 200: + sys.exit(f"PATCH failed: {code} {resp}") + for a in additions: + print(f" ✓ added {a['name']}") + + +if __name__ == "__main__": + main() + print("\nDone.") +``` + +- [ ] **Step 2: Run it** + +```bash +python3 scripts/pb-add-dn-item-units.py +``` + +Expected output: + +``` +auth ok on https://pb.mega.infra.apphub.co.za + +→ delivery_note_items on https://pb.mega.infra.apphub.co.za + ✓ added qty_raw + ✓ added unit_raw + ✓ added unit_converted + +Done. +``` + +- [ ] **Step 3: Verify idempotency** + +Run the same command again. Expected: three `already exists — skipping` lines and `no changes needed`. + +- [ ] **Step 4: Update the TypeScript record type** + +In `src/lib/pb-types.ts`, replace the `DeliveryNoteItemRecord` interface (currently lines 319-330) with: + +```ts +export interface DeliveryNoteItemRecord extends RecordModel { + collectionName: "delivery_note_items"; + delivery_note: string; + /** Optional — empty when OCR could not match the line to a PO item. */ + purchase_order_item: string; + /** In the matched PO line's unit. Zero when the line is unmatched or unconvertible. */ + qty_received: number; + /** Verbatim quantity from the slip, before any conversion. */ + qty_raw: number; + /** Verbatim unit from the slip, e.g. "kg". Empty when the slip showed none. */ + unit_raw: string; + /** True when qty_received differs from qty_raw because a conversion was applied. */ + unit_converted: boolean; + condition_notes: string; + expand?: { + delivery_note?: DeliveryNoteRecord; + purchase_order_item?: PurchaseOrderItemRecord; + }; +} +``` + +- [ ] **Step 5: Typecheck and commit** + +```bash +npm run typecheck +git add scripts/pb-add-dn-item-units.py src/lib/pb-types.ts +git commit -m "feat(delivery-notes): add qty_raw/unit_raw/unit_converted to line items" +``` + +Expected: typecheck passes with no errors. + +--- + +## Task 4: Build delivery note item rows + +This is where the off-by-one dies. The current route builds `suppliedMatchedIds` from **all** items but consumes `validIds[itemIndex++]` only for items surviving the `qty > 0` check, so one zero-quantity line shifts every later line onto the wrong PO item. Filtering once, in one place, makes that impossible by construction. + +**Files:** +- Create: `src/lib/dn-item-row.ts` +- Test: `src/lib/dn-item-row.test.ts` + +- [ ] **Step 1: Write the failing test** + +Create `src/lib/dn-item-row.test.ts`: + +```ts +import { describe, it, expect } from "vitest"; +import { buildDeliveryNoteItemRows } from "./dn-item-row"; + +const PO_ITEMS = [ + { id: "aaaaaaaaaaaaaaa", unit: "ton" }, + { id: "bbbbbbbbbbbbbbb", unit: "ton" }, + { id: "ccccccccccccccc", unit: "ea" }, +]; + +describe("buildDeliveryNoteItemRows", () => { + it("converts the slip quantity into the PO line's unit", () => { + const { rows } = buildDeliveryNoteItemRows({ + items: [ + { description: "G6 Sub Base", qty: 29960, unit: "kg", matched_po_item_id: "aaaaaaaaaaaaaaa" }, + ], + poItems: PO_ITEMS, + confidence: 0.95, + }); + expect(rows).toHaveLength(1); + expect(rows[0].qty_received).toBe(29.96); + expect(rows[0].qty_raw).toBe(29960); + expect(rows[0].unit_raw).toBe("kg"); + expect(rows[0].unit_converted).toBe(true); + expect(rows[0].purchase_order_item).toBe("aaaaaaaaaaaaaaa"); + }); + + it("does not shift matched ids when an item is dropped for zero quantity", () => { + const { rows } = buildDeliveryNoteItemRows({ + items: [ + { description: "first", qty: 5, unit: "ton", matched_po_item_id: "aaaaaaaaaaaaaaa" }, + { description: "dropped", qty: 0, unit: "ton", matched_po_item_id: "bbbbbbbbbbbbbbb" }, + { description: "third", qty: 7, unit: "ea", matched_po_item_id: "ccccccccccccccc" }, + ], + poItems: PO_ITEMS, + confidence: 0.9, + }); + expect(rows).toHaveLength(2); + expect(rows[0].purchase_order_item).toBe("aaaaaaaaaaaaaaa"); + // Before the fix this was "bbbbbbbbbbbbbbb" — the dropped item's id. + expect(rows[1].purchase_order_item).toBe("ccccccccccccccc"); + expect(rows[1].qty_received).toBe(7); + }); + + it("zeroes the quantity and drops the link when units cannot convert", () => { + const { rows } = buildDeliveryNoteItemRows({ + items: [ + { description: "Sand", qty: 12, unit: "m3", matched_po_item_id: "aaaaaaaaaaaaaaa" }, + ], + poItems: PO_ITEMS, + confidence: 0.9, + }); + expect(rows[0].qty_received).toBe(0); + expect(rows[0].qty_raw).toBe(12); + expect(rows[0].unit_raw).toBe("m3"); + expect(rows[0].purchase_order_item).toBeNull(); + expect(rows[0].condition_notes).toContain("incompatible_units"); + }); + + it("zeroes the quantity when the slip showed no unit", () => { + const { rows } = buildDeliveryNoteItemRows({ + items: [{ description: "Stone", qty: 3, matched_po_item_id: "aaaaaaaaaaaaaaa" }], + poItems: PO_ITEMS, + confidence: 0.9, + }); + expect(rows[0].qty_received).toBe(0); + expect(rows[0].qty_raw).toBe(3); + expect(rows[0].unit_raw).toBe(""); + expect(rows[0].purchase_order_item).toBeNull(); + expect(rows[0].condition_notes).toContain("unknown_unit"); + }); + + it("keeps an unmatched line at zero with no link", () => { + const { rows } = buildDeliveryNoteItemRows({ + items: [{ description: "Mystery", qty: 4, unit: "kg" }], + poItems: PO_ITEMS, + confidence: 0.6, + }); + expect(rows[0].purchase_order_item).toBeNull(); + expect(rows[0].qty_received).toBe(0); + expect(rows[0].qty_raw).toBe(4); + expect(rows[0].unit_raw).toBe("kg"); + }); + + it("drops a matched id that is not on the PO when the PO items are known", () => { + const { rows, unverifiable } = buildDeliveryNoteItemRows({ + items: [{ description: "X", qty: 1, unit: "ton", matched_po_item_id: "zzzzzzzzzzzzzzz" }], + poItems: PO_ITEMS, + confidence: 0.9, + }); + expect(rows[0].purchase_order_item).toBeNull(); + expect(unverifiable).toBe(false); + }); + + it("trusts a matched id when the PO items could not be fetched, and flags it", () => { + const { rows, unverifiable } = buildDeliveryNoteItemRows({ + items: [{ description: "X", qty: 1, unit: "ton", matched_po_item_id: "zzzzzzzzzzzzzzz" }], + poItems: [], + confidence: 0.9, + }); + expect(rows[0].purchase_order_item).toBe("zzzzzzzzzzzzzzz"); + // No local unit to convert into, so the quantity cannot be trusted. + expect(rows[0].qty_received).toBe(0); + expect(unverifiable).toBe(true); + }); + + it("skips non-objects and negative quantities", () => { + const { rows } = buildDeliveryNoteItemRows({ + items: [null, "nope", { description: "neg", qty: -3, unit: "kg" }], + poItems: PO_ITEMS, + confidence: 0.9, + }); + expect(rows).toHaveLength(0); + }); + + it("writes condition notes in the legacy format so the migration can parse them", () => { + const { rows } = buildDeliveryNoteItemRows({ + items: [ + { + description: "G6 Sub Base", + qty: 29960, + unit: "kg", + unit_price: 168.5, + match_confidence: 1, + matched_po_item_id: "aaaaaaaaaaaaaaa", + }, + ], + poItems: PO_ITEMS, + confidence: 0.95, + }); + expect(rows[0].condition_notes).toBe( + "OCR 95% · desc: G6 Sub Base · unit: kg · rate: 168.50 · match: 100%", + ); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run src/lib/dn-item-row.test.ts` +Expected: FAIL — `Failed to resolve import "./dn-item-row"`. + +- [ ] **Step 3: Write minimal implementation** + +Create `src/lib/dn-item-row.ts`: + +```ts +/** + * Turns raw OCR line items into the exact rows written to + * delivery_note_items. + * + * Two bugs live here by construction rather than by care: + * + * 1. Matched-id drift. The route this replaced built its matched-id + * list from every item but consumed it only for items surviving the + * qty > 0 check, so one zero-quantity line shifted every later line + * onto the wrong PO item. Filtering happens once, here, and the rows + * carry their own ids — there is no second list to fall out of step. + * + * 2. Unit mismatch. Slips quote kg; PO lines are priced per ton. A line + * whose unit cannot be converted into its PO line's unit contributes + * ZERO and loses its link, exactly as an unmatched line already does. + * It surfaces as work for a human. It does not enter the maths at the + * wrong magnitude. + */ +import { convertQty, normaliseUnit } from "./units"; +import { validateMatchedPoItems } from "./validate-matched-po-items"; + +export interface PoItemLite { + id: string; + unit?: string; +} + +export interface DeliveryNoteItemRow { + /** Null when unmatched or unconvertible. */ + purchase_order_item: string | null; + /** In the PO line's unit. Zero whenever purchase_order_item is null. */ + qty_received: number; + /** Verbatim from the slip. */ + qty_raw: number; + /** Verbatim from the slip. Empty string when the slip showed none. */ + unit_raw: string; + unit_converted: boolean; + condition_notes: string; +} + +export interface BuildDeliveryNoteItemRowsInput { + items: readonly unknown[]; + poItems: readonly PoItemLite[]; + /** Overall OCR confidence, 0..1. Rendered into condition_notes. */ + confidence: number; +} + +export interface BuildDeliveryNoteItemRowsResult { + rows: DeliveryNoteItemRow[]; + /** True when we could not fetch the PO items but OCR supplied ids. */ + unverifiable: boolean; +} + +export function buildDeliveryNoteItemRows( + input: BuildDeliveryNoteItemRowsInput, +): BuildDeliveryNoteItemRowsResult { + const { items, poItems, confidence } = input; + + // Filter ONCE. Everything downstream indexes this list and only this + // list, so the matched ids cannot drift out of step with the rows. + const usable = items + .filter(isPlainObject) + .map((it) => ({ it, qty: asNumber(it.qty) })) + .filter((e): e is { it: Record; qty: number } => + e.qty !== null && e.qty > 0, + ); + + const { validIds, unverifiable } = validateMatchedPoItems({ + poItems, + matchedIds: usable.map((e) => asString(e.it.matched_po_item_id)), + }); + + const poItemById = new Map(poItems.map((p) => [p.id, p])); + + const rows = usable.map((entry, i) => { + const { it, qty } = entry; + const matchedId = validIds[i] ?? null; + const unitRaw = asString(it.unit) ?? ""; + + let poItemId: string | null = matchedId; + let qtyReceived = 0; + let converted = false; + let failure: string | null = null; + + if (matchedId === null) { + // Unmatched line: no PO rate, no target unit. Contributes nothing. + failure = null; + } else { + const poItem = poItemById.get(matchedId); + if (!poItem) { + // The id came from OCR and we could not cross-check it (empty + // local fetch). Keep the link for the audit trail but do not + // invent a quantity we cannot express in the right unit. + failure = "po_item_unavailable"; + } else { + const conv = convertQty(qty, normaliseUnit(unitRaw), normaliseUnit(poItem.unit)); + if (conv.ok) { + qtyReceived = conv.qty; + converted = conv.converted; + } else { + failure = conv.reason; + poItemId = null; + } + } + } + + return { + purchase_order_item: poItemId, + qty_received: qtyReceived, + qty_raw: qty, + unit_raw: unitRaw, + unit_converted: converted, + condition_notes: buildNotes(it, confidence, failure), + }; + }); + + return { rows, unverifiable }; +} + +/** + * The legacy format is preserved deliberately: scripts/migrate-dn-item-units.py + * recovers the slip's unit from the `unit: ` segment of rows + * written before unit_raw existed. + */ +function buildNotes( + it: Record, + confidence: number, + failure: string | null, +): string { + const parts = [ + `OCR ${Math.round(confidence * 100)}%`, + asString(it.description) ? `desc: ${asString(it.description)}` : "", + asString(it.unit) ? `unit: ${asString(it.unit)}` : "", + typeof it.unit_price === "number" ? `rate: ${it.unit_price.toFixed(2)}` : "", + typeof it.match_confidence === "number" + ? `match: ${Math.round(it.match_confidence * 100)}%` + : "", + failure ? `unconverted: ${failure}` : "", + ]; + return parts.filter(Boolean).join(" · "); +} + +function isPlainObject(v: unknown): v is Record { + return typeof v === "object" && v !== null && !Array.isArray(v); +} + +function asString(v: unknown): string | null { + if (typeof v !== "string") return null; + const s = v.trim(); + return s.length > 0 ? s : null; +} + +function asNumber(v: unknown): number | null { + if (typeof v === "number" && Number.isFinite(v)) return v; + if (typeof v === "string" && v.trim() !== "") { + const n = Number(v); + return Number.isFinite(n) ? n : null; + } + return null; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run src/lib/dn-item-row.test.ts` +Expected: PASS, 9 tests. + +- [ ] **Step 5: Run the whole suite** + +Run: `npm run test:run` +Expected: all existing tests still pass. + +- [ ] **Step 6: Commit** + +```bash +git add src/lib/dn-item-row.ts src/lib/dn-item-row.test.ts +git commit -m "feat(delivery-notes): build line item rows with unit conversion + +Filters once so matched PO item ids cannot drift when a zero-quantity +line is dropped, and converts the slip quantity into the PO line's unit. +An unconvertible line contributes zero rather than entering the maths at +the wrong magnitude." +``` + +--- + +## Task 5: Wire the route to the new modules + +**Files:** +- Modify: `src/app/api/delivery-notes/[id]/ocr-result/route.ts` + +- [ ] **Step 1: Fetch the PO item unit** + +The route currently fetches PO items without their unit, so there is nothing to convert into. Find this block (around line 200): + +```ts + poItems = await pb + .collection("purchase_order_items") + .getFullList({ + filter: `purchase_order='${poId}'`, + fields: "id,description,qty", + }); +``` + +Change `fields` to include `unit`: + +```ts + poItems = await pb + .collection("purchase_order_items") + .getFullList({ + filter: `purchase_order='${poId}'`, + fields: "id,description,qty,unit", + }); +``` + +- [ ] **Step 2: Replace the matched-id validation with the new builder** + +Find this block (around lines 215-228): + +```ts + const suppliedMatchedIds = items + .filter(isPlainObject) + .map((it) => + asString((it as Record).matched_po_item_id), + ); + const { validIds, unverifiable } = validateMatchedPoItems({ + poItems, + matchedIds: suppliedMatchedIds, + }); +``` + +Replace it with: + +```ts + // Rows are built before the tier decision because `unverifiable` + // feeds it. The builder is pure — see src/lib/dn-item-row.ts for why + // the filtering happens in exactly one place. + const { rows: itemRows, unverifiable } = buildDeliveryNoteItemRows({ + items, + poItems, + confidence, + }); +``` + +- [ ] **Step 3: Update the imports** + +At the top of the file, remove the now-unused `validateMatchedPoItems` import and add the new one: + +```ts +import { buildDeliveryNoteItemRows } from "@/lib/dn-item-row"; +``` + +Delete this line: + +```ts +import { validateMatchedPoItems } from "@/lib/validate-matched-po-items"; +``` + +(`validate-matched-po-items.ts` itself stays — `dn-item-row.ts` uses it.) + +- [ ] **Step 4: Replace the creation loop** + +Find the loop that begins (around line 318): + +```ts + const createdItems: { id: string; po_item_id: string; qty: number }[] = []; + let itemIndex = 0; + for (const item of items) { +``` + +Replace everything from that line down to the closing brace of the `for` loop (ending just before the `// If tier is auto` comment, around line 392) with: + +```ts + const createdItems: { id: string; po_item_id: string; qty: number }[] = []; + for (const row of itemRows) { + try { + const created = await pb.collection("delivery_note_items").create({ + delivery_note: dnId, + purchase_order_item: row.purchase_order_item, + qty_received: row.qty_received, + qty_raw: row.qty_raw, + unit_raw: row.unit_raw, + unit_converted: row.unit_converted, + condition_notes: row.condition_notes, + }); + createdItems.push({ + id: created.id, + po_item_id: row.purchase_order_item ?? "", + qty: row.qty_received, + }); + } catch (err) { + // FK constraint: PB rejected the link because the id no longer + // exists (race between our local fetch and the create — n8n had a + // stale id). Retry with the link dropped so the row still lands + // and the user can complete it manually. The PocketBase JS SDK + // surfaces this as a ClientResponseError with .status === 400. + const status = (err as { status?: number })?.status; + const message = err instanceof Error ? err.message : String(err); + const isFkError = Boolean(row.purchase_order_item) && status === 400; + if (isFkError) { + console.warn( + `[dn-ocr-result] delivery_note_item create failed for matched_po_item_id=${row.purchase_order_item} (id no longer exists in PB) — retrying without link:`, + message, + ); + try { + const created = await pb.collection("delivery_note_items").create({ + delivery_note: dnId, + purchase_order_item: null, + // No PO line means no target unit, so no trustworthy + // converted quantity — same rule as an unmatched line. + qty_received: 0, + qty_raw: row.qty_raw, + unit_raw: row.unit_raw, + unit_converted: false, + condition_notes: `${row.condition_notes} · link dropped: orphan id`, + }); + createdItems.push({ id: created.id, po_item_id: "", qty: 0 }); + } catch (err2) { + console.warn( + "[dn-ocr-result] delivery_note_item create (no-link retry) failed:", + err2, + ); + } + } else { + console.warn("[dn-ocr-result] delivery_note_item create failed:", err); + } + } + } +``` + +- [ ] **Step 5: Remove the now-dead helper** + +`isPlainObject` is no longer referenced in the route (the builder owns it). Delete its definition near the bottom of the file: + +```ts +function isPlainObject(v: unknown): v is Record { + return typeof v === "object" && v !== null && !Array.isArray(v); +} +``` + +Leave `asString`, `asNumber`, `clamp01` and `escapePb` — they are still used for the header fields. + +- [ ] **Step 6: Typecheck** + +Run: `npm run typecheck` +Expected: no errors. If it reports `'asNumber' is declared but its value is never read`, confirm whether the header parsing still calls it (it does, for `body.confidence`) before deleting anything. + +- [ ] **Step 7: Run the whole suite** + +Run: `npm run test:run` +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add src/app/api/delivery-notes/\[id\]/ocr-result/route.ts +git commit -m "fix(delivery-notes): convert received quantity into the PO line's unit + +Fetches the PO item unit and delegates row construction to +buildDeliveryNoteItemRows, which also removes the matched-id off-by-one +that linked lines to the wrong PO item after a zero-quantity line." +``` + +--- + +## Task 6: Show both units + +**Files:** +- Modify: `src/lib/utils.ts` +- Test: `src/lib/utils.test.ts` (create if absent) +- Modify: `src/app/dashboard/financial/purchase-orders/[poId]/po-detail-view.tsx` + +- [ ] **Step 1: Write the failing test** + +Create or append to `src/lib/utils.test.ts`: + +```ts +import { describe, it, expect } from "vitest"; +import { formatQtyDual } from "./utils"; + +describe("formatQtyDual", () => { + it("shows both figures when a conversion happened", () => { + expect(formatQtyDual(29.96, "ton", 29960, "kg")).toBe("29.96 ton · 29 960 kg"); + }); + + it("shows one figure when the units match", () => { + expect(formatQtyDual(5, "ton", 5, "ton")).toBe("5 ton"); + }); + + it("shows one figure when the slip recorded no unit", () => { + expect(formatQtyDual(5, "ton", 5, "")).toBe("5 ton"); + }); + + it("omits the unit entirely when the PO line has none", () => { + expect(formatQtyDual(5, "", 5, "")).toBe("5"); + }); + + it("falls back to the raw figure when nothing converted", () => { + expect(formatQtyDual(0, "ton", 12, "m3")).toBe("0 ton · 12 m3"); + }); +}); +``` + +Note: `formatNumber` uses the `en-ZA` locale, whose thousands separator is a non-breaking space. Write the expected strings using a literal non-breaking space (U+00A0) in `29 960`, or build them with `formatNumber` in the test. The second is less brittle: + +```ts +import { formatNumber } from "./utils"; +// e.g. expect(formatQtyDual(29.96, "ton", 29960, "kg")) +// .toBe(`${formatNumber(29.96)} ton · ${formatNumber(29960)} kg`); +``` + +Use that form for every assertion containing a number above 999. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run src/lib/utils.test.ts` +Expected: FAIL — no export named `formatQtyDual`. + +- [ ] **Step 3: Write minimal implementation** + +Append to `src/lib/utils.ts`: + +```ts +/** + * A received quantity, shown in the purchase order line's unit with the + * slip's own figure beside it. + * + * Site staff read the slip in kg; the PO is priced per ton. Showing only + * one of the two means somebody eventually reconciles the wrong number + * against the wrong document. + */ +export function formatQtyDual( + qtyReceived: number, + unit: string, + qtyRaw: number, + unitRaw: string, +): string { + const primary = unit ? `${formatNumber(qtyReceived)} ${unit}` : formatNumber(qtyReceived); + const sameUnit = !unitRaw || unitRaw.trim().toLowerCase() === unit.trim().toLowerCase(); + if (sameUnit) return primary; + return `${primary} · ${formatNumber(qtyRaw)} ${unitRaw}`; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run src/lib/utils.test.ts` +Expected: PASS, 5 tests. + +- [ ] **Step 5: Fetch the new fields** + +In `src/app/dashboard/financial/purchase-orders/[poId]/page.tsx`, the delivery lines are fetched around line 49 without the new columns: + +```ts + pb.collection("delivery_note_items").getFullList({ + fields: "id,delivery_note,purchase_order_item,qty_received,condition_notes", + }), +``` + +Change the `fields` string to: + +```ts + fields: "id,delivery_note,purchase_order_item,qty_received,qty_raw,unit_raw,condition_notes", +``` + +- [ ] **Step 6: Carry them into the view model** + +In the same file, the `receivedItems` mapping (around line 112) reads: + +```ts + receivedItems: (receivedByDn.get(d.id) ?? []).map((r) => ({ + id: r.id, + purchaseOrderItemId: r.purchase_order_item, + qtyReceived: r.qty_received, + conditionNotes: r.condition_notes, + })), +``` + +Replace it with: + +```ts + receivedItems: (receivedByDn.get(d.id) ?? []).map((r) => ({ + id: r.id, + purchaseOrderItemId: r.purchase_order_item, + qtyReceived: r.qty_received, + // Rows written before the unit migration have no qty_raw; falling + // back to qty_received makes formatQtyDual render a single figure + // rather than "29.96 ton · 0 ". + qtyRaw: r.qty_raw ?? r.qty_received, + unitRaw: r.unit_raw ?? "", + conditionNotes: r.condition_notes, + })), +``` + +- [ ] **Step 7: Widen the view's type** + +In `src/app/dashboard/financial/purchase-orders/[poId]/po-detail-view.tsx`, the `PoDetailDn` interface declares `receivedItems` around line 62. Replace that member with: + +```ts + receivedItems: { + id: string; + purchaseOrderItemId: string; + qtyReceived: number; + qtyRaw: number; + unitRaw: string; + conditionNotes: string; + }[]; +``` + +- [ ] **Step 8: Render both units** + +In the same file, the delivery line quantity renders at line 486: + +```tsx + + {formatNumber(r.qtyReceived)} {it?.unit ?? ""} + +``` + +Replace it with: + +```tsx + + {formatQtyDual(r.qtyReceived, it?.unit ?? "", r.qtyRaw, r.unitRaw)} + +``` + +and add `formatQtyDual` to the existing `@/lib/utils` import at the top of the file (it already imports `formatNumber` from there). + +Leave line 312 (`{formatNumber(it.qtyReceived)} {it.unit}`) alone. That is the **aggregate** per PO item, which is already summed in PO units after the migration — there is no single slip unit to show beside it. + +- [ ] **Step 9: Verify in the browser** + +Start the dev server and open the PO detail page for `MEGCPO9165`: + +```bash +npm run dev +``` + +Navigate to `/dashboard/financial/purchase-orders/y35epf1bpx9vai8`. + +Expected after Task 7's migration runs: the G6 Sub Base delivery lines read `29.96 ton · 29 960 kg`, and the line's delivered total is roughly `64.78 ton` against `4 273.89 ton` ordered. + +Before Task 7 runs, the figures will still be the old unconverted ones — that is expected. Do the visual check after Task 7. + +- [ ] **Step 10: Commit** + +```bash +git add src/lib/utils.ts src/lib/utils.test.ts \ + src/app/dashboard/financial/purchase-orders/\[poId\]/po-detail-view.tsx \ + src/app/dashboard/financial/purchase-orders/\[poId\]/page.tsx +git commit -m "feat(purchasing): show received quantity in both PO and slip units" +``` + +--- + +## Task 7: Backfill the existing rows + +57 `delivery_note_items` rows were written before `qty_raw` existed. Their slip unit survives only inside `condition_notes`. + +**Files:** +- Create: `scripts/migrate-dn-item-units.py` + +- [ ] **Step 1: Write the migration** + +Create `scripts/migrate-dn-item-units.py`: + +```python +#!/usr/bin/env python3 +""" +Backfills qty_raw / unit_raw / unit_converted on delivery_note_items and +rewrites qty_received into the purchase order line's unit. + +Before this ran, qty_received held whatever the slip said — kg for +weighbridge tickets — while the PO line it points at is priced per ton. +Every fulfilment percentage and received value computed from those rows +was 1000x too large. + +The slip's unit survives only in condition_notes, which the OCR callback +wrote as: + + OCR 95% · desc: G6 Sub Base · unit: kg · match: 100% + +Rows with no `unit:` segment are left ALONE and reported. Guessing their +unit is how the original bug happened. + +Idempotent — rows that already have a non-empty unit_raw are skipped. + +Usage: + python3 scripts/migrate-dn-item-units.py # dry run + python3 scripts/migrate-dn-item-units.py --apply # writes +""" +import re +import sys + +from pb_env import PB, http, superuser_token + +UNIT_RE = re.compile(r"unit:\s*([^·]+)") + +ALIASES = { + "kg": ["kg", "kgs", "kilo", "kilos", "kilogram", "kilograms"], + "ton": ["t", "ton", "tons", "tonne", "tonnes", "mt", "metric ton"], + "m3": ["m3", "m³", "cube", "cubes", "cubic metre", "cubic meter"], + "m2": ["m2", "m²", "square metre", "square meter"], + "m": ["m", "metre", "meter", "metres", "meters", "lm", "linear metre"], + "ea": ["ea", "each", "unit", "units", "no", "nr", "item", "items"], + "bag": ["bag", "bags", "pocket", "pockets"], + "l": ["l", "litre", "litres", "liter", "liters"], + "load": ["load", "loads", "trip", "trips"], +} +LOOKUP = {a: c for c, al in ALIASES.items() for a in al} + +FACTORS = { + ("kg", "ton"): 1 / 1000, + ("ton", "kg"): 1000, + ("l", "m3"): 1 / 1000, + ("m3", "l"): 1000, +} + + +def normalise_unit(raw): + """Mirror of normaliseUnit in src/lib/units.ts. Keep the two in step.""" + if not isinstance(raw, str): + return None + cleaned = re.sub(r"\s+", " ", re.sub(r"[^a-z0-9²³ ]", " ", raw.lower())).strip() + return LOOKUP.get(cleaned) if cleaned else None + + +def convert(qty, from_unit, to_unit): + """Returns (qty, converted, failure_reason).""" + f, t = normalise_unit(from_unit), normalise_unit(to_unit) + if f is None or t is None: + return None, False, "unknown_unit" + if f == t: + return qty, False, None + factor = FACTORS.get((f, t)) + if factor is None: + return None, False, "incompatible_units" + return round(qty * factor, 4), True, None + + +def get_all(token, collection, fields=None, filt=None): + out, page = [], 1 + while True: + q = f"?page={page}&perPage=200" + if fields: + q += f"&fields={fields}" + if filt: + q += f"&filter={filt}" + code, resp = http("GET", f"/api/collections/{collection}/records{q}", token=token) + if code != 200: + sys.exit(f"GET {collection} failed: {code} {resp}") + out.extend(resp["items"]) + if page >= resp["totalPages"]: + return out + page += 1 + + +def main(apply: bool) -> None: + token = superuser_token() + print(f"\n→ delivery_note_items on {PB} ({'APPLY' if apply else 'DRY RUN'})") + + po_items = { + p["id"]: p + for p in get_all(token, "purchase_order_items", "id,unit,description") + } + rows = get_all( + token, + "delivery_note_items", + "id,delivery_note,purchase_order_item,qty_received,qty_raw,unit_raw,condition_notes", + ) + print(f" {len(rows)} rows, {len(po_items)} purchase order items\n") + + changed = skipped = no_unit = unconvertible = 0 + print(f" {'id':17s} {'from':>14s} {'to':>14s} note") + print(f" {'-' * 17} {'-' * 14} {'-' * 14} {'-' * 30}") + + for r in rows: + rid = r["id"] + if (r.get("unit_raw") or "").strip(): + skipped += 1 + continue + + qty = r.get("qty_received") or 0 + notes = r.get("condition_notes") or "" + m = UNIT_RE.search(notes) + if not m: + no_unit += 1 + print(f" {rid:17s} {qty:>14} {'—':>14} no unit: segment — LEFT ALONE") + continue + unit_raw = m.group(1).strip() + + poi = po_items.get(r.get("purchase_order_item") or "") + if not poi: + # Unlinked line: record what the slip said, leave qty_received. + patch = {"qty_raw": qty, "unit_raw": unit_raw, "unit_converted": False} + print(f" {rid:17s} {qty:>10} {unit_raw:<3} {'unlinked':>14} qty_raw only") + if apply: + code, resp = http("PATCH", f"/api/collections/delivery_note_items/records/{rid}", patch, token) + if code != 200: + sys.exit(f"PATCH {rid} failed: {code} {resp}") + changed += 1 + continue + + po_unit = poi.get("unit") or "" + new_qty, converted, failure = convert(qty, unit_raw, po_unit) + + if failure: + unconvertible += 1 + patch = { + "qty_raw": qty, + "unit_raw": unit_raw, + "unit_converted": False, + "qty_received": 0, + "purchase_order_item": None, + "condition_notes": f"{notes} · unconverted: {failure}", + } + print(f" {rid:17s} {qty:>10} {unit_raw:<3} {'0':>10} {po_unit:<3} {failure} — link dropped") + else: + patch = { + "qty_raw": qty, + "unit_raw": unit_raw, + "unit_converted": converted, + "qty_received": new_qty, + } + print(f" {rid:17s} {qty:>10} {unit_raw:<3} {new_qty:>10} {po_unit:<3} {poi.get('description', '')[:26]}") + + if apply: + code, resp = http("PATCH", f"/api/collections/delivery_note_items/records/{rid}", patch, token) + if code != 200: + sys.exit(f"PATCH {rid} failed: {code} {resp}") + changed += 1 + + print( + f"\n changed: {changed} already done: {skipped} " + f"no unit recorded: {no_unit} unconvertible: {unconvertible}" + ) + if not apply: + print("\n DRY RUN — nothing written. Re-run with --apply.") + + +if __name__ == "__main__": + main("--apply" in sys.argv) + print("\nDone.") +``` + +- [ ] **Step 2: Dry run** + +```bash +python3 scripts/migrate-dn-item-units.py +``` + +Expected: a per-row table. The two G6 Sub Base lines on PO `MEGCPO9165` should show `29960 kg → 29.96 ton` and `34820 kg → 34.82 ton`. The footer should report `DRY RUN — nothing written`. + +**Read the table before continuing.** If more than a handful of rows report `no unit: segment`, stop and investigate rather than applying — it means `condition_notes` was written in some other format and the parse needs widening. + +- [ ] **Step 3: Back up the records** + +`scripts/pb-backup-collection.py` dumps a collection's **schema only** — it +does not save records, so it is not a safety net for this migration. Dump the +rows themselves: + +```bash +python3 - <<'EOF' +import json, sys +from datetime import datetime +sys.path.insert(0, "scripts") +from pb_env import http, superuser_token + +t = superuser_token() +items, page = [], 1 +while True: + code, resp = http( + "GET", + f"/api/collections/delivery_note_items/records?page={page}&perPage=200", + token=t, + ) + if code != 200: + sys.exit(f"backup failed: {code} {resp}") + items.extend(resp["items"]) + if page >= resp["totalPages"]: + break + page += 1 + +stamp = datetime.now().strftime("%Y%m%d-%H%M%S") +path = f"/tmp/delivery_note_items-records-{stamp}.json" +with open(path, "w") as f: + json.dump(items, f, indent=2) +print(f"backed up {len(items)} records to {path}") +EOF +``` + +Expected: `backed up 57 records to /tmp/delivery_note_items-records-.json`. +Note the path — it is how you undo this if the figures come out wrong. + +- [ ] **Step 4: Apply** + +```bash +python3 scripts/migrate-dn-item-units.py --apply +``` + +Expected: the same table as the dry run, footer reports `changed: N`, no `DRY RUN` line. + +- [ ] **Step 5: Verify idempotency** + +```bash +python3 scripts/migrate-dn-item-units.py +``` + +Expected: `changed: 0` and `already done: N` for every row the apply pass touched. + +- [ ] **Step 6: Verify the corrected figures** + +```bash +python3 - <<'EOF' +import sys; sys.path.insert(0, "scripts") +from pb_env import http, superuser_token +t = superuser_token() +_, r = http("GET", "/api/collections/delivery_note_items/records" + "?perPage=200&filter=purchase_order_item='bxqbo8rs04pjqwc'" + "&fields=qty_received,qty_raw,unit_raw", token=t) +tot = sum(i["qty_received"] for i in r["items"]) +print(f"lines: {len(r['items'])} delivered: {tot} ton (ordered 4273.89 ton)") +print(f"value: R{tot * 168.50:,.2f}") +EOF +``` + +Expected: delivered ≈ `64.78 ton`, value ≈ `R10,915.43`. **Not** 64 780 and **not** R10.9 m. + +- [ ] **Step 7: Commit** + +```bash +git add scripts/migrate-dn-item-units.py +git commit -m "chore(delivery-notes): backfill line item units and correct qty_received + +Rewrites qty_received into the purchase order line's unit, recovering the +slip's own unit from the condition_notes format the OCR callback wrote. +Rows with no recorded unit are left untouched and reported." +``` + +--- + +## Task 8: Stop the orphan production and recover the lost notes + +These are operational steps against n8n and PocketBase, not code. Do them in order. + +- [ ] **Step 1: Disconnect the split branch in n8n** + +Open workflow `Delivery Note OCR` (id `wVbLw4paISkbrIpE`) at https://wfdash.n8n.apphub.co.za. + +Delete the connection from `Is first delivery note?` **false** output to `Create split DN record`. Leave the false output unconnected. Save and keep the workflow active. + +Effect: a multi-note document now lands as a single note in review, which is visible and recoverable. Today it mints records with no data in a status the dashboard cannot display. + +- [ ] **Step 2: Verify no new orphans appear** + +```bash +python3 - <<'EOF' +import sys; sys.path.insert(0, "scripts") +from pb_env import http, superuser_token +t = superuser_token() +_, r = http("GET", "/api/collections/delivery_notes/records" + "?perPage=200&filter=received_by='n8n-ocr-split'" + "&fields=id,delivery_note_number,processing_status", token=t) +for i in r["items"]: + print(i["id"], i.get("delivery_note_number"), i.get("processing_status")) +print("total:", r["totalItems"]) +EOF +``` + +Expected: 3 rows (the existing ones). Re-run after a day; the count must not grow. + +- [ ] **Step 3: Surface the two lost notes** + +```bash +python3 - <<'EOF' +import sys; sys.path.insert(0, "scripts") +from pb_env import http, superuser_token +t = superuser_token() +for dn in ("jn154emn5c1h2ho", "2dkwx357mnexrk8"): + code, resp = http("PATCH", f"/api/collections/delivery_notes/records/{dn}", + {"processing_status": "needs-review"}, t) + print(dn, code, resp.get("delivery_note_number") if code == 200 else resp) +EOF +``` + +Expected: both return `200` with `DQ0010239` and `DQ0010217`. + +- [ ] **Step 4: Capture their line items by hand** + +Both are G6 Sub Base against PO item `bxqbo8rs04pjqwc` (PO `MEGCPO9165`, priced per ton at R168.50). Quantities read directly off pages 3 and 4 of the source scan: + +| Delivery note | Record | Slip quantity | In PO units | +| --- | --- | --- | --- | +| DQ0010239 | `jn154emn5c1h2ho` | 34 200 kg | 34.20 ton | +| DQ0010217 | `2dkwx357mnexrk8` | 34 520 kg | 34.52 ton | + +```bash +python3 - <<'EOF' +import sys; sys.path.insert(0, "scripts") +from pb_env import http, superuser_token +t = superuser_token() +ROWS = [ + ("jn154emn5c1h2ho", 34200, 34.20), + ("2dkwx357mnexrk8", 34520, 34.52), +] +for dn, kg, ton in ROWS: + code, resp = http("POST", "/api/collections/delivery_note_items/records", { + "delivery_note": dn, + "purchase_order_item": "bxqbo8rs04pjqwc", + "qty_received": ton, + "qty_raw": kg, + "unit_raw": "kg", + "unit_converted": True, + "condition_notes": "manual capture · desc: G6 Sub Base · unit: kg · recovered from lost n8n split", + }, t) + print(dn, code, resp.get("id") if code == 200 else resp) +EOF +``` + +Expected: both return `200` with a new record id. + +- [ ] **Step 5: Confirm the PO line now reconciles** + +Re-run the verification from Task 7 Step 5. + +Expected: delivered ≈ `133.50 ton` across four notes (29.96 + 34.82 + 34.20 + 34.52), value ≈ `R22,494.75`. That is the full load from `scan_20260915_c5p7fkmzgv.pdf`. + +- [ ] **Step 6: Rotate the ingest secret** + +The secret is a literal string in four nodes of the n8n workflow, so it is in every export of it. Generate a new one, set `N8N_INGEST_SECRET` in Coolify and in `.env.local`, and update the n8n nodes to read it from a **credential** rather than a pasted value. + +This is a production credential change — do it yourself in the Coolify and n8n UIs. + +- [ ] **Step 7: Record the operational steps** + +Append a short section to `docs/n8n/delivery-note-ocr.md` noting that the split branch is disconnected pending the inbox pipeline, with a link to the spec. Commit: + +```bash +git add docs/n8n/delivery-note-ocr.md +git commit -m "docs(n8n): note the disconnected split branch pending the inbox pipeline" +``` + +--- + +## Done when + +- [ ] `npm run test:run` passes. +- [ ] `npm run typecheck` passes. +- [ ] PO `MEGCPO9165` line `bxqbo8rs04pjqwc` shows roughly **133.50 ton delivered against 4 273.89 ton ordered** — about 3 % fulfilled, not 1 515 %. +- [ ] The four notes from `scan_20260915_c5p7fkmzgv.pdf` carry a combined received value near **R22 495**, not R10.9 m. +- [ ] Delivery quantities render as `29.96 ton · 29 960 kg` on the PO detail page. +- [ ] `python3 scripts/migrate-dn-item-units.py` reports `changed: 0` on a second run. +- [ ] No new `received_by='n8n-ocr-split'` rows have appeared since Task 8 Step 1. + +--- + +## Not in this plan + +Phase 2 (the inbox batch pipeline) and phase 3 (the Kanban board) are specified in §4 and §5 of the spec and get their own plan. Phase 4 (purchase order cleanup — the dead state machine, `#` prefix normalisation, the `#MEGCP09109` and `IN132201` rows) is not yet specified. diff --git a/docs/superpowers/specs/2026-09-17-delivery-note-inbox-design.md b/docs/superpowers/specs/2026-09-17-delivery-note-inbox-design.md index 1bcf680..da59804 100644 --- a/docs/superpowers/specs/2026-09-17-delivery-note-inbox-design.md +++ b/docs/superpowers/specs/2026-09-17-delivery-note-inbox-design.md @@ -228,7 +228,10 @@ then derive `suppliedMatchedIds` from `usableItems` and index `validIds` by the loop's own position over `usableItems`. Add a regression test covering a three-item payload whose middle item has `qty: 0`. -### 3.5 Migration — `scripts/migrate-dn-item-units.ts` +### 3.5 Migration — `scripts/migrate-dn-item-units.py` + +Python, matching every other data migration in `scripts/` and reusing +`scripts/pb_env.py` for credentials. Backfills the 57 existing `delivery_note_items` rows.