From 49b6547c0cfeac90a8cf738540ea957cba2d1365 Mon Sep 17 00:00:00 2001 From: DEls Date: Thu, 17 Sep 2026 15:23:16 +0200 Subject: [PATCH] docs(n8n): rewrite for the two-node tick workflow --- docs/n8n/delivery-note-ocr.md | 486 +++++++--------------------------- 1 file changed, 99 insertions(+), 387 deletions(-) diff --git a/docs/n8n/delivery-note-ocr.md b/docs/n8n/delivery-note-ocr.md index 0490dea..dec6dfd 100644 --- a/docs/n8n/delivery-note-ocr.md +++ b/docs/n8n/delivery-note-ocr.md @@ -1,395 +1,107 @@ -# n8n Workflow — Delivery Note OCR +# n8n Workflow — Delivery Note Tick -This is the handoff doc for the n8n workflow that processes delivery -note slips uploaded from the MEGA Infrastructure dashboard. The Next.js -side is implemented by Claude; **you build this workflow in n8n using -your MCP access**. +This is the handoff doc for the n8n workflow that drives the +delivery note inbox pipeline. The pipeline itself (extract → segment +→ split → checks → promote) lives in `src/lib/dn-pipeline/` and is +run entirely inside Next.js. n8n's only job is to call the tick +endpoint every minute. ## What this workflow does -1. **Watches** the `delivery_notes` PocketBase collection for new rows - where `processing_status = "pending"`. Either poll every 30–60 s - or wire a PocketBase webhook to trigger on row insert. -2. For each pending DN, downloads `slip_image` from PocketBase. -3. Runs OCR on the slip (image → vision model, PDF → text extract first - then vision on rendered pages). -4. Sends the OCR text through an LLM with a structured-output prompt to - extract: - - PO number - - Delivery note number - - Date received - - Received-by name (signature) - - Line items: description, qty, unit, unit_price - - Overall confidence (0..1) the AI has in the extraction -5. **POSTs the structured result back to Next.js** at - `POST /api/delivery-notes/[id]/ocr-result` (where `[id]` is the - PocketBase record id of the delivery note). +Two nodes. That's it. -The Next.js route handles: -- Updating the DN's `processing_status` based on confidence -- Creating `received_items` rows -- Marking `is_verified = true` for high-confidence matches -- Idempotency (safe to replay) +1. **Schedule Trigger** — fires every 1 minute. +2. **HTTP Request** — `POST https://mega.dash.apphub.co.za/api/dn-batches/tick` + with header `x-ingest-secret: `. + +The Next.js route (`src/app/api/dn-batches/tick/route.ts`) picks up +the call, walks every `dn_batches` row whose `status` is one of +`inbox` / `analysing` / `needs_attention`, and advances each one +through the pipeline until it reaches `processed` (or stays in +`needs_attention` until a human acts). All OCR, segmentation, +checks, and PocketBase writes happen in Next.js — n8n does not see +a slip. ## When this workflow runs -Poll PocketBase on a schedule (every 30–60 s) for rows where: - -``` -processing_status = "pending" -``` - -After processing, the route will set `processing_status` to either: - -| Value | Meaning | -|---|---| -| `auto-verified` | Confidence ≥ 0.8 — fully verified, `received_items` created, `is_verified = true` | -| `needs-review` | Confidence 0.5–0.8 — `received_items` created as draft, `is_verified = false`, user reviews | -| `needs-review` | Confidence < 0.5 — no `received_items` created, user manually captures line items | -| `failed` | OCR pipeline errored; row stays pending until retried | - -Use this status in your polling query to find work. - -## Downloading the slip - -``` -GET /api/files/{collectionId}/{recordId}/{fieldName}/{filename} -``` - -Or use the PocketBase SDK to fetch the file directly. The `slip_image` -field on `delivery_notes` records is the file. - -## What to extract (LLM structured output) - -| Field | Type | Notes | -|---|---|---| -| `po_number` | string \| null | Match against `purchase_orders.purchase_order_number` | -| `delivery_note_number` | string \| null | Match against `delivery_note_number` on existing DNs | -| `date_received` | `YYYY-MM-DD` \| null | From the slip header. Overrides the upload-time date. | -| `received_by_name` | string \| null | Signature/name on the slip | -| `confidence` | number 0..1 (or 0..100) | Overall confidence. The route auto-clamps. | -| `raw_ocr` | string \| null | Full OCR text — saved on `raw_ocr_data.raw_ocr` for audit (truncated to 50 KB) | -| `items` | array (see below) | One entry per line item on the slip | - -### `items[]` shape - -| Field | Type | Notes | -|---|---|---| -| `description` | string | Free text from the slip | -| `qty` | number | Quantity received | -| `unit` | string \| null | bag, m², ea, etc. | -| `unit_price` | number \| null | From the slip (optional, used in audit) | -| `matched_po_item_id` | PocketBase id \| null | The id of the matching `purchase_order_items` row, if you can match it | -| `match_confidence` | number 0..1 | Per-line confidence. Saved in `condition_notes`. | - -If you can't match a line to a PO item, leave `matched_po_item_id = null`. -The user will fix it manually in the exception inbox. - -## Confidence score (how to compute it) - -The route uses three thresholds: **0.8** (auto), **0.5** (draft), below -that (manual). Compute it however you want, but here's a reasonable -starting formula: - -``` -score = 0 -if po_number matches an existing PO: score += 0.30 -if delivery_note_number parseable: score += 0.10 -if supplier matches PO supplier: score += 0.10 -if line items fuzzy-match PO items: score += 0.30 × (% matched) -if line totals add up to grand total: score += 0.20 -``` - -Tune the weights after running for a week or two. The thresholds above -assume a default of `gpt-4o-mini` or similar mid-tier vision model. - -## Callback endpoint - -``` -POST /api/delivery-notes/{id}/ocr-result -Headers: - Content-Type: application/json - x-ingest-secret: ${N8N_INGEST_SECRET} ← same env var as the PO webhook -Body: (see shape above) -``` - -`{id}` is the PocketBase id of the delivery note (15-char alphanumeric). - -### Response codes - -| Code | Meaning | What to do | -|---|---|---| -| `200` | Accepted. `received_items` rows created (or skipped if tier=manual). | Mark `processing_status="ocr-complete"` in your n8n node so you don't reprocess | -| `401` | Bad / missing `x-ingest-secret` | Log "secret mismatch — check N8N_INGEST_SECRET env". Do **not** retry. | -| `404` | Delivery note id doesn't exist | Log "dn_not_found for id={id}" — the record may have been deleted. | -| `400` | Body validation failed | Log the body; do not retry; user needs to investigate. | -| `500` | PocketBase update failed | Retry once with backoff. After 2 failures, mark `processing_status="failed"` so the user sees it in the exception inbox. | - -### Response body (200) - -```json -{ - "ok": true, - "tier": "auto" | "draft" | "manual", - "receivedItemsCreated": 3, - "receivedItems": [ - { "id": "...", "po_item_id": "...", "qty": 25 } - ], - "durationMs": 1234 -} -``` - -## Suggested n8n flow (high-level) - -1. **Schedule Trigger** every 30 s. -2. **PocketBase HTTP Request**: list delivery_notes where - `processing_status = "pending"` (filter via PB REST API). -3. For each row: - 1. Download `slip_image` - 2. OCR (vision model) - 3. LLM structured extraction (with the prompt below) - 4. Compute confidence - 5. **HTTP Request** → `POST /api/delivery-notes/{id}/ocr-result` - with the body and `x-ingest-secret` - 6. On 200: log `receivedItemsCreated`, move on - 7. On 5xx: retry once after 5 s; on second failure, update the DN's - `processing_status="failed"` directly via PocketBase API so the - exception inbox surfaces it - -## Recommended LLM prompt (skeleton) - -``` -You are extracting structured data from a delivery note slip image. - -Return ONLY valid JSON matching this schema: -{ - "po_number": string | null, - "delivery_note_number": string | null, - "date_received": "YYYY-MM-DD" | null, - "received_by_name": string | null, - "confidence": number, // 0..1 — your overall certainty - "items": [ - { - "description": string, - "qty": number, - "unit": string | null, - "unit_price": number | null - } - ] -} - -Rules: -- confidence is your honest assessment: 0.9+ only if everything is - clearly legible, 0.5–0.8 if some fields are guesses, < 0.5 if - the slip is unreadable -- dates must be ISO format or null -- quantities must be numbers (not strings) -- if the slip has no readable items, return an empty items array -``` - -If you have a PO list to match against (recommended), inject it as -context: - -``` -Reference purchase orders: -- PO-1234 | Cement 50kg bag × 50 @ R110 | Acme Supplies -- PO-1235 | ... - -For each line item, return `matched_po_item_id` if you can identify -the matching PO line; otherwise null. -``` - -## Where things live in PocketBase - -- `delivery_notes.processing_status` — `"pending"` (your queue), or `"auto-verified"` / `"needs-review"` / `"failed"` after the webhook fires -- `delivery_notes.ocr_confidence` — 0..100 integer -- `delivery_notes.ocr_completed_at` — ISO timestamp of when OCR finished -- `delivery_notes.raw_ocr_data` — JSON, includes the full payload you sent (so audits can replay) -- `received_items` — auto-created rows from successful (auto + draft) tiers - -## Environment - -- `N8N_INGEST_SECRET` — shared secret header (same value as `.env.local`) -- `MEGA_PB_URL` — PocketBase URL (already in `.env.local`) -- `MEGA_PB_ADMIN_TOKEN` — admin token for the workflow (PB service token, not user) - ---- - -## Workflow code (n8n Workflow SDK) - -The reference workflow is in [`docs/n8n/delivery-note-ocr-workflow.ts`](./delivery-note-ocr-workflow.ts). It's written against `@n8n/workflow-sdk`. To import into your n8n instance: - -```bash -# In your n8n project (or a separate dir with the SDK installed) -npm install @n8n/workflow-sdk -npx n8n-workflow-sdk import ./delivery-note-ocr-workflow.ts \ - --base-url "$MEGA_N8N_URL" \ - --api-key "$MEGA_N8N_API_KEY" -``` - -(Or whatever import subcommand the SDK exposes — check `npx @n8n/workflow-sdk --help` for current usage.) - -If the SDK import command isn't available in your environment, paste the workflow JSON into n8n manually via *Workflows → Import from File*. The TypeScript source documents every node + its config so the JSON shape is one-to-one. - -### Workflow shape (one node per line) - -``` -[Schedule Trigger — every 30 s] - │ - ▼ -[List pending DNs (PB HTTP)] ── 0..N items - │ - ▼ -[SplitInBatches — batchSize 1] - │ - ▼ -[Download slip image] ← loop body - │ - ▼ -[Vision LLM extract (GPT-4o-mini)] - │ - ▼ -[Structured output parser] - │ - ▼ -[POST /api/delivery-notes/{id}/ocr-result] - │ - ▼ -[Mark as ocr-complete in PB] - │ - ▼ -[nextBatch → loop back] -``` - -### Credentials to configure after import - -| Node | Credential name | What to put | -|---|---|---| -| List pending DNs / Download slip | PocketBase Admin Token | `POCKETBASE_ADMIN_TOKEN` from `.env.local` | -| OCR extract | OpenAI Vision API | your OpenAI key (or swap to Claude/Gemini — node config is open to whatever the LangChain node supports) | -| POST result to app | (none — uses `x-ingest-secret` header from `N8N_INGEST_SECRET`) | — | -| Mark as ocr-complete | PocketBase Admin Token | same as above | - -### Workflow environment variables - -| Variable | Value | -|---|---| -| `MEGA_PB_URL` | `https://pb.mega.infra.apphub.co.za` | -| `MEGA_NEXT_URL` | The public URL of your Next.js app (the one the app sees as `request.url`) | -| `N8N_INGEST_SECRET` | Same value as in `.env.local` | - -### Customising for Claude / Gemini / etc. - -Swap the `@n8n/n8n-nodes-langchain.openAi` node in `ocrExtract` for the equivalent provider (e.g. `@n8n/n8n-nodes-langchain.anthropic` with model `claude-3-5-sonnet-latest`, or Google's Gemini vision model). Keep the structured-output prompt the same — vision models all accept JSON schema instructions. - ---- - -## Importable JSON workflow - -The workflow is also available as a ready-to-import JSON file at -[`docs/n8n/delivery-note-ocr-workflow.json`](./delivery-note-ocr-workflow.json). It has been validated as parseable JSON. - -### How to import - -1. Open your n8n instance (https://wfdash.n8n.apphub.co.za) -2. **Workflows → ⋯ menu → Import from File** (or paste JSON via *Import from Clipboard*) -3. Select `delivery-note-ocr-workflow.json` -4. n8n will create the workflow with placeholder credential IDs (`REPLACE_PB_TOKEN_ID`, `REPLACE_OPENAI_KEY_ID`) -5. Replace those placeholders with real credentials: - - Create credential **PocketBase Admin Token** (header auth, value = your PB superuser token), update the 4 nodes that reference it - - Create credential **OpenAI API Key** (header auth, value = `Bearer `), update the OCR node - - Or use the generic credential name `pbToken` / `openaiKey` and update the value in each node's expression `{{ $credentials.pbToken }}` / `{{ $credentials.openaiKey }}` -6. Set workflow-level env vars: - - `MEGA_PB_URL=https://pb.mega.infra.apphub.co.za` - - `MEGA_NEXT_URL=https://your-nextjs-app.example.com` - - `N8N_INGEST_SECRET=` -7. Toggle **Active** to ON - -### Notes on the JSON shape - -- I deliberately avoided the `@n8n/n8n-nodes-langchain.openAi` node because: - - The LangChain node requires a specific LLM credential type per provider, and the credential ID needs to match a real one in your instance - - The raw `httpRequest` to OpenAI's `/v1/chat/completions` endpoint works with a simpler `httpHeaderAuth` credential that's easier to set up - - You can swap it for the LangChain node later if you want richer streaming / tool-calling -- The `Parse OCR JSON` Code node uses JS — n8n's standard Code node. It pulls `choices[0].message.content` out of OpenAI's response and parses the embedded JSON. -- The `POST result to app` node uses `{{ $('List pending DNs').item.json.id }}` to reference the upstream record id from outside the splitInBatches loop body. -- The `Mark ocr-complete in PB` PATCH runs unconditionally (`continueOnFail: true`) — if PB is briefly down, we don't want to block the OCR result delivery; the next poll will pick up any row still in "pending". - -### Once imported, test with one row - -```bash -# Trigger a test by uploading one delivery note via the existing UI: -# /dashboard/purchasing → click "Upload Delivery Note" → pick a PO → upload any image -# -# Or insert directly: -curl -X POST https://pb.mega.infra.apphub.co.za/api/collections/delivery_notes/records \ - -H "Authorization: " \ - -H "Content-Type: application/json" \ - -d '{ - "delivery_note_number": "TEST-DN-001", - "project_code": "TEST", - "purchase_order": "", - "slip_image": "", - "processing_status": "pending", - "is_verified": false - }' -# -# Within 30 s the workflow should pick it up and POST back to the webhook. -``` - -Watch the workflow's execution log in n8n and the DN's `processing_status` in PocketBase admin to confirm the round-trip works. - - ---- - -## Changelog - -### 2026-09-17 — split branch pending disconnection (workflow edit pending) - -The branch `Is first delivery note?` (false) → `Create split DN record` -in workflow `Delivery Note OCR` (`wVbLw4paISkbrIpE`) **needs to be -disconnected**. As of 2026-09-17 the automated disconnection via -`update_workflow` did not land — the MCP tool's `operations` array -parameter is currently being unwrapped to a single object server-side, -so the agent cannot apply the change. A human must remove the -connection in the n8n UI at https://wfdash.n8n.apphub.co.za. - -**Why.** The branch minted a new `delivery_notes` row per slip when a -single scan contained multiple slips, stamping `received_by='n8n-ocr-split'` -and `processing_status='ocr-complete'` with no line items. The dashboard -exception inbox does not list rows in either state, so the orphans -were invisible. Three of those rows existed before the planned -disconnection; they are surfaced and recovered by hand in Task 8 of -the unit-fix plan. - -**Effect once disconnected.** A multi-slip document will land as a -single note (the first item the LLM extracted) with -`processing_status='needs-review'` and a payload that covers all the -items. It shows up in the exception inbox. Phase 2 of the spec -replaces this with the inbox batch pipeline (§4) — at which point -the explicit split branch can be removed. - -**Tracking.** Once disconnected, the orphaned `received_by='n8n-ocr-split'` -row count must stay at 3 from then on. See Task 8 in `.superpowers/sdd/` -for the recovery flow. - -### 2026-08-26 — initial filter bug fix - -The first version of the workflow filter was: - -``` -=processing_status='pending' && slip_image!='' -``` - -PocketBase rejects this with **HTTP 400 "Something went wrong while -processing your request"** because `!=''` doesn't work on file fields — -file fields compare against `null`, not empty string. The corrected -filter is: - -``` -=processing_status='pending' -``` - -The `slip_image` check is also redundant: the upload route only sets -`processing_status='pending'` after a successful file upload, so every -pending row already has a slip. +Every minute, on the minute. The trigger has no special cadence. + +## What you configure in n8n + +### Nodes + +| # | Node | Type | Notes | +|---|------|------|-------| +| 1 | Tick | Schedule Trigger | interval: 1 minute | +| 2 | POST /api/dn-batches/tick | HTTP Request | method `POST`, URL `https://mega.dash.apphub.co.za/api/dn-batches/tick`, header `x-ingest-secret` from a credential | + +Connect Tick → POST. + +### Credentials to set after import + +| Credential name | Type | Value | +|-----------------|------|-------| +| `DN Ingest Secret` (or whatever you name it) | Header Auth | `x-ingest-secret` = the rotated `N8N_INGEST_SECRET` value | + +The HTTP Request node should reference the credential by name — do +**not** paste the secret as a literal in the node's headers table. +Anyone exporting the workflow will leak the secret otherwise, and +that's exactly what happened with the previous four-node workflow. + +### Save and keep active + +After both nodes are in place and the credential is wired, save and +flip the workflow Active. The pipeline is idempotent: running the +tick twice produces identical figures and no duplicate rows, so a +double-fire from a transient n8n restart is harmless. + +## Why it collapsed to two nodes + +The previous workflow polled PocketBase for `processing_status = +"pending"` rows, downloaded each slip, ran an LLM extraction, and +POSTed results back. Tasks 1–16 of the unit-fix plan moved every +one of those steps into Next.js: + +- **Upload route** (`src/app/api/delivery-notes/upload/route.ts`) + hashes the file, creates a `dn_batches` row with `status = + "inbox"`, and persists the bytes to PocketBase. +- **Tick route** (`src/app/api/dn-batches/tick/route.ts`) is the + one place pipeline progression lives. +- **Pipeline modules** in `src/lib/dn-pipeline/`: + - `extract.ts` — page-by-page OCR / PDF read. + - `segment.ts` — group pages into candidate notes. + - `split.ts` — resolve ambiguous segmentations (`unassigned`, + `needsSubsplit`). + - `checks.ts` — per-note and per-batch gating. + - `promote.ts` — write promoted notes to `delivery_notes` / + `delivery_note_items`. + - `run.ts` — the orchestrator the tick route calls. + +What used to be eight nodes is now one HTTP call. Anyone tempted to +"just add the OCR back into n8n" should read the pipeline modules +first — the rules (no cross-page inference, idempotent re-runs, +check-gated promotion) only hold because the steps run in +deterministic order inside one process. + +## What an operator sees + +- `dn_batches.status = "inbox"` — uploaded, not yet ticked. +- `dn_batches.status = "analysing"` — a tick is running for this batch. +- `dn_batches.status = "needs_attention"` — pipeline stopped; open + the row in the inbox board to see why. +- `dn_batches.status = "processed"` — done. `notes_promoted` > 0 + means new rows landed on the purchasing page. + +If the n8n workflow is OFF, batches will sit in `inbox` until it +comes back. There is no fallback timer inside Next.js — the tick +is the heartbeat. + +## Reference + +- Pipeline spec: this repository, `docs/n8n/delivery-note-ocr.md` + (this file) and `src/lib/dn-pipeline/types.ts` for the data + shapes. +- Tick route implementation: `src/app/api/dn-batches/tick/route.ts`. +- Existing workflow id in n8n: `wVbLw4paISkbrIpE` (name: "Delivery + Note OCR" — the name is historical; the workflow body is now the + two-node tick).