David/MEGAINFRA-WEBAPP
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(n8n): rewrite for the two-node tick workflow

Browse source
This commit is contained in:
DEls 2026-09-17 15:23:16 +02:00
parent 6317f5da09
commit 49b6547c0c
1 changed files with 99 additions and 387 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

486
docs/n8n/delivery-note-ocr.md
View file

@ -1,395 +1,107 @@
# n8n Workflow — Delivery Note OCR
# n8n Workflow — Delivery Note Tick
This is the handoff doc for the n8n workflow that processes delivery
note slips uploaded from the MEGA Infrastructure dashboard. The Next.js
side is implemented by Claude; **you build this workflow in n8n using
your MCP access**.
This is the handoff doc for the n8n workflow that drives the
delivery note inbox pipeline. The pipeline itself (extract → segment
→ split → checks → promote) lives in `src/lib/dn-pipeline/` and is
run entirely inside Next.js. n8n's only job is to call the tick
endpoint every minute.
## What this workflow does
1. **Watches** the `delivery_notes` PocketBase collection for new rows
where `processing_status = "pending"`. Either poll every 30–60 s
or wire a PocketBase webhook to trigger on row insert.
2. For each pending DN, downloads `slip_image` from PocketBase.
3. Runs OCR on the slip (image → vision model, PDF → text extract first
then vision on rendered pages).
4. Sends the OCR text through an LLM with a structured-output prompt to
extract:
- PO number
- Delivery note number
- Date received
- Received-by name (signature)
- Line items: description, qty, unit, unit_price
- Overall confidence (0..1) the AI has in the extraction
5. **POSTs the structured result back to Next.js** at
`POST /api/delivery-notes/[id]/ocr-result` (where `[id]` is the
PocketBase record id of the delivery note).
Two nodes. That's it.
The Next.js route handles:
- Updating the DN's `processing_status` based on confidence
- Creating `received_items` rows
- Marking `is_verified = true` for high-confidence matches
- Idempotency (safe to replay)
1. **Schedule Trigger** — fires every 1 minute.
2. **HTTP Request** — `POST https://mega.dash.apphub.co.za/api/dn-batches/tick`
with header `x-ingest-secret: <credential value>`.
The Next.js route (`src/app/api/dn-batches/tick/route.ts`) picks up
the call, walks every `dn_batches` row whose `status` is one of
`inbox` / `analysing` / `needs_attention`, and advances each one
through the pipeline until it reaches `processed` (or stays in
`needs_attention` until a human acts). All OCR, segmentation,
checks, and PocketBase writes happen in Next.js — n8n does not see
a slip.
## When this workflow runs
Poll PocketBase on a schedule (every 30–60 s) for rows where:
```
processing_status = "pending"
```
After processing, the route will set `processing_status` to either:
| Value | Meaning |
|---|---|
| `auto-verified` | Confidence ≥ 0.8 — fully verified, `received_items` created, `is_verified = true` |
| `needs-review` | Confidence 0.5–0.8 — `received_items` created as draft, `is_verified = false`, user reviews |
| `needs-review` | Confidence < 0.5 — no `received_items` created, user manually captures line items |
| `failed` | OCR pipeline errored; row stays pending until retried |
Use this status in your polling query to find work.
## Downloading the slip
```
GET /api/files/{collectionId}/{recordId}/{fieldName}/{filename}
```
Or use the PocketBase SDK to fetch the file directly. The `slip_image`
field on `delivery_notes` records is the file.
## What to extract (LLM structured output)
| Field | Type | Notes |
|---|---|---|
| `po_number` | string \| null | Match against `purchase_orders.purchase_order_number` |
| `delivery_note_number` | string \| null | Match against `delivery_note_number` on existing DNs |
| `date_received` | `YYYY-MM-DD` \| null | From the slip header. Overrides the upload-time date. |
| `received_by_name` | string \| null | Signature/name on the slip |
| `confidence` | number 0..1 (or 0..100) | Overall confidence. The route auto-clamps. |
| `raw_ocr` | string \| null | Full OCR text — saved on `raw_ocr_data.raw_ocr` for audit (truncated to 50 KB) |
| `items` | array (see below) | One entry per line item on the slip |
### `items[]` shape
| Field | Type | Notes |
|---|---|---|
| `description` | string | Free text from the slip |
| `qty` | number | Quantity received |
| `unit` | string \| null | bag, m², ea, etc. |
| `unit_price` | number \| null | From the slip (optional, used in audit) |
| `matched_po_item_id` | PocketBase id \| null | The id of the matching `purchase_order_items` row, if you can match it |
| `match_confidence` | number 0..1 | Per-line confidence. Saved in `condition_notes`. |
If you can't match a line to a PO item, leave `matched_po_item_id = null`.
The user will fix it manually in the exception inbox.
## Confidence score (how to compute it)
The route uses three thresholds: **0.8** (auto), **0.5** (draft), below
that (manual). Compute it however you want, but here's a reasonable
starting formula:
```
score = 0
if po_number matches an existing PO: score += 0.30
if delivery_note_number parseable: score += 0.10
if supplier matches PO supplier: score += 0.10
if line items fuzzy-match PO items: score += 0.30 × (% matched)
if line totals add up to grand total: score += 0.20
```
Tune the weights after running for a week or two. The thresholds above
assume a default of `gpt-4o-mini` or similar mid-tier vision model.
## Callback endpoint
```
POST /api/delivery-notes/{id}/ocr-result
Headers:
Content-Type: application/json
x-ingest-secret: ${N8N_INGEST_SECRET} ← same env var as the PO webhook
Body: (see shape above)
```
`{id}` is the PocketBase id of the delivery note (15-char alphanumeric).
### Response codes
| Code | Meaning | What to do |
|---|---|---|
| `200` | Accepted. `received_items` rows created (or skipped if tier=manual). | Mark `processing_status="ocr-complete"` in your n8n node so you don't reprocess |
| `401` | Bad / missing `x-ingest-secret` | Log "secret mismatch — check N8N_INGEST_SECRET env". Do **not** retry. |
| `404` | Delivery note id doesn't exist | Log "dn_not_found for id={id}" — the record may have been deleted. |
| `400` | Body validation failed | Log the body; do not retry; user needs to investigate. |
| `500` | PocketBase update failed | Retry once with backoff. After 2 failures, mark `processing_status="failed"` so the user sees it in the exception inbox. |
### Response body (200)
```json
{
"ok": true,
"tier": "auto" | "draft" | "manual",
"receivedItemsCreated": 3,
"receivedItems": [
{ "id": "...", "po_item_id": "...", "qty": 25 }
],
"durationMs": 1234
}
```
## Suggested n8n flow (high-level)
1. **Schedule Trigger** every 30 s.
2. **PocketBase HTTP Request**: list delivery_notes where
`processing_status = "pending"` (filter via PB REST API).
3. For each row:
1. Download `slip_image`
2. OCR (vision model)
3. LLM structured extraction (with the prompt below)
4. Compute confidence
5. **HTTP Request** → `POST /api/delivery-notes/{id}/ocr-result`
with the body and `x-ingest-secret`
6. On 200: log `receivedItemsCreated`, move on
7. On 5xx: retry once after 5 s; on second failure, update the DN's
`processing_status="failed"` directly via PocketBase API so the
exception inbox surfaces it
## Recommended LLM prompt (skeleton)
```
You are extracting structured data from a delivery note slip image.
Return ONLY valid JSON matching this schema:
{
"po_number": string | null,
"delivery_note_number": string | null,
"date_received": "YYYY-MM-DD" | null,
"received_by_name": string | null,
"confidence": number, // 0..1 — your overall certainty
"items": [
{
"description": string,
"qty": number,
"unit": string | null,
"unit_price": number | null
}
]
}
Rules:
- confidence is your honest assessment: 0.9+ only if everything is
clearly legible, 0.5–0.8 if some fields are guesses, < 0.5 if
the slip is unreadable
- dates must be ISO format or null
- quantities must be numbers (not strings)
- if the slip has no readable items, return an empty items array
```
If you have a PO list to match against (recommended), inject it as
context:
```
Reference purchase orders:
- PO-1234 | Cement 50kg bag × 50 @ R110 | Acme Supplies
- PO-1235 | ...
For each line item, return `matched_po_item_id` if you can identify
the matching PO line; otherwise null.
```
## Where things live in PocketBase
- `delivery_notes.processing_status` — `"pending"` (your queue), or `"auto-verified"` / `"needs-review"` / `"failed"` after the webhook fires
- `delivery_notes.ocr_confidence` — 0..100 integer
- `delivery_notes.ocr_completed_at` — ISO timestamp of when OCR finished
- `delivery_notes.raw_ocr_data` — JSON, includes the full payload you sent (so audits can replay)
- `received_items` — auto-created rows from successful (auto + draft) tiers
## Environment
- `N8N_INGEST_SECRET` — shared secret header (same value as `.env.local`)
- `MEGA_PB_URL` — PocketBase URL (already in `.env.local`)
- `MEGA_PB_ADMIN_TOKEN` — admin token for the workflow (PB service token, not user)
---
## Workflow code (n8n Workflow SDK)
The reference workflow is in [`docs/n8n/delivery-note-ocr-workflow.ts`](./delivery-note-ocr-workflow.ts). It's written against `@n8n/workflow-sdk`. To import into your n8n instance:
```bash
# In your n8n project (or a separate dir with the SDK installed)
npm install @n8n/workflow-sdk
npx n8n-workflow-sdk import ./delivery-note-ocr-workflow.ts \
--base-url "$MEGA_N8N_URL" \
--api-key "$MEGA_N8N_API_KEY"
```
(Or whatever import subcommand the SDK exposes — check `npx @n8n/workflow-sdk --help` for current usage.)
If the SDK import command isn't available in your environment, paste the workflow JSON into n8n manually via *Workflows → Import from File*. The TypeScript source documents every node + its config so the JSON shape is one-to-one.
### Workflow shape (one node per line)
```
[Schedule Trigger — every 30 s]
│
▼
[List pending DNs (PB HTTP)] ── 0..N items
│
▼
[SplitInBatches — batchSize 1]
│
▼
[Download slip image] ← loop body
│
▼
[Vision LLM extract (GPT-4o-mini)]
│
▼
[Structured output parser]
│
▼
[POST /api/delivery-notes/{id}/ocr-result]
│
▼
[Mark as ocr-complete in PB]
│
▼
[nextBatch → loop back]
```
### Credentials to configure after import
| Node | Credential name | What to put |
|---|---|---|
| List pending DNs / Download slip | PocketBase Admin Token | `POCKETBASE_ADMIN_TOKEN` from `.env.local` |
| OCR extract | OpenAI Vision API | your OpenAI key (or swap to Claude/Gemini — node config is open to whatever the LangChain node supports) |
| POST result to app | (none — uses `x-ingest-secret` header from `N8N_INGEST_SECRET`) | — |
| Mark as ocr-complete | PocketBase Admin Token | same as above |
### Workflow environment variables
| Variable | Value |
|---|---|
| `MEGA_PB_URL` | `https://pb.mega.infra.apphub.co.za` |
| `MEGA_NEXT_URL` | The public URL of your Next.js app (the one the app sees as `request.url`) |
| `N8N_INGEST_SECRET` | Same value as in `.env.local` |
### Customising for Claude / Gemini / etc.
Swap the `@n8n/n8n-nodes-langchain.openAi` node in `ocrExtract` for the equivalent provider (e.g. `@n8n/n8n-nodes-langchain.anthropic` with model `claude-3-5-sonnet-latest`, or Google's Gemini vision model). Keep the structured-output prompt the same — vision models all accept JSON schema instructions.
---
## Importable JSON workflow
The workflow is also available as a ready-to-import JSON file at
[`docs/n8n/delivery-note-ocr-workflow.json`](./delivery-note-ocr-workflow.json). It has been validated as parseable JSON.
### How to import
1. Open your n8n instance (https://wfdash.n8n.apphub.co.za)
2. **Workflows → ⋯ menu → Import from File** (or paste JSON via *Import from Clipboard*)
3. Select `delivery-note-ocr-workflow.json`
4. n8n will create the workflow with placeholder credential IDs (`REPLACE_PB_TOKEN_ID`, `REPLACE_OPENAI_KEY_ID`)
5. Replace those placeholders with real credentials:
- Create credential **PocketBase Admin Token** (header auth, value = your PB superuser token), update the 4 nodes that reference it
- Create credential **OpenAI API Key** (header auth, value = `Bearer <your-key>`), update the OCR node
- Or use the generic credential name `pbToken` / `openaiKey` and update the value in each node's expression `{{ $credentials.pbToken }}` / `{{ $credentials.openaiKey }}`
6. Set workflow-level env vars:
- `MEGA_PB_URL=https://pb.mega.infra.apphub.co.za`
- `MEGA_NEXT_URL=https://your-nextjs-app.example.com`
- `N8N_INGEST_SECRET=<same value as in your .env.local>`
7. Toggle **Active** to ON
### Notes on the JSON shape
- I deliberately avoided the `@n8n/n8n-nodes-langchain.openAi` node because:
- The LangChain node requires a specific LLM credential type per provider, and the credential ID needs to match a real one in your instance
- The raw `httpRequest` to OpenAI's `/v1/chat/completions` endpoint works with a simpler `httpHeaderAuth` credential that's easier to set up
- You can swap it for the LangChain node later if you want richer streaming / tool-calling
- The `Parse OCR JSON` Code node uses JS — n8n's standard Code node. It pulls `choices[0].message.content` out of OpenAI's response and parses the embedded JSON.
- The `POST result to app` node uses `{{ $('List pending DNs').item.json.id }}` to reference the upstream record id from outside the splitInBatches loop body.
- The `Mark ocr-complete in PB` PATCH runs unconditionally (`continueOnFail: true`) — if PB is briefly down, we don't want to block the OCR result delivery; the next poll will pick up any row still in "pending".
### Once imported, test with one row
```bash
# Trigger a test by uploading one delivery note via the existing UI:
# /dashboard/purchasing → click "Upload Delivery Note" → pick a PO → upload any image
#
# Or insert directly:
curl -X POST https://pb.mega.infra.apphub.co.za/api/collections/delivery_notes/records \
-H "Authorization: <your token>" \
-H "Content-Type: application/json" \
-d '{
"delivery_note_number": "TEST-DN-001",
"project_code": "TEST",
"purchase_order": "<some PO id>",
"slip_image": "<upload a test slip first>",
"processing_status": "pending",
"is_verified": false
}'
#
# Within 30 s the workflow should pick it up and POST back to the webhook.
```
Watch the workflow's execution log in n8n and the DN's `processing_status` in PocketBase admin to confirm the round-trip works.
---
## Changelog
### 2026-09-17 — split branch pending disconnection (workflow edit pending)
The branch `Is first delivery note?` (false) → `Create split DN record`
in workflow `Delivery Note OCR` (`wVbLw4paISkbrIpE`) **needs to be
disconnected**. As of 2026-09-17 the automated disconnection via
`update_workflow` did not land — the MCP tool's `operations` array
parameter is currently being unwrapped to a single object server-side,
so the agent cannot apply the change. A human must remove the
connection in the n8n UI at https://wfdash.n8n.apphub.co.za.
**Why.** The branch minted a new `delivery_notes` row per slip when a
single scan contained multiple slips, stamping `received_by='n8n-ocr-split'`
and `processing_status='ocr-complete'` with no line items. The dashboard
exception inbox does not list rows in either state, so the orphans
were invisible. Three of those rows existed before the planned
disconnection; they are surfaced and recovered by hand in Task 8 of
the unit-fix plan.
**Effect once disconnected.** A multi-slip document will land as a
single note (the first item the LLM extracted) with
`processing_status='needs-review'` and a payload that covers all the
items. It shows up in the exception inbox. Phase 2 of the spec
replaces this with the inbox batch pipeline (§4) — at which point
the explicit split branch can be removed.
**Tracking.** Once disconnected, the orphaned `received_by='n8n-ocr-split'`
row count must stay at 3 from then on. See Task 8 in `.superpowers/sdd/`
for the recovery flow.
### 2026-08-26 — initial filter bug fix
The first version of the workflow filter was:
```
=processing_status='pending' && slip_image!=''
```
PocketBase rejects this with **HTTP 400 "Something went wrong while
processing your request"** because `!=''` doesn't work on file fields —
file fields compare against `null`, not empty string. The corrected
filter is:
```
=processing_status='pending'
```
The `slip_image` check is also redundant: the upload route only sets
`processing_status='pending'` after a successful file upload, so every
pending row already has a slip.
Every minute, on the minute. The trigger has no special cadence.
## What you configure in n8n
### Nodes
| # | Node | Type | Notes |
|---|------|------|-------|
| 1 | Tick | Schedule Trigger | interval: 1 minute |
| 2 | POST /api/dn-batches/tick | HTTP Request | method `POST`, URL `https://mega.dash.apphub.co.za/api/dn-batches/tick`, header `x-ingest-secret` from a credential |
Connect Tick → POST.
### Credentials to set after import
| Credential name | Type | Value |
|-----------------|------|-------|
| `DN Ingest Secret` (or whatever you name it) | Header Auth | `x-ingest-secret` = the rotated `N8N_INGEST_SECRET` value |
The HTTP Request node should reference the credential by name — do
**not** paste the secret as a literal in the node's headers table.
Anyone exporting the workflow will leak the secret otherwise, and
that's exactly what happened with the previous four-node workflow.
### Save and keep active
After both nodes are in place and the credential is wired, save and
flip the workflow Active. The pipeline is idempotent: running the
tick twice produces identical figures and no duplicate rows, so a
double-fire from a transient n8n restart is harmless.
## Why it collapsed to two nodes
The previous workflow polled PocketBase for `processing_status =
"pending"` rows, downloaded each slip, ran an LLM extraction, and
POSTed results back. Tasks 1–16 of the unit-fix plan moved every
one of those steps into Next.js:
- **Upload route** (`src/app/api/delivery-notes/upload/route.ts`)
hashes the file, creates a `dn_batches` row with `status =
"inbox"`, and persists the bytes to PocketBase.
- **Tick route** (`src/app/api/dn-batches/tick/route.ts`) is the
one place pipeline progression lives.
- **Pipeline modules** in `src/lib/dn-pipeline/`:
- `extract.ts` — page-by-page OCR / PDF read.
- `segment.ts` — group pages into candidate notes.
- `split.ts` — resolve ambiguous segmentations (`unassigned`,
`needsSubsplit`).
- `checks.ts` — per-note and per-batch gating.
- `promote.ts` — write promoted notes to `delivery_notes` /
`delivery_note_items`.
- `run.ts` — the orchestrator the tick route calls.
What used to be eight nodes is now one HTTP call. Anyone tempted to
"just add the OCR back into n8n" should read the pipeline modules
first — the rules (no cross-page inference, idempotent re-runs,
check-gated promotion) only hold because the steps run in
deterministic order inside one process.
## What an operator sees
- `dn_batches.status = "inbox"` — uploaded, not yet ticked.
- `dn_batches.status = "analysing"` — a tick is running for this batch.
- `dn_batches.status = "needs_attention"` — pipeline stopped; open
the row in the inbox board to see why.
- `dn_batches.status = "processed"` — done. `notes_promoted` > 0
means new rows landed on the purchasing page.
If the n8n workflow is OFF, batches will sit in `inbox` until it
comes back. There is no fallback timer inside Next.js — the tick
is the heartbeat.
## Reference
- Pipeline spec: this repository, `docs/n8n/delivery-note-ocr.md`
(this file) and `src/lib/dn-pipeline/types.ts` for the data
shapes.
- Tick route implementation: `src/app/api/dn-batches/tick/route.ts`.
- Existing workflow id in n8n: `wVbLw4paISkbrIpE` (name: "Delivery
Note OCR" — the name is historical; the workflow body is now the
two-node tick).