14b63e50ab
Adds design spec and n8n handoff doc for the new PO upload flow: browser -> Next.js /api/purchase-orders/upload -> n8n OCR -> Next.js /api/purchase-orders/ingest -> PocketBase via pbAdmin(). Co-Authored-By: Claude <noreply@anthropic.com>
195 lines
6.9 KiB
Markdown
195 lines
6.9 KiB
Markdown
# n8n Workflow — Purchase Order Ingest
|
||
|
||
This is the handoff doc for the n8n workflow that processes Purchase Order uploads from the MEGA Infrastructure dashboard. The Next.js side is implemented by Claude; **you build this workflow in n8n using your MCP access**.
|
||
|
||
## What this workflow does
|
||
|
||
1. Receives a PO file (PDF or image) via webhook from `/api/purchase-orders/upload`.
|
||
2. Extracts text via OCR (Tesseract for images, `pdf-parse` for PDFs).
|
||
3. Parses PO header fields and line items.
|
||
4. Calls back to Next.js with a structured JSON payload + the original file (base64-encoded).
|
||
5. Handles errors per the retry policy below.
|
||
|
||
## Webhook trigger
|
||
|
||
- **Method:** POST
|
||
- **Path:** `purchase-order-upload` (the full URL is `N8N_PO_INGEST_URL` set in your `.env.local`)
|
||
- **Content-Type:** `multipart/form-data`
|
||
- **Body fields:**
|
||
- `file` — the PO file (PDF / JPG / PNG / WEBP, ≤20 MB)
|
||
- `filename` — original filename
|
||
- `contentType` — MIME type
|
||
- `size` — file size in bytes
|
||
|
||
## OCR step
|
||
|
||
Branch on `contentType`:
|
||
|
||
- `image/jpeg`, `image/png`, `image/webp` → **Tesseract OCR** (n8n has a Tesseract node, or use an HTTP node to call a Tesseract microservice).
|
||
- `application/pdf` → **`pdf-parse`** (npm package, available via n8n Function node). Falls back to Tesseract on each rendered page if `pdf-parse` returns empty text.
|
||
|
||
Output: a single string `ocrText`.
|
||
|
||
## Parser step
|
||
|
||
Extract from `ocrText` using regex or an LLM node (recommend LLM for robustness — gpt-4o-mini or your preferred model).
|
||
|
||
**Header fields:**
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| `po_number` | string | Required. Unique. Pattern varies — usually "PO-1234" or just digits. |
|
||
| `supplier` | string | Required. Supplier company name. |
|
||
| `project_code` | string | Required. Match against the `sites.project_code` collection in PocketBase. |
|
||
| `required_by_date` | ISO date | Optional. `YYYY-MM-DD`. |
|
||
| `ordered_by` | string | Optional. Person who placed the order. |
|
||
|
||
**Line items array** (`items[]`):
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| `description` | string | Required. Item description. |
|
||
| `qty` | number | Required. > 0. |
|
||
| `rate` | number | Required. ≥ 0. Unit price. |
|
||
| `unit` | string | Optional. e.g. "ea", "m", "m²". |
|
||
| `material_code` | string | Optional. |
|
||
| `cost_code` | string | Optional. |
|
||
| `disc_percent` | number | Optional. 0–100. |
|
||
| `vat_percent` | number | Optional. 0–100. |
|
||
|
||
Drop any items missing `description`, `qty`, or `rate`. If the entire `items[]` is empty, fail the workflow — don't send an empty PO.
|
||
|
||
## Site lookup (optional)
|
||
|
||
You can optionally look up the site in PocketBase before calling back:
|
||
|
||
```
|
||
GET ${POCKETBASE_URL}/api/collections/sites/records?filter=project_code="${project_code}"&perPage=5
|
||
```
|
||
|
||
But this is **not required** — the Next.js `/api/purchase-orders/ingest` route does the same lookup server-side and will ignore your result. Only do this if you want to enrich the `raw_ocr` payload with site metadata for debugging.
|
||
|
||
## Callback
|
||
|
||
```
|
||
POST ${APP_URL}/api/purchase-orders/ingest
|
||
Headers:
|
||
Content-Type: application/json
|
||
x-ingest-secret: ${N8N_INGEST_SECRET}
|
||
Body: PurchaseOrderIngestPayload (see below)
|
||
```
|
||
|
||
### Payload shape
|
||
|
||
```typescript
|
||
{
|
||
po_number: string,
|
||
project_code: string,
|
||
supplier: string,
|
||
required_by_date?: string, // ISO "YYYY-MM-DD"
|
||
ordered_by?: string,
|
||
items: [
|
||
{
|
||
description: string,
|
||
qty: number,
|
||
rate: number,
|
||
unit?: string,
|
||
material_code?: string,
|
||
cost_code?: string,
|
||
disc_percent?: number,
|
||
vat_percent?: number
|
||
}
|
||
],
|
||
file: {
|
||
filename: string,
|
||
contentType: "application/pdf" | "image/jpeg" | "image/png" | "image/webp",
|
||
base64: string // raw base64, NO "data:" prefix
|
||
},
|
||
raw_ocr: { /* anything useful for debugging */ }
|
||
}
|
||
```
|
||
|
||
### File transport
|
||
|
||
n8n reads the file from the incoming webhook, then re-encodes to base64 in the Function node:
|
||
|
||
```javascript
|
||
const fileBuffer = items[0].binary.file.data; // n8n binary buffer
|
||
const base64 = Buffer.from(fileBuffer).toString('base64');
|
||
|
||
return {
|
||
json: {
|
||
...parsedData,
|
||
file: {
|
||
filename: items[0].binary.file.fileName,
|
||
contentType: items[0].binary.file.mimeType,
|
||
base64
|
||
}
|
||
}
|
||
};
|
||
```
|
||
|
||
## Response handling
|
||
|
||
Branch on the HTTP status code:
|
||
|
||
| Status | Meaning | Action |
|
||
|---|---|---|
|
||
| `200` | Success | Log `id` and `po_number`. Mark workflow as successful. |
|
||
| `400` | Malformed payload | Log `error` + `details`. **Do not retry** — configuration error. |
|
||
| `401` | Bad secret | Log "secret mismatch — check N8N_INGEST_SECRET env". **Do not retry**. |
|
||
| `409` | Duplicate `po_number` | Log "duplicate PO — already in system". **Do not retry**. |
|
||
| `500` | Server error | Log `error` + `step`. **Retry per policy below**. |
|
||
| `408` / network timeout | Transient | **Retry per policy below**. |
|
||
|
||
## Retry policy
|
||
|
||
For `500` and `408`/timeout only:
|
||
|
||
- Retry up to **3 times** with exponential backoff: 10s, 30s, 90s.
|
||
- After 3 failed retries, send to an "Error" workflow branch (dead letter) that logs the failure to your monitoring channel (Slack, email, etc.).
|
||
|
||
Use n8n's built-in **Retry on Fail** node setting with `maxTries: 3` and a custom `waitBetweenTries` expression, or chain three HTTP nodes with `Wait` nodes between them.
|
||
|
||
## Environment variables to set in n8n
|
||
|
||
| Variable | Value | Notes |
|
||
|---|---|---|
|
||
| `APP_URL` | `https://mega.dash.apphub.co.za` | Base URL of the Next.js dashboard. |
|
||
| `N8N_INGEST_SECRET` | (same as in `.env.local`) | Shared secret header. |
|
||
| `POCKETBASE_URL` | `https://pb.mega.infra.apphub.co.za` | Only needed if you do the optional site lookup. |
|
||
|
||
## Testing the workflow
|
||
|
||
Before wiring up the dashboard button, test the webhook directly:
|
||
|
||
```bash
|
||
curl -X POST "$N8N_PO_INGEST_URL" \
|
||
-F "file=@/path/to/test-po.pdf" \
|
||
-F "filename=test-po.pdf" \
|
||
-F "contentType=application/pdf" \
|
||
-F "size=$(stat -f%z /path/to/test-po.pdf)"
|
||
```
|
||
|
||
Watch the n8n execution log. Confirm:
|
||
- The webhook receives the file.
|
||
- OCR runs (check OCR node output).
|
||
- The parsed payload has all required fields populated.
|
||
- The callback POST to `/api/purchase-orders/ingest` returns `200` with `id` and `po_number`.
|
||
- The new PO appears in the PocketBase `purchase_orders` collection.
|
||
|
||
Test cases:
|
||
|
||
1. **Clean PDF** — well-structured PO, all fields present. Expect 200.
|
||
2. **Scanned image (JPG)** — Tesseract OCR. Expect 200 (or 400 if OCR is too garbled to parse).
|
||
3. **PDF with no extractable text** — OCR fallback should kick in.
|
||
4. **Duplicate PO** — upload the same PO twice. First call 200, second call 409.
|
||
5. **Unknown project_code** — expect 200 with `warnings: ["site_unmatched"]`.
|
||
|
||
## Future: email ingestion
|
||
|
||
The same `/api/purchase-orders/ingest` callback will be reused when email ingestion is built. Add a second n8n workflow with an **Email Trigger** (IMAP / Gmail / Outlook node) that:
|
||
1. Downloads the PO attachment from the email.
|
||
2. Runs the same OCR + parser + callback flow.
|
||
|
||
No changes needed on the Next.js side.
|