MEGAINFRA-WEBAPP/docs/n8n/purchase-order-ingest-implementation.md

# Purchase Order Ingest — Implementation Notes

Cron-driven async queue. Built via n8n MCP on 2026-08-19.

## What it does

Every 5 minutes the workflow polls `data_import_batches` for `batch_type=purchase_orders && status=pending` rows, atomically claims them by patching `status=processing && attempts+1`, downloads the original file from PocketBase, runs the AI Agent (MiniMax-M3 + Structured Output Parser) to extract the PO header and line items, and POSTs the structured payload to `/api/purchase-orders/process-batch` on the dashboard. The dashboard route owns the PB inserts and updates the batch row to `processed`/`failed`/`dead`.

The previous webhook-based synchronous flow is retired — the browser upload no longer forwards to n8n at all.

## Workflow

- **Workflow ID:** `005iI4jX5mq9DsTR`
- **URL:** https://wfdash.n8n.apphub.co.za/workflow/005iI4jX5mq9DsTR
- **Trigger:** `Every 5 minutes` (Schedule Trigger)
- **Status:** published (active version is the cron flow)
- **Project:** Personal (David Els <david.els1@outlook.com>)
- **Credentials:** `MiniMax account` is auto-bound to all four `lmChatMinimax` chat-model subnodes (PDF Agent, Image Agent, Parser Model PDF, Parser Model Image).

## Node map

```
Every 5 minutes → Auth PB Admin
                    ├→ Watchdog: List Stuck → Watchdog: Reset Stuck Rows (terminal)
                    └→ Fetch Pending Batches → Loop Over Items
                                                  ├→ Skip Empty Items → Has ID? → Atomic Claim → Claim Successful?
                                                  │                                                            ├ (true)  → Download File → Route by Content Type → AI Agent / AI Agent1
                                                  │                                                            └ (false) → Loop Over Items
                                                  └──────────────────────────────────────────────────────────────────────────────── AI Agent / AI Agent1 → Build Callback Payload → Send Callback
                                                                                                                                                                                                                      ├ (ok)    → Loop Over Items
                                                                                                                                                                                                                      └ (error) → Escalate Failure → Loop Over Items
```

| Node | Type | Purpose |
|---|---|---|
| Every 5 minutes | scheduleTrigger | Cron tick |
| Auth PB Admin | httpRequest (executeOnce) | POST `/api/collections/_superusers/auth-with-password` → returns `{ token, record }`. Downstream nodes reference `$('Auth PB Admin').item.json.token`. |
| Watchdog: List Stuck | httpRequest (executeOnce) | GET batches where `status=processing` and `last_attempt_at < 15 min ago` |
| Watchdog: Reset Stuck Rows | code (runOnceForAllItems) | PATCH each stuck row back to `pending` with `last_error="watchdog_reset"` |
| Fetch Pending Batches | httpRequest (executeOnce) | GET batches where `status=pending` and (`next_retry_at=""` or `next_retry_at<now`), sorted by `date_uploaded`, max 10 |
| Loop Over Items | splitInBatches (batchSize=1) | Process one batch at a time |
| Skip Empty Items | code (runOnceForAllItems) | Filter the empty wrapper that splitInBatches emits when input has 0 items; replace with `{ skipped: true }` |
| Has ID? | if | Route items with `id` to Atomic Claim; terminate items without |
| Atomic Claim | httpRequest | PATCH `/records/{id}?filter=status="pending"` — atomic state transition to `processing` (race-safe) |
| Claim Successful? | if | `status=processing` → Download File; otherwise loop back |
| Download File | httpRequest (responseFormat=file, outputPropertyName=data) | GET `/files/data_import_batches/{id}/{original_file}` — file appears in `$binary.data` |
| Route by Content Type | switch | PDF branch → AI Agent; image branch → AI Agent1 |
| AI Agent / AI Agent1 | langchain agent + parser + MiniMax chat model | Extract PO header + line items per the JSON schema |
| Build Callback Payload | code (runOnceForEachItem) | Unwrap AI Agent `output` wrapper; build `{ batch_id, extracted: {...} }`. The file is NOT in the payload — PB has it on the batch row. |
| Send Callback | httpRequest (onError: continueErrorOutput) | POST `/api/purchase-orders/process-batch` with `x-ingest-secret` header |
| Escalate Failure | code (runOnceForEachItem) | On Send Callback error: PATCH batch to `failed` with `next_retry_at=+1min` (or `+5min` after attempt 2); or `dead` after attempt 3. Loop continues. |

## Hardcoded values (no n8n-side env vars)

Self-hosted n8n on Coolify does **not** expose `$env.*` to workflows and has **no Variables UI** (paid feature only). Every value the workflow needs is baked into the workflow JSON itself. Search for these literals if you ever rotate them:

| Where | Value | Notes |
|---|---|---|
| Auth PB Admin, Watchdog, Fetch Pending, Atomic Claim, Download File | `https://pb.mega.infra.apphub.co.za/...` | PB base URL |
| Auth PB Admin | `jsonBody: {"identity":"admin@mega.co.za","password":"12345678"}` | PB superuser creds — change both before going live |
| Watchdog: List Stuck filter | `batch_type="purchase_orders" && status="processing" && last_attempt_at<"{{ $now.minus({minutes:15}).toISO() }}"` | Stuck-row threshold is 15 min |
| Fetch Pending Batches filter | `batch_type="purchase_orders" && status="pending" && (next_retry_at="" \|\| next_retry_at<"{{ $now.toISO() }}")` | Respects retry backoff |
| Send Callback | `url: https://mega.dash.apphub.co.za/api/purchase-orders/process-batch` | APP_URL hardcoded |
| Send Callback | `x-ingest-secret: b8dffe7b00fca75272bf9a090d15a3c59fa0ec20d6bb232f7bb792db0921ac3b` | Mirrors `cooler.env N8N_INGEST_SECRET`. Rotate both sides together. |
| Escalate Failure | Backoff constants `60000` (1 min) and `300000` (5 min) inline in JS | Adjust in the jsCode |
| AI Agent / AI Agent1 | `model: MiniMax-M3` | Hardcoded model name in `lmChatMinimax` subnode |

## Atomic claim semantics

Atomic Claim uses PB's filter-on-PATCH trick:

```
PATCH /api/collections/data_import_batches/records/{id}?filter=status="pending"
Body: { "status": "processing", "attempts": "={{ $json.attempts + 1 }}", ... }
```

If the row was already claimed by another worker (status != pending), the filter rejects the PATCH and PB returns 0 records. Claim Successful? checks `$json.status === "processing"` — if not, the loop skips this batch. Two cron workers can never claim the same row.

## Escalation semantics

Escalate Failure runs only on Send Callback error. It does **not** run if the AI Agent returned no items (that's a "nothing to do" condition that the process-batch route handles by transitioning the batch to `dead` itself).

- attempt 1 → status=`failed`, `next_retry_at=now+1min`
- attempt 2 → status=`failed`, `next_retry_at=now+5min`
- attempt 3 → status=`dead`, `next_retry_at=null`
- attempt ≥ 4 → still `dead` (idempotent)

The cron tick's `Fetch Pending Batches` filter respects `next_retry_at`, so a row with `next_retry_at=...future` is skipped until the backoff elapses.

## Watchdog semantics

Watchdog runs in parallel with Fetch Pending on every tick. It picks up rows that have been stuck in `processing` for more than 15 minutes (e.g. crash mid-process, n8n restart, killed execution) and resets them to `pending`. Without this, a crashed row would block its `attempts` counter from advancing forever.

## Why the file is NOT in the callback payload

The new `/api/purchase-orders/process-batch` route accepts just `{ batch_id, extracted: {...} }`. The original file lives on the batch row in PB; the route fetches it from PB on success, copies it to the new `purchase_orders` row, and clears `original_file` from the batch. This keeps payloads small and avoids the binary-passthrough bugs that plagued the old `/api/purchase-orders/ingest` route.

## PB filter syntax gotcha

PocketBase uses `&&` and `||` for logical operators — `AND` and `OR` return 400. Both filters in this workflow use the `&&`/`||` form. The `=` prefix is required on expression fields containing `{{ ... }}` (n8n only evaluates fields that explicitly opt in to templating).

## Known cosmetic warnings

Two `INVALID_PARAMETER` warnings persist:

```
Field "parameters.queryParameters.parameters.1.value" has wrong type. Expected string, but got number.
Field "parameters.queryParameters.parameters.2.value" has wrong type. Expected string, but got number.
```

These fire on the `perPage` query parameter (which the schema expects as a string). n8n auto-coerces numeric values into the URL query string at HTTP time, so the warnings are harmless. Suppressing them would require switching to a string literal; not worth the loss of readability.

## Future work

- **Per-node retry** — Send Callback currently uses `onError: continueErrorOutput` so the error branch can run Escalate Failure. The HTTP node itself doesn't retry. If you want to retry transient network errors within the same execution, set `retryOnFail: true, maxTries: 3, waitBetweenTries: 5000` on Send Callback.
- **Email ingestion** — the callback contract is identical, so adding email attachments to the same queue requires no changes here.
- **Webhook fallback** — if cron latency (up to 5 min) is too long for some flows, add a webhook node that writes the batch row directly to PB and lets the cron pick it up.

## Manual smoke test

The cron fires every 5 minutes — the empty-input path is verified (execution
`55206` succeeded). For the happy path with a real file, run these steps
from the deployed stack. Each step has an expected outcome you can check.

### Step 1 — Confirm the cron is alive

Open [the workflow](https://wfdash.n8n.apphub.co.za/workflow/005iI4jX5mq9DsTR).
The last execution timestamp at top-right should advance within 5 min of
the previous one. If it doesn't, check `Every 5 minutes` is enabled and
the workflow is Published.

### Step 2 — Upload a real PO via the dashboard

1. Visit `/dashboard/purchasing` on the deployed app.
2. Click **Upload Purchase Order** and pick a real PDF PO that has at least
   one matching site in the `sites` collection.
3. Expect a 202 + `{ batch_id, status: "pending" }` toast within a couple
   seconds. Copy the `batch_id`.

### Step 3 — Watch the cron pick it up

Wait up to 5 min. In the n8n workflow's **Executions** tab, a new run
should appear with the following node trace:

```
Every 5 minutes → Auth PB Admin → Fetch Pending Batches → Loop Over Items
  → Skip Empty Items → Has ID? → Atomic Claim → Claim Successful?
  → Download File → Route by Content Type → AI Agent → Build Callback Payload
  → Send Callback
```

### Step 4 — Verify the batch row transitioned

In the PB admin UI for `data_import_batches`, find the row by `batch_id`.
Expect:

| Field | Value |
|---|---|
| `status` | `processed` |
| `attempts` | `1` |
| `processed_at` | ISO timestamp from the run |
| `processed_record_id` | new `purchase_orders` row id |
| `last_error` | empty |
| `original_file` | empty (file moved off the batch) |

A matching `purchase_orders` row should exist with `original_file` pointing
to the same filename, and the PO header + line items populated.

### Step 5 — Verify dedup

Upload the **same** PO file again. Expect either:

- The upload route returns 409 with the existing `purchase_orders.id`, or
- A new batch row is created but the AI Agent extracts the same
  `po_number`, and `process-batch` either links to the existing PO or
  rejects with `duplicate`.

(Exact behavior is set in Task 7 — check the route's source if unsure.)

### Step 6 — Verify dead-state on unmatched site

Upload a PO whose site name doesn't match any row in the `sites`
collection. After the cron runs, expect:

- `status = dead`
- `attempts = 3`
- `next_retry_at = null`
- `last_error` mentions "site not matched" (or similar)
- No `purchase_orders` row created

### Step 7 — Verify retry UI

In `/dashboard/purchasing` or wherever the batch list lives, find the
`dead` row from Step 6. Expect a **Retry** button. Clicking it sets
`status=pending` and `attempts=0` (or whatever the route contract is —
see the retry route source).

### Step 8 — Verify manual retry runs the cron

Within 5 min of clicking Retry, the row should re-enter `processing` and
either transition to `processed` or back to `failed`/`dead` per the
outcome. Check the **Executions** tab for a new run that includes the
batch id.

### Step 9 — Cleanup

Delete any test `purchase_orders` rows created during the smoke test.
Delete any test `data_import_batches` rows that don't have
`status=processed` and a real `processed_record_id`.

---

If any step fails, check the **Executions** tab in n8n first — the node
trace shows exactly which step died. The most common failure modes:

- **AI Agent timeout** → row goes `failed` → `next_retry_at=+1min`; the
  next cron tick re-attempts. If it keeps timing out, check the model
  name in the `lmChatMinimax` subnode.
- **Send Callback 4xx** → row goes `dead` after 3 attempts. Check
  `x-ingest-secret` matches `cooler.env N8N_INGEST_SECRET`, and that
  `/api/purchase-orders/process-batch` is reachable from n8n.
- **Send Callback 5xx** → row goes `failed` → re-attempted. Check the
  Next.js logs for the actual exception.