MEGAINFRA-WEBAPP/docs/n8n/purchase-order-ingest-implementation.md

Purchase Order Ingest — Implementation Notes

Cron-driven async queue. Built via n8n MCP on 2026-08-19.

What it does

Every 5 minutes the workflow polls data_import_batches for batch_type=purchase_orders && status=pending rows, atomically claims them by patching status=processing && attempts+1, downloads the original file from PocketBase, runs the AI Agent (MiniMax-M3 + Structured Output Parser) to extract the PO header and line items, and POSTs the structured payload to /api/purchase-orders/process-batch on the dashboard. The dashboard route owns the PB inserts and updates the batch row to processed/failed/dead.

The previous webhook-based synchronous flow is retired — the browser upload no longer forwards to n8n at all.

Workflow

• Workflow ID: 005iI4jX5mq9DsTR
• URL: https://wfdash.n8n.apphub.co.za/workflow/005iI4jX5mq9DsTR
• Trigger: Every 5 minutes (Schedule Trigger)
• Status: published (active version is the cron flow)
• Project: Personal (David Els david.els1@outlook.com)
• Credentials: MiniMax account is auto-bound to all four lmChatMinimax chat-model subnodes (PDF Agent, Image Agent, Parser Model PDF, Parser Model Image).

Node map

Every 5 minutes → Auth PB Admin
                    ├→ Watchdog: List Stuck → Watchdog: Reset Stuck Rows (terminal)
                    └→ Fetch Pending Batches → Loop Over Items
                                                  ├→ Skip Empty Items → Has ID? → Atomic Claim → Claim Successful?
                                                  │                                                            ├ (true)  → Download File → Route by Content Type → AI Agent / AI Agent1
                                                  │                                                            └ (false) → Loop Over Items
                                                  └──────────────────────────────────────────────────────────────────────────────── AI Agent / AI Agent1 → Build Callback Payload → Send Callback
                                                                                                                                                                                                                      ├ (ok)    → Loop Over Items
                                                                                                                                                                                                                      └ (error) → Escalate Failure → Loop Over Items

Node	Type	Purpose
Every 5 minutes	scheduleTrigger	Cron tick
Auth PB Admin	httpRequest (executeOnce)	POST /api/collections/_superusers/auth-with-password → returns { token, record }. Downstream nodes reference $('Auth PB Admin').item.json.token.
Watchdog: List Stuck	httpRequest (executeOnce)	GET batches where status=processing and last_attempt_at < 15 min ago
Watchdog: Reset Stuck Rows	code (runOnceForAllItems)	PATCH each stuck row back to pending with last_error="watchdog_reset"
Fetch Pending Batches	httpRequest (executeOnce)	GET batches where status=pending and (next_retry_at="" or next_retry_at<now), sorted by date_uploaded, max 10
Loop Over Items	splitInBatches (batchSize=1)	Process one batch at a time
Skip Empty Items	code (runOnceForAllItems)	Filter the empty wrapper that splitInBatches emits when input has 0 items; replace with { skipped: true }
Has ID?	if	Route items with id to Atomic Claim; terminate items without
Atomic Claim	httpRequest	PATCH /records/{id}?filter=status="pending" — atomic state transition to processing (race-safe)
Claim Successful?	if	status=processing → Download File; otherwise loop back
Download File	httpRequest (responseFormat=file, outputPropertyName=data)	GET /files/data_import_batches/{id}/{original_file} — file appears in $binary.data
Route by Content Type	switch	PDF branch → AI Agent; image branch → AI Agent1
AI Agent / AI Agent1	langchain agent + parser + MiniMax chat model	Extract PO header + line items per the JSON schema
Build Callback Payload	code (runOnceForEachItem)	Unwrap AI Agent output wrapper; build { batch_id, extracted: {...} }. The file is NOT in the payload — PB has it on the batch row.
Send Callback	httpRequest (onError: continueErrorOutput)	POST /api/purchase-orders/process-batch with x-ingest-secret header
Escalate Failure	code (runOnceForEachItem)	On Send Callback error: PATCH batch to failed with next_retry_at=+1min (or +5min after attempt 2); or dead after attempt 3. Loop continues.

Hardcoded values (no n8n-side env vars)

Self-hosted n8n on Coolify does not expose $env.* to workflows and has no Variables UI (paid feature only). Every value the workflow needs is baked into the workflow JSON itself. Search for these literals if you ever rotate them:

Where	Value	Notes
Auth PB Admin, Watchdog, Fetch Pending, Atomic Claim, Download File	https://pb.mega.infra.apphub.co.za/...	PB base URL
Auth PB Admin	jsonBody: {"identity":"admin@mega.co.za","password":"12345678"}	PB superuser creds — change both before going live
Watchdog: List Stuck filter	batch_type="purchase_orders" && status="processing" && last_attempt_at<"{{ $now.minus({minutes:15}).toISO() }}"	Stuck-row threshold is 15 min
Fetch Pending Batches filter	batch_type="purchase_orders" && status="pending" && (next_retry_at="" || next_retry_at<"{{ $now.toISO() }}")	Respects retry backoff
Send Callback	url: https://mega.dash.apphub.co.za/api/purchase-orders/process-batch	APP_URL hardcoded
Send Callback	x-ingest-secret: b8dffe7b00fca75272bf9a090d15a3c59fa0ec20d6bb232f7bb792db0921ac3b	Mirrors cooler.env N8N_INGEST_SECRET. Rotate both sides together.
Escalate Failure	Backoff constants 60000 (1 min) and 300000 (5 min) inline in JS	Adjust in the jsCode
AI Agent / AI Agent1	model: MiniMax-M3	Hardcoded model name in lmChatMinimax subnode

Atomic claim semantics

Atomic Claim uses PB's filter-on-PATCH trick:

PATCH /api/collections/data_import_batches/records/{id}?filter=status="pending"
Body: { "status": "processing", "attempts": "={{ $json.attempts + 1 }}", ... }

If the row was already claimed by another worker (status != pending), the filter rejects the PATCH and PB returns 0 records. Claim Successful? checks $json.status === "processing" — if not, the loop skips this batch. Two cron workers can never claim the same row.

Escalation semantics

Escalate Failure runs only on Send Callback error. It does not run if the AI Agent returned no items (that's a "nothing to do" condition that the process-batch route handles by transitioning the batch to dead itself).

• attempt 1 → status=failed, next_retry_at=now+1min
• attempt 2 → status=failed, next_retry_at=now+5min
• attempt 3 → status=dead, next_retry_at=null
• attempt ≥ 4 → still dead (idempotent)

The cron tick's Fetch Pending Batches filter respects next_retry_at, so a row with next_retry_at=...future is skipped until the backoff elapses.

Watchdog semantics

Watchdog runs in parallel with Fetch Pending on every tick. It picks up rows that have been stuck in processing for more than 15 minutes (e.g. crash mid-process, n8n restart, killed execution) and resets them to pending. Without this, a crashed row would block its attempts counter from advancing forever.

Why the file is NOT in the callback payload

The new /api/purchase-orders/process-batch route accepts just { batch_id, extracted: {...} }. The original file lives on the batch row in PB; the route fetches it from PB on success, copies it to the new purchase_orders row, and clears original_file from the batch. This keeps payloads small and avoids the binary-passthrough bugs that plagued the old /api/purchase-orders/ingest route.

PB filter syntax gotcha

PocketBase uses && and || for logical operators — AND and OR return 400. Both filters in this workflow use the &&/|| form. The = prefix is required on expression fields containing {{ ... }} (n8n only evaluates fields that explicitly opt in to templating).

Known cosmetic warnings

Two INVALID_PARAMETER warnings persist:

Field "parameters.queryParameters.parameters.1.value" has wrong type. Expected string, but got number.
Field "parameters.queryParameters.parameters.2.value" has wrong type. Expected string, but got number.

These fire on the perPage query parameter (which the schema expects as a string). n8n auto-coerces numeric values into the URL query string at HTTP time, so the warnings are harmless. Suppressing them would require switching to a string literal; not worth the loss of readability.

Future work

• Per-node retry — Send Callback currently uses onError: continueErrorOutput so the error branch can run Escalate Failure. The HTTP node itself doesn't retry. If you want to retry transient network errors within the same execution, set retryOnFail: true, maxTries: 3, waitBetweenTries: 5000 on Send Callback.
• Email ingestion — the callback contract is identical, so adding email attachments to the same queue requires no changes here.
• Webhook fallback — if cron latency (up to 5 min) is too long for some flows, add a webhook node that writes the batch row directly to PB and lets the cron pick it up.

Manual smoke test

The cron fires every 5 minutes — the empty-input path is verified (execution 55206 succeeded). For the happy path with a real file, run these steps from the deployed stack. Each step has an expected outcome you can check.

Step 1 — Confirm the cron is alive

Open the workflow. The last execution timestamp at top-right should advance within 5 min of the previous one. If it doesn't, check Every 5 minutes is enabled and the workflow is Published.

Step 2 — Upload a real PO via the dashboard

1. Visit /dashboard/purchasing on the deployed app.
2. Click Upload Purchase Order and pick a real PDF PO that has at least one matching site in the sites collection.
3. Expect a 202 + { batch_id, status: "pending" } toast within a couple seconds. Copy the batch_id.

Step 3 — Watch the cron pick it up

Wait up to 5 min. In the n8n workflow's Executions tab, a new run should appear with the following node trace:

Every 5 minutes → Auth PB Admin → Fetch Pending Batches → Loop Over Items
  → Skip Empty Items → Has ID? → Atomic Claim → Claim Successful?
  → Download File → Route by Content Type → AI Agent → Build Callback Payload
  → Send Callback

Step 4 — Verify the batch row transitioned

In the PB admin UI for data_import_batches, find the row by batch_id. Expect:

Field	Value
status	processed
attempts	1
processed_at	ISO timestamp from the run
processed_record_id	new purchase_orders row id
last_error	empty
original_file	empty (file moved off the batch)

A matching purchase_orders row should exist with original_file pointing to the same filename, and the PO header + line items populated.

Step 5 — Verify dedup

Upload the same PO file again. Expect either:

• The upload route returns 409 with the existing purchase_orders.id, or
• A new batch row is created but the AI Agent extracts the same po_number, and process-batch either links to the existing PO or rejects with duplicate.

(Exact behavior is set in Task 7 — check the route's source if unsure.)

Step 6 — Verify dead-state on unmatched site

Upload a PO whose site name doesn't match any row in the sites collection. After the cron runs, expect:

• status = dead
• attempts = 3
• next_retry_at = null
• last_error mentions "site not matched" (or similar)
• No purchase_orders row created

Step 7 — Verify retry UI

In /dashboard/purchasing or wherever the batch list lives, find the dead row from Step 6. Expect a Retry button. Clicking it sets status=pending and attempts=0 (or whatever the route contract is — see the retry route source).

Step 8 — Verify manual retry runs the cron

Within 5 min of clicking Retry, the row should re-enter processing and either transition to processed or back to failed/dead per the outcome. Check the Executions tab for a new run that includes the batch id.

Step 9 — Cleanup

Delete any test purchase_orders rows created during the smoke test. Delete any test data_import_batches rows that don't have status=processed and a real processed_record_id.

---

If any step fails, check the Executions tab in n8n first — the node trace shows exactly which step died. The most common failure modes:

• AI Agent timeout → row goes failed → next_retry_at=+1min; the next cron tick re-attempts. If it keeps timing out, check the model name in the lmChatMinimax subnode.
• Send Callback 4xx → row goes dead after 3 attempts. Check x-ingest-secret matches cooler.env N8N_INGEST_SECRET, and that /api/purchase-orders/process-batch is reachable from n8n.
• Send Callback 5xx → row goes failed → re-attempted. Check the Next.js logs for the actual exception.