Every 5 minutes the workflow polls `data_import_batches` for `batch_type=purchase_orders && status=pending` rows, atomically claims them by patching `status=processing && attempts+1`, downloads the original file from PocketBase, runs the AI Agent (MiniMax-M3 + Structured Output Parser) to extract the PO header and line items, and POSTs the structured payload to `/api/purchase-orders/process-batch` on the dashboard. The dashboard route owns the PB inserts and updates the batch row to `processed`/`failed`/`dead`.
- **Project:** Personal (David Els <david.els1@outlook.com>)
- **Credentials:** `MiniMax account` is auto-bound to all four `lmChatMinimax` chat-model subnodes (PDF Agent, Image Agent, Parser Model PDF, Parser Model Image).
| Watchdog: List Stuck | httpRequest (executeOnce) | GET batches where `status=processing` and `last_attempt_at < 15 min ago` |
| Watchdog: Reset Stuck Rows | code (runOnceForAllItems) | PATCH each stuck row back to `pending` with `last_error="watchdog_reset"` |
| Fetch Pending Batches | httpRequest (executeOnce) | GET batches where `status=pending` and (`next_retry_at=""` or `next_retry_at<now`), sorted by `date_uploaded`, max 10 |
| Loop Over Items | splitInBatches (batchSize=1) | Process one batch at a time |
| Skip Empty Items | code (runOnceForAllItems) | Filter the empty wrapper that splitInBatches emits when input has 0 items; replace with `{ skipped: true }` |
| Has ID? | if | Route items with `id` to Atomic Claim; terminate items without |
| Atomic Claim | httpRequest | PATCH `/records/{id}?filter=status="pending"` — atomic state transition to `processing` (race-safe) |
| Claim Successful? | if | `status=processing` → Download File; otherwise loop back |
| Download File | httpRequest (responseFormat=file, outputPropertyName=data) | GET `/files/data_import_batches/{id}/{original_file}` — file appears in `$binary.data` |
| Route by Content Type | switch | PDF branch → AI Agent; image branch → AI Agent1 |
| AI Agent / AI Agent1 | langchain agent + parser + MiniMax chat model | Extract PO header + line items per the JSON schema |
| Build Callback Payload | code (runOnceForEachItem) | Unwrap AI Agent `output` wrapper; build `{ batch_id, extracted: {...} }`. The file is NOT in the payload — PB has it on the batch row. |
| Send Callback | httpRequest (onError: continueErrorOutput) | POST `/api/purchase-orders/process-batch` with `x-ingest-secret` header |
| Escalate Failure | code (runOnceForEachItem) | On Send Callback error: PATCH batch to `failed` with `next_retry_at=+1min` (or `+5min` after attempt 2); or `dead` after attempt 3. Loop continues. |
Self-hosted n8n on Coolify does **not** expose `$env.*` to workflows and has **no Variables UI** (paid feature only). Every value the workflow needs is baked into the workflow JSON itself. Search for these literals if you ever rotate them:
If the row was already claimed by another worker (status != pending), the filter rejects the PATCH and PB returns 0 records. Claim Successful? checks `$json.status === "processing"` — if not, the loop skips this batch. Two cron workers can never claim the same row.
Escalate Failure runs only on Send Callback error. It does **not** run if the AI Agent returned no items (that's a "nothing to do" condition that the process-batch route handles by transitioning the batch to `dead` itself).
Watchdog runs in parallel with Fetch Pending on every tick. It picks up rows that have been stuck in `processing` for more than 15 minutes (e.g. crash mid-process, n8n restart, killed execution) and resets them to `pending`. Without this, a crashed row would block its `attempts` counter from advancing forever.
The new `/api/purchase-orders/process-batch` route accepts just `{ batch_id, extracted: {...} }`. The original file lives on the batch row in PB; the route fetches it from PB on success, copies it to the new `purchase_orders` row, and clears `original_file` from the batch. This keeps payloads small and avoids the binary-passthrough bugs that plagued the old `/api/purchase-orders/ingest` route.
PocketBase uses `&&` and `||` for logical operators — `AND` and `OR` return 400. Both filters in this workflow use the `&&`/`||` form. The `=` prefix is required on expression fields containing `{{ ... }}` (n8n only evaluates fields that explicitly opt in to templating).
These fire on the `perPage` query parameter (which the schema expects as a string). n8n auto-coerces numeric values into the URL query string at HTTP time, so the warnings are harmless. Suppressing them would require switching to a string literal; not worth the loss of readability.
- **Per-node retry** — Send Callback currently uses `onError: continueErrorOutput` so the error branch can run Escalate Failure. The HTTP node itself doesn't retry. If you want to retry transient network errors within the same execution, set `retryOnFail: true, maxTries: 3, waitBetweenTries: 5000` on Send Callback.
- **Email ingestion** — the callback contract is identical, so adding email attachments to the same queue requires no changes here.
- **Webhook fallback** — if cron latency (up to 5 min) is too long for some flows, add a webhook node that writes the batch row directly to PB and lets the cron pick it up.
