WFDASH-WEBAPP/docs/audit/order-pipeline-investigation-2026-08-20.md

# Order Pipeline Investigation — 2026-08-20

Scope: investigate (1) hit-and-miss `contact_name` population on orders,
(2) duplicate-order creation when the same RFQ email arrives more than
once, and (3) general email-intake failures. Audit covers the n8n
workflows, the WFDASH Next.js app, and the live PocketBase data.

## TL;DR

| Symptom | Root cause | Severity |
|---|---|---|
| 84.6% of auto-created orders are duplicates of the same email | The IMAP poll keeps returning the same message every 5 minutes; the workflow has no `email_message_id` dedup check before `pb.collection('orders').create(...)` | **Critical** |
| `contact_name` is "hit and miss" (sometimes "Neville Benadé", sometimes "Nbenade\n", sometimes `\n`) | The contact name is sourced from a non-deterministic LLM call (no temperature cap, no fallback to deterministic email-signature parsing); the same email produces a different name on every poll cycle | **High** |
| The 9 most recent orders (2026-08-19) have empty `email_message_id` / `email_subject` | Those are counter-created orders — the post-create email field is not set on the manual `/counter` form. Not a bug, but it makes the "duplicate by msgid" dedup path useless for counter orders | **Low** |
| "Some emails fail" — orders created with `NO-REF`, `[NEEDS REVIEW]`, or `description = '[NEEDS REVIEW] AUTO-FLAGGED: ...'` | The AI Agent returns `order_items = []` whenever the PDF text extraction can't find a clean line-items table. The workflow then takes the "no items extracted" branch which writes a flag and a raw blob, but the *customer* / `contact_name` / `order_number` come from the same non-deterministic AI call so they may still be wrong | **Medium** |

The dedup mechanism that already exists (`POST /api/v1/orders/duplicates/cancel`) is **reactive** — it finds duplicates after the fact and lets the operator cancel them. Nothing prevents the duplicates from being created in the first place.

---

## 1. Pipeline overview

### Active order-related n8n workflows (as of 2026-08-19 23:46 UTC)

| ID | Name | Trigger | Purpose |
|---|---|---|---|
| `DWkxko0SS1uoelsg` | **WFDASH — ORDER MONITOR (Unified)** | Schedule (5 min) | The only pipeline that creates `orders` records. Polls IMAP, classifies, extracts via AI Agent, creates order + items. |
| `DWkxko0SS1uoelsg` (multi branch) | (same workflow) | (same) | The `(Multi)` branch handles multi-PO emails by splitting attachments by PO number detected in subject/body. |
| `O0yDcNVHEhA0Dd5n` | WFDASH — Multi-Order Fulfillment Pipeline | (other) | Creates `quotes` (not orders) for multi-PO emails via `/api/v1/quotes/from-email`. |
| `45h7IYegCmhtkV9t` | WFDASH — Multi-Order RFQ Fulfillment Pipeline | Schedule (2 min) | Creates `quotes` (not orders) for multi-PO emails via `/api/v1/quotes/from-email`. |
| `mXX2yu5svRurnfwO` | WFDASH - CATEGORISE ORDERS ITEMS | Schedule (1 min) | Backfills `ai_item_insights` and category enums. Does **not** create orders. |
| `paqlZICDbXbmY61V` | WFDASH — Populate Item Intel (Live) | Schedule | Item intel backfill. |
| `UqO3ayETx9cQHOzF` | WFDASH — Purchase Order Send | Webhook | Sends PO emails. |
| `GGS5zgVbdfsRyLcY` | ORDER-MARKUP-APPROVAL-LINKS | Webhook | WhatsApp/email markup-approval links. |
| `kVylcetxy858PcfP` | WFDASH — Error Handler | Error trigger | Telegram error alerts. |
| `aI0Ze2vwgp25ZgGH` | WFDASH — Error Notifier (Telegram) | Error trigger | Telegram error alerts. |
| `DDFN9NBmh95yG2Jk` | WFDASH — AI Insights Generator (v2) | Webhook | Re-runs AI insights on demand. |

The ORDER MONITOR is the only thing that creates `orders` records from email. Its flow is:

```
Schedule Trigger (every 5 min)
  -> CHECK FOR NEW MAIL              (IMAP read, seen:false)
  -> SetEmailFlags                   (mark as seen immediately)
  -> Classify Email (Deterministic)  (bounce / supplier / multi_order / customer / uncertain)
  -> Route Email (Switch)            (route to one of 5 branches)
  -> Limit                           (limit to 1 — but parameters: {}, so probably a no-op)
  -> Download ATTACHMENT
  -> Split Attachments1
  -> Switch1                         (by MIME type -> PDF / CSV / XLS / XLSX / TXT / image)
  -> Extract PDF / CSV / XLS / XLSX / TXT (or Analyze image for image attachments)
  -> Merge2
  -> Code in JavaScript4             (combine all extracted text)
  -> AI Agent                        (LLM extraction -> JSON { order_info, order_items })
  -> If1                             (if order_items.length > 0 -> single-order branch)
  -> Search customers                (PB getList by name/email fuzzy)
  -> Create orders                   (PB create on orders collection)
  -> Code in JavaScript2             (split order_items -> order_items rows)
  -> Create order_items1             (PB create per item)
  -> Aggregate
  -> Merge1 + PocketBase1 (none) + MoveEmail email2 (move to INBOX.202608.Completed)
```

The single-order and multi-order branches converge at `Merge order completion` and
finally move the email to `INBOX.202608.Completed` (or `Failed` on the failure
branch).

### PocketBase `orders` schema (live — `pbc_3527180448`)

30 fields, **0 indexes**, **1 unique constraint** (the implicit primary key).
Relevant fields:

- `order_number` (text, required) — no unique constraint
- `status` (select: `new` / `processing` / `processed` / `cancelled`)
- `company` (relation, required)
- `customer` (relation, optional)
- `contact_name` (text, optional) — written by the `Create orders` n8n node from the AI agent's `order_info.contact_name`
- `email_subject`, `email_body`, `email_sender`, `email_message_id` (text, optional)
- `rfq_number`, `description`, `notes`, `rep` (text, optional)

The n8n `Create orders` node writes `email_message_id = $('Limit').item.json.headers['message-id']` — but the field is **read from the email header**, not from the actual Message-ID envelope of the IMAP fetch. If the IMAP proxy / email provider rewrites the header, this might come back empty or differ from what `emailMessageId` exposes elsewhere.

---

## 2. The duplicate-order problem

### What the data shows

500 most-recent auto-created orders queried on 2026-08-19 23:46 UTC:

- **423 / 500 (84.6%)** are duplicates — i.e. the same `email_message_id` appears on 2 or more `orders` records
- **50 / 127 distinct message_ids** are duplicated (the rest are singletons)
- The worst case: a single email `<GV4PR05MB12159B25B35D2BD7FA50838D4D1A62@...>` produced **138 duplicate orders** — one every 5 minutes from 15:31 to 17:51 UTC on 2026-08-18, all with `order_number = ZA02PO26002142`
- Several emails have **9–10 duplicates created in the same second** (e.g. `MSP17933` at 19:10:27–19:10:28, `MSP17934` at 19:15:31). The same-second pattern means multiple workflow executions are processing the same email concurrently — the IMAP poll's `seen:false` filter is not preventing parallel runs from picking it up

### Why it's happening

Looking at the ORDER MONITOR (`DWkxko0SS1uoelsg`):

1. **Schedule Trigger** fires every 5 minutes (`{"field":"minutes"}` with no
   value, n8n defaults to 5min — confirmed by the 5-min spacing in execution
   log `56337 (23:45)` -> `56320 (23:40)` -> `56301 (23:35)`).
2. **CHECK FOR NEW MAIL** (IMAP-enhanced node) — fetches every email with
   `seen: false` from `INBOX`. Include parts `textContent, htmlContent,
   markdownContent, attachmentsInfo, flags, size, bodyStructure, headers`.
3. **SetEmailFlags (mark on fetch)** — marks the fetched email as `Seen`
   immediately. This is the ONLY thing preventing the same email from being
   picked up again on the next 5-min tick.
4. **Limit** — parameters `{}`. Confirmed by inspecting the workflow JSON.
   In n8n the Limit node with no `maxItems` defaults to 1, but the
   "merge-2" + multi-attachment paths also feed back into it, so this limit
   is unreliable across branches.
5. **Create orders** — the `n8n-nodes-pocketbase.pocketbaseHttp` node does
   `pb.collection('orders').create({...})` with no pre-check for an
   existing record with the same `email_message_id`.

Three things have to be true for duplicates to happen, and they all are:

- **A.** The IMAP poll keeps returning the same email. It happens because:
  the `seen:false` flag is set in the SAME workflow execution that processes
  the email — but if the workflow is retried (e.g. n8n re-triggers it after a
  transient error), or if the user reloads the IMAP mailbox and the IMAP
  server returns the same UID before the seen flag is committed, the next
  poll picks it up again. Same-second duplicates (9 in `19:10:27–19:10:28`)
  mean the workflow is running in parallel for the same email.
- **B.** There is no server-side constraint preventing it. The PocketBase
  `orders` collection has **no unique index** on `email_message_id` and
  no application-level pre-check in `createOrder` (`src/lib/server/orders-service.ts:1828`).
  `pb.collection('orders').create({...})` will happily insert a duplicate.
- **C.** The 5-min polling cadence multiplies the damage: every 5 minutes
  the same email gets reprocessed, and every cycle the AI extracts a
  slightly different `contact_name` (see §3) and a slightly different
  `order_number` if the AI misses the PO number on that particular call
  (the duplicate `NO-REF` records are the AI failing to extract the PO).

### Mitigations That Already Exist (Reactive Only)

- **POST /api/v1/orders/duplicates** (`src/app/api/v1/orders/duplicates/route.ts`)
  — `findDuplicateOrders()` groups by `order_number` (within 60s) and by a
  line-aggregate heuristic. Returns the grouped list. The UI
  (`src/components/duplicate-orders-dialog.tsx`) lets the operator bulk-cancel.
- **POST /api/v1/orders/{id}/cancel** — soft-cancels an order (all items
  flip to `cancelled`, record stays in DB for audit).

These are **post-creation** tools. They do not prevent the duplicates from
appearing in the first place.

### Fixes (proposed)

**A. Add a unique constraint on `email_message_id` in PB.** This is the
most robust fix — it makes the database reject duplicate inserts. Since the
field is currently `required: false`, the migration is:

1. In the PB admin: index `CREATE UNIQUE INDEX idx_orders_email_message_id
   ON orders (email_message_id) WHERE email_message_id != ''` (or a
   partial unique index — Postgres/SQLite both support it).
2. Backfill: cancel the existing 423 duplicates (or use the existing
   `cancelOrdersBulk` op) so the unique constraint can be created.
3. In the ORDER MONITOR workflow, **before** `Create orders`, add a
   `Search orders (email_message_id = X)` step. If a record exists,
   short-circuit and move the email to `INBOX.202608.Completed` with a
   `replay_skipped` flag in the post-create log. This makes the workflow
   idempotent even if the IMAP poll does fire on the same email.

**B. In the app, harden `createOrder` to refuse duplicates.** With a
unique constraint in place, the second `create()` will throw a 409 — wrap
the `pb.collection('orders').create(...)` call in
`src/lib/server/orders-service.ts:1828` to catch the constraint violation
and treat it as a "duplicate email, skipped" success. The function should
return the existing record.

**C. Tighten the IMAP poll.** Replace the `seen:false` filter with an
explicit `since` cursor: store the last-fetched UID in the workflow's
`staticData` and only fetch `UID > lastUID`. This works even if the seen
flag is reset by a reconnection.

**D. Move the email ingestion off the 5-min poll.** The poll is the only
reason duplicates get 5-min apart. A webhook-based trigger (e.g.
Postmark inbound webhook -> Next.js `/api/v1/orders/intake`) would
eliminate the polling surface entirely. The CREATE QUOTE FROM EMAIL
endpoint (`POST /api/v1/quotes/from-email`) already accepts the same
payload, so the work is mostly plumbing.

---

## 3. The `contact_name` hit-and-miss problem

### What the data shows

Same email `<GV4PR05MB12159B25B35D2BD7FA50838D4D1A62@...>` (ZA02PO26002142
from Nbenade@masterdrilling.com) produced these `contact_name` values across
its 138 duplicate orders:

```
Nbenade\n
Nbenade\n
N. Benade\n
Nicole Benade\n
Nbenade\n
Not specified\n
Nbenade\n
Drilling Technical Services (PTY) Ltd\n
Nbenade\n
Nico Benade\n
N. Benade\n
Nbenade\n
Nbenade\n
```

Plus empty-string `""` records (`~8% of the same-email sample`). Some
records have literally `"\n"` as the contact name — the AI returned JSON
like `{"contact_name": "\n"}` and it was stored verbatim.

The relevant emails are simple:

```
Good Day.

Please see attached copy of the order, please deliver soonest possible date.

Regards
```

There is no signature in the email body for the AI to extract from. The
PO attachment is what contains the actual name (Neville Benadé), and the
AI Agent prompt requires it to look at the attachments first — but it's
not deterministic about what it picks out.

### Why it's "hit and miss"

The contact_name pipeline:

1. **AI Agent** (`DWkxko0SS1uoelsg` "AI Agent" node) — Anthropic LLM call
   with `hasOutputParser: true`. The system prompt:
   - "Extract the actual End-User or Requester of the goods (Look for
     fields like 'Requester', 'Deliver To', 'Attention', or names in the
     Notes). Do NOT use the 'Buyer' or 'Created By' name if a Requester is
     listed."
   - There is **no `temperature` override** — Claude's default is 1.0.
   - The LLM is non-deterministic by design and the prompt does not
     enforce a JSON retry on invalid output.
2. **Search customers** (PB getList) — builds a filter from the LLM's
   `contact_name`:
   ```js
   const nameKeywords = name
     .replace(/mr\.|mrs\.|ms\.|mr|mrs|ms/gi, '')
     .split(/[\s,]+/)
     .filter(w => w.length > 2);
   ```
   - `replace(/mr\.|.../gi, '')` only strips the literal prefix
     "Mr./Mrs./Ms." — it doesn't strip "Mr " (no dot) or "N." (no Mr.)
   - `.filter(w => w.length > 2)` drops keywords of length ≤ 2. So
     "N. Benade" -> ["Benade"], "Nbenade" -> ["Nbenade"], "Neville Benadé"
     -> ["Neville", "Benadé"].
   - The `customer ~ 'X'` substring filter then matches the customer
     collection. If the AI returned "Nbenade" and the customer is
     "Neville Benadé", the substring match **succeeds** because "Nbenade"
     is a substring of no one — but the search is fuzzy enough to return
     multiple matches (the `elementsPerPage: 12` returns up to 12).
3. **Create orders** — `customer = $('Search customers').item.json?.id ||
   'xmd68ps32k1x5fu'` — the first match wins. If the AI returned names
   that don't match any customer, the fallback ID is the placeholder
   `xmd68ps32k1x5fu` (which is "Unknown - needs review" in the customers
   table).
4. **Page rendering** — `src/app/(app)/orders/page.tsx:942`:
   ```tsx
   {order.contact_name && (
     <span>{order.contact_name}</span>
   )}
   ```
   Empty string `""` is falsy -> row shows no name. Newline `"\n"` is
   truthy -> row shows just a separator dot with nothing after it. The
   operator sees "GARY-JV-MR · Master Drilling · ·" with a dangling
   separator and no name — clearly "miss".

### Fixes (proposed)

**A. Stop using the LLM for the contact name.** The email body has a
signature; the PDF has a "Requester:" line. Both are deterministic. Add a
pre-LLM regex pass:

1. Parse the email body for the signature (everything after the last
   `Regards` / `Kind regards` / `Best` / `Thanks` block, or the last 5
   non-empty lines).
2. Parse the PDF text for `Requester:`, `Deliver To:`, `Attention:`,
   `End-User:`. The first one is the contact name.
3. Only call the LLM if both fail.

If a deterministic parse succeeds, the LLM is skipped entirely — the
hit/miss disappears.

**B. If you keep the LLM, constrain it.** Add to the system prompt:
- "If you cannot find a Requester/Deliver-To/Attention name, return the
  empty string — do NOT return placeholder text like 'Unknown', 'Not
  specified', or the company name."
- Emit `contact_name = ""` (not `"\n"`) when uncertain. The page's
  truthy-check will hide it cleanly.
- Set `temperature: 0` on the Anthropic node. The LLM is still
  non-deterministic for the same prompt + temperature, but the variance
  drops substantially.

**C. Post-process the LLM output.** Wrap the AI Agent call in a Code
node that:
- trims trailing newlines (`s.replace(/\s+$/,'')`)
- validates the result against the customers collection (search by
  email first, then by name) — if a `customer` row exists with the same
  `email`, use `customer.customer` as the contact name regardless of what
  the LLM said.

**D. Move the display sanitisation into the renderer.** Even if the LLM
keeps hallucinating newlines and trailing whitespace, the row should not
show a dangling separator. Patch `order.contact_name` to a JSX-safe
display:

```tsx
const displayContact = (order.contact_name || "").trim();
{displayContact && <span>· {displayContact}</span>}
```

This is a one-line fix and removes the visible "miss" symptom even if
the data is still messy.

---

## 4. The "some emails fail" question

Looking at the live data, two failure modes are visible:

### 4a. AI returns empty `contact_name` / `order_items`

The `If1` / `If1 (Multi)` branches check `$json.output.order_items.length`
and route:

- `<= 0` -> "no items extracted" branch -> `Create order (no items extracted - raw)` / `Create order (single - no items - raw)` / `Create order (no PO detected - raw)`. The resulting order has:
  - `description = "[NEEDS REVIEW] AUTO-FLAGGED: AI Agent could not extract any line items for this PO. Raw extracted text: ..."`
  - the raw extracted text dumped in the description
  - the email still moved to `INBOX.202608.Completed`

This is the right fallback — the operator sees the flag. But the customer
search may still be wrong because the LLM returned garbage for the contact
name (see §3). The order lands on the board with a flag, but the
customer/contact fields may be incorrect.

### 4b. AI returns "Unknown" / "Not specified" / company name as contact

Same root cause as §3. The LLM hallucinates an answer instead of returning
empty. The Search customers query then searches for "Unknown" or "Not
specified" or "Drilling Technical Services" — and finds the company
record instead of the customer record. The order is created with the
**company** as the customer, which is the wrong relation.

### 4c. The IMAP poll loses emails

When the IMAP mailbox is polled, the `seen:false` filter is set in the
SAME workflow execution as the processing. If the workflow crashes
(retry, timeout, network error, PocketBase 500), the email is still
marked as seen but never processed. The next 5-min poll won't pick it up.
This is silent failure — the email is in limbo until the operator manually
checks the inbox.

There is no `onError` feedback to the ORDER MONITOR. The "Error Handler"
n8n workflow (`kVylcetxy858PcfP`) only catches errors from workflows that
have the error workflow set. ORDER MONITOR does not.

### Fixes (proposed)

**A. Make the email flag roll back on error.** Wrap the `Create orders`
+ `Create order_items1` chain in a Rollback node: if either fails, set
`Seen = false` on the email so the next poll picks it up.

**B. Mark the workflow as error-to-Workflow.** In n8n workflow Settings,
set `errorWorkflow: kVylcetxy858PcfP` so the Telegram error notifier
fires on any failure.

**C. Add a sender-domain blacklist to the deterministic classifier.** The
classifier already has `supplierDomains = ['festo.com']` but that's
incomplete — most of the duplicates come from
`masterdrilling.com`, `mreee.co.za`, `harmony.co.za`. Add a `customer`
domain list (the inverse of supplier). The classifier already routes
"supplier" -> ignored. Mirror it with "customer" -> process, but make sure
the operator can override the list.

**D. Add a fallback "create order from raw email" path.** If the AI
Agent fails to extract structured data after N retries, the workflow
should still create an order with the raw email body as the description
and a flag. This is already the "no items extracted" branch — but verify
the branch is reachable for ALL routes (not just the customer branch).

---

## 5. Recommended fixes, ranked by impact / effort

| # | Fix | Impact | Effort |
|---|---|---|---|
| 1 | **Add UNIQUE index on `orders.email_message_id`** in PB, backfill the 423 existing duplicates via the bulk-cancel tool, and pre-check in `createOrder` | Stops the duplicate flood at the DB level | 2 hours (admin UI + service-layer try/catch) |
| 2 | **Add a "Search orders by email_message_id" pre-check** in the ORDER MONITOR workflow before `Create orders` | Stops duplicates even if the IMAP poll is unreliable | 30 min in n8n |
| 3 | **Deterministic contact_name extraction** (regex signature + PO requester line) before the LLM call | Eliminates the "hit and miss" name pop | 1 day (depends on how many PO formats need handling) |
| 4 | **Constrain the LLM**: temperature 0, "return empty string when uncertain" rule | Reduces LLM hallucination | 15 min |
| 5 | **Renderer sanitisation**: trim `contact_name` and only render when non-empty | Hides the visible "miss" symptom with no data loss | 5 min |
| 6 | **Replace IMAP poll with a webhook intake** (Postmark inbound -> `/api/v1/orders/intake`) | Eliminates the 5-min polling surface entirely | 1 week (depends on the email provider) |
| 7 | **Wire the ORDER MONITOR to the error workflow** | Surfaces failures on Telegram instead of silently losing emails | 15 min |
| 8 | **Roll back Seen flag on workflow error** | Recovers emails that fail to process | 30 min in n8n |

If we only do one thing: **#1 + #2** together. They stop the bleeding. If
we do two more: **#3 + #5** clean up the visible UX. #6 is the long-term
fix that fixes all polling-class bugs at once.

---

## 6. Files / artefacts that were touched or referenced

- n8n workflows (live, active): `DWkxko0SS1uoelsg`, `O0yDcNVHEhA0Dd5n`, `45h7IYegCmhtkV9t`, `mXX2yu5svRurnfwO`, `UqO3ayETx9cQHOzF`, `GGS5zgVbdfsRyLcY`, `kVylcetxy858PcfP`, `aI0Ze2vwgp25ZgGH`, `DDFN9NBmh95yG2Jk`
- App sources: `src/app/(app)/orders/page.tsx`, `src/components/duplicate-orders-dialog.tsx`, `src/app/api/v1/orders/duplicates/route.ts`, `src/app/api/v1/orders/create/route.ts`, `src/lib/server/orders-service.ts` (createOrder, findDuplicateOrders)
- PocketBase collections: `orders` (pbc_3527180448), `customers` (pbc_108570809), `companies` (pbc_3866053794)
- Docs: `docs/order-workflow-rules.md`, `docs/AGENT_INSTRUCTIONS.md`

### Query commands used (reproducible)

```bash
# List all orders, look at duplicates
curl -H "Authorization: Bearer $AGENT_API_KEY" \
  "https://wfdash.apphub.co.za/api/v1/orders?perPage=500&sort=-created&fields=id,order_number,contact_name,email_message_id,email_sender,created,email_subject"

# Get a single n8n workflow's JSON
curl -H "X-N8N-API-KEY: $N8N_API_KEY" \
  "http://n8n-...sslip.io/api/v1/workflows/DWkxko0SS1uoelsg"

# Get a single execution's data
curl -H "X-N8N-API-KEY: $N8N_API_KEY" \
  "http://n8n-...sslip.io/api/v1/executions/56337?includeData=true"
```