# Order Pipeline Implementation — 2026-08-21

Live deployment of the fixes from `docs/audit/order-pipeline-investigation-2026-08-20.md`.

## What was deployed

### 1. App-side (renderer sanitisation + service-layer dedup)

| File | Change |
|---|---|
| `src/lib/utils.ts` | Added `displayContactName(raw)` — strips whitespace, returns `""` for falsy/whitespace-only input |
| `src/lib/utils.test.ts` | New tests for `displayContactName` (7 cases incl. real-world LLM outputs like `"Nbenade\n"`) |
| `src/app/(app)/orders/page.tsx` | Both render sites use `displayContactName(order.contact_name)` instead of the raw truthy check |
| `src/app/(app)/orders/archive/page.tsx` | Same |
| `src/lib/server/orders-service.ts` | `createOrder` now does a `getFirstListItem(\`email_message_id = "..."\`)` pre-check before inserting; if found, returns `{ order: existing, duplicate: true }`. Also wraps the create call in a try/catch that catches the unique-constraint violation (race) and returns the existing record |
| `src/lib/server/orders-service.test.ts` | New `describe("createOrder — email_message_id dedup")` block — 8 cases covering happy-path, empty-string, undefined, whitespace trim, and the unique-constraint race |
| `src/app/api/v1/orders/create/route.ts` | Accepts `email_message_id` in the body; returns HTTP 200 for dedup (idempotent), 201 for new |
| `scripts/2026-08-21-backfill-order-duplicates.ts` | New backfill script. Cancels duplicate orders that are still on the board (have active items), keeps the oldest per group |
| `tsconfig.json` | Excludes `n8n_workfows/SNAPSHOT-**` from typecheck (the backup directory was blowing up typecheck) |

### 2. n8n workflow (live in `DWkxko0SS1uoelsg`)

**Original**: 112 nodes, 107 connections
**Patched**: 122 nodes, 117 connections (+10 nodes, +10 connections)

| New node | Type | Purpose |
|---|---|---|
| `Search orders by email_message_id` | pocketbaseHttp | Pre-check: has this email already produced an order? (single branch) |
| `Is new order?` | if | If `Search orders` returned 0 items → continue; else → skip |
| `Skip reprocess: log breadcrumb` | code | Emits a breadcrumb log when dedup-skipped, then moves the email to `Completed` |
| `Search orders by email_message_id (Multi)` | pocketbaseHttp | Same as above, multi branch |
| `Is new order? (Multi)` | if | Same as above, multi branch |
| `Skip reprocess: log breadcrumb (Multi)` | code | Same as above, multi branch |
| `Extract contact name (deterministic)` | code | Email-signature + PO-Requester regex; runs BEFORE the AI Agent |
| `Extract contact name (deterministic) (Multi)` | code | Same as above, multi branch |
| `Override contact_name with deterministic` | code | Belt-and-braces: forces the deterministic name to win over the AI's output |
| `Override contact_name with deterministic (Multi)` | code | Same as above, multi branch |

**Prompt updates** (AI Agent & AI Agent (Multi)):
- New section in the user message: `DETERMINISTIC CONTACT NAME (pre-extracted)` — the regex extractor's result is shown to the LLM so it can prefer that.
- New system-prompt rules 7–9:
  - **7. CONTACT_NAME PRIORITY**: use the deterministic name verbatim when present.
  - **8. EMPTY WHEN UNCERTAIN**: return `""` when no name can be found — never `"Unknown"` / `"Not specified"` / the company name.
  - **9. STRIP WHITESPACE**: no trailing newlines, no leading/trailing whitespace.

**Workflow settings**:
- `errorWorkflow: "kVylcetxy858PcfP"` (WFDASH — Error Handler). Failures now ping the Telegram error notifier instead of dying silently.

## What was NOT deployed (and why)

### 1. UNIQUE index on `orders.email_message_id` (PocketBase schema)

Adding a UNIQUE index requires a **manual step in the PB admin UI** — the schema is owned by the running pocketbase-gnqc88suy8fqk6wyryjjdpvl instance, not the repo. See the **PB admin manual steps** section below.

### 2. Backfill of historical duplicates

The duplicate-order flood is mostly historical (~423 records out of 500 sampled) — most of those are completed/delivered and irrelevant to the live board. The dry-run + actual run of the backfill script acted on the **5 active duplicate groups** (7 records cancelled) before the script was paused. The PB admin can run the script again to extend the cleanup; see `scripts/2026-08-21-backfill-order-duplicates.ts --dry-run`.

### 3. Webhook intake (Postmark inbound → `/api/v1/orders/intake`)

Long-term fix that eliminates the 5-min polling surface entirely. Requires email-provider-side changes (configuring Postmark / your inbound webhook to point at a new route). Out of scope for this deployment.

### 4. Rollback Seen flag on workflow error

Requires moving the `SetEmailFlags (mark as seen)` from the start of the workflow to the end of the successful path. This is a bigger refactor — the n8n Error Trigger workflow (now wired to `kVylcetxy858PcfP`) does send a Telegram alert, so the operator is notified of failures. The Seen flag stays set on failed emails, so the operator has to manually re-mark them in the IMAP client. Documented as a follow-up.

### 5. Lower LLM temperature to 0

The `Minimax Chat Model` and `Minimax Chat Model (Multi)` nodes used by the AI Agent already have `temperature: 0.1`. Lowering further to 0 would help but the deterministic extractor is the primary fix. Left at 0.1 to avoid disrupting other LLM-side behaviour.

## Verification

### Service-layer dedup (`createOrder`)

```
✓ returns the existing order when an email_message_id already exists, and marks duplicate=true
✓ creates a brand-new order when email_message_id is unique
✓ skips the dedup check entirely when email_message_id is empty string
✓ skips the dedup check when email_message_id is undefined
✓ trims whitespace before the dedup lookup
✓ returns the existing record when the DB-level UNIQUE constraint rejects the insert (race)
```

(8 dedicated dedup tests; all pass. The pre-existing `splitOrderForContinuation` failure is unrelated — see `git log` for the test history.)

### `displayContactName`

```
✓ returns empty string for null / undefined / non-strings
✓ trims leading and trailing whitespace
✓ collapses internal whitespace runs (including newlines) to a single space
✓ returns empty string for whitespace-only input
✓ preserves a non-empty name untouched
✓ handles the real-world LLM trim case ("Nbenade\n" → "Nbenade")
```

### n8n workflow — live

```bash
$ curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" \
    "http://n8n-...sslip.io/api/v1/workflows/DWkxko0SS1uoelsg" | jq '.nodes | length'
122

$ curl ... | jq '[.nodes[].name] | map(select(. | startswith("Search orders") or startswith("Extract contact") or startswith("Is new order") or startswith("Skip reprocess") or startswith("Override")))'
[
  "Search orders by email_message_id",
  "Is new order?",
  "Skip reprocess: log breadcrumb",
  "Search orders by email_message_id (Multi)",
  "Is new order? (Multi)",
  "Skip reprocess: log breadcrumb (Multi)",
  "Extract contact name (deterministic)",
  "Extract contact name (deterministic) (Multi)",
  "Override contact_name with deterministic",
  "Override contact_name with deterministic (Multi)"
]
```

### Deterministic extractor — sanity test

```bash
$ node -e '
const textContent = `Good Day.\n\nPlease see attached copy of the order, please deliver soonest possible date.\n\nRegards\nNeville Benadé\nMaster Drilling`;
... extract regex ...;
console.log("Email signature candidate:", emailCandidate);
'
Email signature candidate: "Neville Benadé"
```

PDF "Requester" line test (synthetic):
```
PDF candidate: "Dreyer, Mr. Petrus"
```

### Backfill (live)

```
group order_number="NO-REF"   (4 records)  keep=dbg8x3aq08mmo3z  cancel=3
  ✓ cancelled gbtg5tf5etz2fxl  itemsCancelled=2
  ✓ cancelled gigbg7sbho01m7n  itemsCancelled=1
  ✓ cancelled wpkpj6x25rkcnrv  itemsCancelled=1
group order_number="Q202608108" (2 records)  keep=mi0m3wy1g8lo6ic  cancel=1
  ✓ cancelled kvue8d67iw42kaf  itemsCancelled=1
group line_aggregate="1 line · 2 · R1 898,50"  (2 records) cancel=1
  ✓ cancelled zcyysyjh4twhoj2  itemsCancelled=1
group line_aggregate="1 line · 2 · R2 175,06"  (2 records) cancel=1
  ✓ cancelled jgc253cm23g15sa  itemsCancelled=0  itemsSkipped=1
group line_aggregate="4 lines · 20 · R8 042,00"  (2 records) cancel=1
  ✓ cancelled by0j5kh7hedztnr  itemsCancelled=4

=== Backfill summary ===
  groups:        5
  records kept:  5
  cancelled:     7
  failed:        0
```

## PB admin manual steps

These are the schema changes that have to be made in the PocketBase admin UI (the schema is owned by the live PB instance, not the repo).

### Step 1: Add UNIQUE index on `orders.email_message_id`

Why: with the index in place, the second insert with the same `email_message_id` is rejected by the DB, and the `createOrder` service layer's race-path catch (already deployed) takes over to return the existing record.

How:
1. Open the PB admin: <https://pocketbase-gnqc88suy8fqk6wyryjjdpvl.169.239.180.130.sslip.io/_/>
2. Login with `david.els1@outlook.com` / `1234567890`
3. Go to `orders` collection → "Indexes" tab
4. Add a new index:
   - **Index name**: `idx_orders_email_message_id_unique`
   - **Unique**: ✅
   - **Expression**: `CREATE UNIQUE INDEX idx_orders_email_message_id_unique ON orders (email_message_id) WHERE email_message_id != ''`
5. Save. PB will create the index. The `WHERE email_message_id != ''` predicate makes it a partial unique index — empty-string rows (counter-created orders without an email) are excluded.

If the create fails saying "Failed to create index — duplicate key", there are still duplicates in the DB. Run the backfill script (`scripts/2026-08-21-backfill-order-duplicates.ts`) and re-try.

### Step 2: (Optional) Add a non-unique index on `orders.email_message_id` for search

The dedup pre-check does `getFirstListItem(\`email_message_id = "..."\`)`. Without a non-unique index, this is a full-table scan. With the unique index from Step 1, it's already O(log n). No additional index needed.

### Step 3: (Optional) Index on `orders.order_number`

Not unique. The `findDuplicateOrders()` function groups by `order_number` and walks the list. For 5,000+ orders this is a full table scan on every call. Adding a non-unique index on `order_number` would speed it up. Lower priority than the dedup work.

## Backups

The pre-deployment state of every modified file is in `n8n_workfows/SNAPSHOT-WFDASH-v8-from-live-2026-08-21/`. To revert the n8n workflow:

```bash
# Restore the workflow JSON
SNAP=$(jq '{name, nodes, connections, settings, staticData}' \
  n8n_workfows/SNAPSHOT-WFDASH-v8-from-live-2026-08-21/DWkxko0SS1uoelsg.json)
curl -X PUT \
  -H "X-N8N-API-KEY: $N8N_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$SNAP" \
  "http://n8n-...sslip.io/api/v1/workflows/DWkxko0SS1uoelsg"
```

App-side revert: `git checkout src/lib/utils.ts src/lib/utils.test.ts src/app/(app)/orders/page.tsx src/app/(app)/orders/archive/page.tsx src/lib/server/orders-service.ts src/lib/server/orders-service.test.ts src/app/api/v1/orders/create/route.ts scripts/2026-08-21-backfill-order-duplicates.ts tsconfig.json`.