David/WFDASH-WEBAPP
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
WFDASH-WEBAPP/docs/quotes-email-monitor.md

418 lines
32 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Quote Email Monitor (n8n)
The n8n workflow `WFDASH — Quote Email Monitor (v3.1 agentic + PDF text)` is the inbound quote-request pipeline. It polls the `quotes@wfsales.co.za` mailbox every 5 minutes, runs AI extraction over each RFQ (including PDF text extraction from attachments), and creates structured `Quote` records in PocketBase via the WFDASH MCP endpoint. It also recognises replies to existing quotes and appends them as `quote_replies` rows.
A separate workflow `WFDASH — Error Handler` catches every failure in the main workflow and posts a Telegram alert so nothing falls into a black hole.
**One scheduler, one workflow, one source of truth.** No long-running process in the Next.js app, no racy counters, no fabricated cost prices, no fuzzy customer merges, no manual-review step.
> **v2 note:** as of v2 the workflow **no longer moves emails out of `INBOX`**. The n8n JS task-runner image (`n8nio/runners:2.25.7`) blocks `require('imapflow')` outright, and there is no env-var to allowlist new modules in that sandbox. Processed emails stay in `INBOX` UNSEEN; the next poll re-fires the workflow; WFDASH dedups the quote by `message_id` (or `synth-…@wfdash.local`). Re-fires are no-ops at the database level. A human (or a cPanel-side IMAP filter) handles inbox state.
> **v3.1 (this version):** replaces the v3 "Information Extractor → Code" pair with a **langchain AI Agent** (with two `toolCode` subnodes) so the LLM directly calls `quotes.create_from_email` and `attachments.upload_for_quote` per attachment. Adds a **Split Binary → Extract from File (PDF) → Aggregate** pipeline so PDF line items reach the LLM as text in the user message. Flips `passthroughBinaryImages: false` so PNG attachments survive to the upload tool (the v3 bug where `attachment_1` was eaten by LLM vision is fixed).
---
## Flow
```
IMAP poll (every 5 min, INBOX, UNSEEN only)
↓
Edit Fields ── injects mcp server + agent_api_key
↓
Parse Email ── normalises payload (handles from.value[n] arrays, Message-ID objects),
preserves binary as $binary
↓
Drop Blacklist / Auto-replies ── sender + subject filters
↓
Is Reply?
├── yes → MCP: quotes.append_reply
│ ↓
│ Parse MCP Reply Result
│ ↓
│ Route Reply Result (terminal)
└── no → Carry Email Ctx (preserves binary)
↓
Split Binary into Items (Code, runOnceForAllItems: emits N items,
one per attachment, key always "attachment")
↓
Extract from File (PDF) (n8n Extract from File v1.1, operation: pdf,
binaryPropertyName: "attachment",
onError: continueRegularOutput)
↓
Aggregate + Enrich (Code, runOnceForAllItems: groups back to one item
per email, restores binary by original key, adds
attachments[i].extracted_text to the JSON)
↓
AI Agent: Create Quote From Email
├─ languageModel: MiniMax Chat Model
└─ tools:
├─ create_quote_tool (calls quotes.create_from_email)
└─ upload_attachment_tool (calls attachments.upload_for_quote)
(terminal)
```
A reply is detected when **either** an `In-Reply-To` header is present **or** the subject matches `/Q-\d{4}-\d{2}-\d{3}/`. The `route: "reply" | "new"` field is computed in the Parse Email node and the If node is a simple string equality check.
On any unhandled error in the main workflow, n8n fires the Error Handler workflow automatically (via the workflow's `errorWorkflow` setting). The Error Trigger does **not** receive the original email payload directly — it gets the error context (subject, from_email, uid, mailbox) only because every Code node in the main workflow passes those through. **If you add a new branch, make sure to forward `subject`, `from_email`, `uid`, and `mailbox` so the Error Handler has context for its alert.**
---
## Why the Split → Extract → Aggregate pipeline (v3.1)
n8n's `Extract from File` v1.1 takes a **single binary per item** (one PDF, one output). The IMAP trigger produces **one item with N binary entries** (PDF + PNG + …). We can't point `binaryPropertyName` at both `attachment_0` and `attachment_1` at the same time.
The pattern is the canonical n8n one:
1. **Split Binary into Items** (Code, `runOnceForAllItems`) — emit N items, each with exactly one binary, always keyed `"attachment"`. The original key (`attachment_0`, `attachment_1`, …) is preserved on `$json.binary_key` for the aggregate step.
2. **Extract from File (PDF)** — `binaryPropertyName: "attachment"`, `operation: pdf`, `joinPages: true`, `onError: continueRegularOutput`. Non-PDFs (e.g. PNG banner) pass through with empty `$json.text`.
3. **Aggregate + Enrich** (Code, `runOnceForAllItems`) — group the N items back into one item per email, put the extracted text on `attachments[i].extracted_text`, and restore the binaries under their **original** keys (`attachment_0`, `attachment_1`, …) so the AI Agent's `upload_attachment_tool` can read them by `binary_key`.
The LLM sees the email body + a 2 KB preview of each PDF's extracted text in its user message. It uses that for line items. The full text is preserved on `$json.attachments[i].extracted_text` (up to 10 KB per attachment) if a follow-up tool needs more.
---
## Retry behaviour
The IMAP trigger is configured with `customEmailConfig: "[\"UNSEEN\"]"` and `postProcessAction: "nothing"`. Two consequences:
1. **Implicit retry on crash.** If the workflow dies mid-run, the email stays UNSEEN. The next 5-minute poll re-fires the workflow on the same message.
2. **No "mark as read" until success.** A failed email is left in `INBOX` until a human triages it. There is no silent loss.
**Important:** in v2+ a successful run also leaves the email in `INBOX` (UNSEEN). The success signal is the WFDASH database row (a new `quotes` row, or a new `quote_replies` row). The next poll re-fires on the same email, but WFDASH dedups by `message_id` and returns `deduped: true` with no side-effects. To keep `INBOX` tidy, a human (or a periodic cPanel IMAP filter) should move processed emails out of `INBOX` after each batch.
---
## Prerequisites
1. **PocketBase migrations applied** — `quote_attachments`, `quote_replies`, `quote_counters` collections must exist and `quotes.email_message_id` must be indexed. Run:
```bash
npm run pb:migrate:quote-attachments
```
Verify with `curl http://127.0.0.1:8090/api/collections | jq '.[] | select(.name | startswith("quote_"))'`.
2. **n8n credentials** — set up in the n8n vault (never in the Next.js env):
| Credential name | Type | Used by |
| --- | --- | --- |
| `IMAP account` | IMAP | IMAP Quote Mailbox trigger |
| `Bearer Auth account` | HTTP Bearer (`AGENT_API_KEY`) | Both MCP HTTP Request nodes |
| `Telegram Alerts` | Telegram API | Error Handler → Telegram Alert |
The auto-assigner wires up `IMAP account` (on the IMAP trigger). **You must manually create `Bearer Auth account` and `Telegram Alerts`** — n8n's auto-assigner skips HTTP Request and Telegram nodes.
The MiniMax chat model is wired via the langchain `lmChatMinimax` subnode and authenticates via its own per-credential setup; this is not the WFDASH bearer.
3. **n8n environment variables**:
| Name | Example | Purpose |
| --- | --- | --- |
| `WFDASH_MCP_URL` | `https://wfdash.example.com` | Base URL the workflow POSTs to. Falls back to `http://localhost:3000` if unset. The IMAP tool subnodes also accept the value via `Edit Fields` → `mcp server` for per-workflow override. |
| `TELEGRAM_CHAT_ID` | `-1001234567890` | The group/chat that receives alerts. Set in n8n's env (not in a credential). |
4. **Error Handler is linked.** Open the main workflow → **Settings** → **Error workflow** → pick `WFDASH — Error Handler`. (The SDK has no operation for this setting, so it's a one-time manual step after the first deploy.)
---
## IMAP folder layout
| Folder | What goes there | Populated by |
| --- | --- | --- |
| `INBOX` | Inbound queue — the trigger reads from here. Processed mail also stays here in v2+. | Inbound mail server (+ human / cPanel filter in v2+) |
The trigger reads only UNSEEN mail. There are no `INBOX.Quotes-*` sub-folders in v2+ — the n8n workflow does not create or move mail between folders.
---
## Nodes — main workflow
### 1. IMAP Quote Mailbox (trigger)
- `mailbox: "INBOX"`
- `postProcessAction: "nothing"` (don't mark as read; the WFDASH database row is the success signal)
- `downloadAttachments: true` (binary data flows into `$binary`)
- `format: "resolved"`
- `customEmailConfig: "[\"UNSEEN\"]"`
- `trackLastMessageId: true` (avoids re-firing on the same email on workflow restart)
### 2. Edit Fields (Set)
Injects two fields used by every downstream node:
- `mcp server` — the WFDASH MCP base URL. Defaults to `https://hzbtf-196-39-76-18.run.pinggy-free.link` (pinggy dev tunnel) in this build; override with the `WFDASH_MCP_URL` env or by editing the field.
- `agent_api_key` — the WFDASH `AGENT_API_KEY` bearer. Defaults to `123456` in this build; replace with the real key.
> **Security note:** the agent_api_key is a long-lived secret. Move it to a credential (`Bearer Auth account`) or an env var (`WFDASH_AGENT_API_KEY`) before this workflow leaves the dev box.
### 3. Parse Email (Code, `runOnceForEachItem`)
Coerces all the IMAP field shapes into stable strings. Specifically handles:
- `from` as a string, `{ value: [...] , name: "..." }`, `{ address: "..." }`, or `{ text: "Name <email>" }`. **Also handles the n8n-serialized array form** (`{ item: "x@y.com" }`).
- `messageId` / `inReplyTo` as a string, an object with `value: "..."`, or an array.
- Strips the `<` and `>` brackets around Message-IDs.
- Iterates `$binary` and produces an `attachments: [{ binary_key, filename, content_type, size }]` array.
- Emits a `route: "reply" | "new"` field used by the If node.
- **Returns `{json, binary: $binary || {}}` so binary survives the Code-node round-trip.** Without this, the next Code node (Carry Email Ctx) sees an empty `$binary` and the Split → Extract → Aggregate pipeline has nothing to operate on. (This was a v3 bug; fixed in v3.1.)
### 4. Drop Blacklist / Auto-replies (Filter)
7 conditions combined with `AND`:
- `from_email` does not contain `noreply`, `donotreply`, `mailer-daemon`, `postmaster`
- `subject` does not contain `out of office`, `automatic reply`, `vacation`
### 5. Is Reply? (If)
Single condition: `{{ $json.route }} equals "reply"`. String equality, not boolean.
### 6a. MCP: quotes.append_reply (HTTP Request, reply branch)
Posts JSON-RPC `tools/call` to `{{ WFDASH_MCP_URL }}/api/mcp`. Arguments mirror `EmailReplyPayload`: `from_email`, `from_name`, `subject`, `body_plain`, `body_html`, `message_id`, `in_reply_to`, `received_at`. Auth: HTTP Bearer (`Bearer Auth account`).
### 6b. Carry Email Ctx (Code, `runOnceForEachItem`) — new-quote branch
Explicitly returns `{json, binary: $binary || {}}`. This is the second "binary preservation" node — it catches the case where Parse Email's binary output (in production) gets reshaped by intermediate nodes and makes sure the downstream Split → Extract → Aggregate pipeline has the original binary.
### 6c. Split Binary into Items (Code, `runOnceForAllItems`)
Emits one item per attachment. Each item has:
- `json.binary_key` — the original key (`attachment_0`, `attachment_1`, …) so the aggregate step can restore the binary under the same key for the upload tool.
- `json.filename`, `json.content_type`, `json.size` — attachment metadata.
- `json._email_*` — a flat copy of all the email context fields (from_email, from_name, subject, body_*, message_id, in_reply_to, received_at, is_reply, route, uid, mailbox, mcp server, agent_api_key) so the aggregate step can rebuild the email.
- `binary.attachment` — the single binary for this attachment, always keyed `attachment` (the fixed key Extract from File expects).
**Why `runOnceForAllItems` (not `runOnceForEachItem`):** `runOnceForEachItem` validates the return shape as a single item. Returning an array of items in that mode triggers `"A 'json' property isn't an object [item 0]"`. `runOnceForAllItems` processes the (single) input item once and returns multiple output items correctly.
If there are no attachments, the node emits one placeholder item (`{json: {binary_key: null, ...}, binary: {}}`) so the chain continues to Aggregate (which handles 0 items safely).
### 6d. Extract from File (PDF) (n8n Extract from File v1.1)
- `operation: pdf`
- `binaryPropertyName: attachment`
- `options.joinPages: true`
- `onError: continueRegularOutput` — non-PDFs (PNG banner) pass through with empty `$json.text`; the workflow does not crash.
### 6e. Aggregate + Enrich (Code, `runOnceForAllItems`)
- Groups the N split items back into one item per email.
- For each attachment, puts the extracted PDF text (truncated to 10 KB) on `attachments[i].extracted_text`.
- Restores the original binary entries under their original keys (`attachment_0`, `attachment_1`, …) so `upload_attachment_tool` can read them by key.
- Rebuilds the top-level email fields (`from_email`, `subject`, etc.) and forwards `mcp server` + `agent_api_key` for the tool subnodes.
### 6f. AI Agent: Create Quote From Email (langchain agent v3.1)
- `promptType: define`
- `text` — user message composed of email metadata + a JSON list of attachments (each with `extracted_text_preview` = first 2 KB of the PDF text) + email body.
- `options.systemMessage` — see the next section.
- `options.maxIterations: 20` — enough for one create + N uploads + summary.
- `options.passthroughBinaryImages: false` — **the v3.1 fix**. In v3 this was `true`, which let the LLM "see" the PNG via vision but consumed the binary (the upload tool then reported `error: binary not found for key attachment_1`).
- `options.returnIntermediateSteps: false` — keeps the agent's final summary short; the audit trail lives in the per-tool calls.
### 6g. MiniMax Chat Model (subnode, `lmChatMinimax` v1)
- `model: MiniMax-M2.7`
- `options.temperature: 0.1`
- `options.timeout: 60s`
### 6h. create_quote_tool (subnode, `toolCode` v1.3)
Calls `quotes.create_from_email` via the WFDASH MCP. The LLM only produces `company_hint` and `extracted_items`; every other email field is read from `$input.first().json` (the Aggregate + Enrich output) verbatim. The LLM cannot paraphrase or drop required fields. Returns a one-line summary: `quote_id=… quote_number=… items=… deduped=…`.
### 6i. upload_attachment_tool (subnode, `toolCode` v1.3)
Calls `attachments.upload_for_quote` via the WFDASH MCP. Reads the binary from `$input.first().binary[binaryKey]`. **Belt-and-suspenders:** if upstream is empty, falls back to `$('IMAP Quote Mailbox').first().binary[binaryKey]` — the IMAP trigger's output is the source of truth for binary and never strips attachments. Converts the binary `data` to base64 and POSTs it. Returns: `uploaded=<filename> id=… kind=…`.
---
## AI Agent system prompt (verbatim)
> You are a purchase-order parser for WFDASH.
>
> You have two tools:
> - create_quote_tool: create a new quote row in PocketBase. Use it once per inbound email, after extracting line items.
> - upload_attachment_tool: upload a single attachment to an existing quote. Use it once per attachment in the email, AFTER create_quote_tool returns a quote_id.
>
> Workflow:
> 1. Read the email body (in your user message). For each attachment, also look at the extracted_text_preview field — that is the first 2KB of the PDF text. If the email body is short or empty, the line items are almost certainly in the PDF.
> 2. Extract the buyer/end-customer company name (company_hint). Empty string if unclear.
> 3. Extract the line items: an array of {description, quantity, unit?, notes?}. Look BOTH in the email body AND in extracted_text_preview of each attachment. DO NOT invent items, prices, or quantities.
> 4. Call create_quote_tool. Pass ALL the email metadata VERBATIM. Do NOT paraphrase.
> 5. Capture the quote_id from the tool result.
> 6. For each entry in the email's attachments array, call upload_attachment_tool with that entry's binary_key, filename, content_type, size, the captured quote_id, and the email's message_id. Call it once per attachment.
> 7. After all uploads, reply with a one-line summary: quote_number=... items=... uploaded=... failed=...
>
> Important:
> - Empty body / no items is OK — create the quote anyway with an empty extracted_items array.
> - attachments metadata only (no base64) — the binary lives in the workflow's binary store; only upload_attachment_tool knows how to read it.
---
## MCP tools this workflow calls
| Tool | Branch | Purpose |
| --- | --- | --- |
| `quotes.create_from_email` | new (via `create_quote_tool`) | Idempotent: dedupes on `email_message_id` (or synthesized `synth-…@wfdash.local`). Creates customer/company/quote/items. |
| `attachments.upload_for_quote` | new (via `upload_attachment_tool`) | Stores the file in the `quote_attachments` collection, linked to the quote_id. |
| `quotes.append_reply` | reply | Resolves the target quote (In-Reply-To first, then subject regex), appends a `quote_replies` row, flips `has_new_reply`. |
| `quotes.find_by_message_id` | (helper, not used in v3.1) | Look up a quote by the original `Message-ID`. Use this when the subject has been stripped by the mail client. |
The full MCP tool list and their schemas live in [api-mcp-layer.md](api-mcp-layer.md).
---
## Testing
Both workflows are pinned in n8n so you can dry-run with synthetic data without hitting IMAP or MCP.
**Main workflow pinned nodes:** `IMAP Quote Mailbox`, `MCP: quotes.append_reply` (and any HTTP-Request or credentialed node). Logic nodes (Code, If, Extract from File, AI Agent subnodes) execute normally.
**Error workflow pinned nodes:** none — it uses an Error Trigger, which can't be pinned in the same way; dry-run the main workflow with a known-failing MCP call to exercise it.
### Scenario 1 — new RFQ with PDF + PNG attachment
```json
{
"IMAP Quote Mailbox": [{
"json": {
"uid": "9999",
"mailbox": "INBOX",
"from": { "value": [{ "address": "david@wfsales.co.za", "name": "David Els" }], "text": "David Els <david@wfsales.co.za>" },
"subject": "Rfq 4034 — TEST PDF + PNG",
"messageId": "<test-pin-rfq-pdf-png-001@wfsales.co.za>",
"inReplyTo": "",
"textPlain": "Hi,\n\nPlease see attached for the RFQ.\n\nRegards,\nDavid",
"date": "2026-06-14T10:00:00.000Z"
},
"binary": {
"attachment_0": { "data": "<base64 of test-rfq.pdf>", "fileName": "Invoice INV8018.pdf", "mimeType": "application/pdf", "fileSize": 298 },
"attachment_1": { "data": "<base64 of test-1x1.png>", "fileName": "Outlook-Banner.png", "mimeType": "image/png", "fileSize": 67 }
}
}]
}
```
Expected: Parse Email emits `route: "new"`, Is Reply? goes false, Carry Email Ctx → Split → Extract (text for PDF, empty for PNG) → Aggregate (both binaries restored) → AI Agent. A new `quotes` row appears in WFDASH; both attachments appear in `quote_attachments` (PDF as `application/pdf`, PNG as `image/png`); the email remains in `INBOX` (UNSEEN).
### Scenario 2 — new RFQ with no attachments
```json
{
"IMAP Quote Mailbox": [{
"json": {
"uid": "1000",
"mailbox": "INBOX",
"from": { "value": [{ "address": "buyer@newco.co.za", "name": "Buyer Person" }] },
"subject": "RFQ: Mining cable ties and glands",
"messageId": "<rfq-newco-001@example.com>",
"inReplyTo": "",
"textPlain": "Please quote on 20mm cable ties qty 5 and cable gland M20 qty 10.",
"date": "2026-06-12T10:00:00.000Z"
}
}]
}
```
Expected: Parse Email → Is Reply? false → Carry Email Ctx → Split Binary emits one placeholder (no attachments) → Extract from File (no-op) → Aggregate → AI Agent. `create_quote_tool` runs once with `extracted_items` from the email body; no `upload_attachment_tool` calls.
### Scenario 3 — reply with In-Reply-To
```json
{
"IMAP Quote Mailbox": [{
"json": {
"uid": "101",
"mailbox": "INBOX",
"from": { "value": ["buyer@newco.co.za"], "name": "Buyer Person" },
"subject": "Re: Quotation Q-2026-06-001",
"messageId": "<reply1@example.com>",
"inReplyTo": "<rfq-newco-001@example.com>",
"textPlain": "Approved, please proceed.",
"date": "2026-06-12T11:00:00.000Z"
}
}],
"MCP: quotes.append_reply": [{
"json": {
"result": { "content": [{ "type": "text", "text": "{\"matched_by\":\"in_reply_to\",\"quote\":{\"id\":\"q-1\"},\"reply\":{\"id\":\"r-1\"}}" }] }
}
}]
}
```
Expected: Parse Email emits `route: "reply"`, Is Reply? goes true, `append_reply` called, `matched_by: "in_reply_to"` parsed. A new `quote_replies` row appears; `has_new_reply` flips to true; the email remains in `INBOX` (UNSEEN).
### Scenario 4 — subject-only reply (no In-Reply-To)
Same as Scenario 3 but `inReplyTo: ""` and subject `"Quote Q-2026-06-001 follow up"`. The subject regex catches it, `route: "reply"` is set, `append_reply` is called. The MCP server resolves by quote number and reports `matched_by: "subject"`.
### Scenario 5 — unmatched reply
Same as Scenario 3 but `inReplyTo: ""` and subject has no `Q-…` in it. `append_reply` returns `matched_by: "none"`, Route Reply Result goes false, the workflow ends. Nothing is moved; nothing is created. The email remains in `INBOX` (UNSEEN); a human triages it.
### Scenario 6 — error path
Pinned MCP node returns an error. n8n fires the Error Handler. Build Alert formats the message; Telegram posts to the chat. The email is still in `INBOX`; the human triages.
---
## Runbook — "an email didn't create a quote"
| Symptom | Where to look | Fix |
| --- | --- | --- |
| Nothing in the workflow execution list | n8n not running, or the workflow isn't activated | Open the workflow in n8n → toggle **Active**. Verify the IMAP trigger has `trackLastMessageId` so the same email isn't skipped. |
| Execution shows the email but it stopped at Filter | Filter rejected it | Inspect the data: sender is blacklisted, subject contains `out of office`, etc. Loosen the filter or whitelist the sender. |
| Telegram alert fires | Something in the main workflow threw | Open the error execution in n8n → see the failed node. The email is still in `INBOX` (UNSEEN); a human triages. |
| `INBOX` keeps growing with already-processed emails | n8n executions all show `success` for the same email, but the email is still in `INBOX` | **Expected in v2+.** n8n no longer moves mail out of `INBOX` — the JS task-runner sandbox cannot load `imapflow`. WFDASH dedups re-fires by `message_id` / `synth-…@wfdash.local`, so the duplicate workflow runs are no-ops. Move processed mail out of `INBOX` manually (or via a periodic cPanel IMAP filter). |
| Email has no `Message-ID` header | `Parse Email.message_id: ""` | Handled: the MCP server synthesizes a stable `synth-XXXXXXXX@wfdash.local` id from the sender + subject + received_at. The quote still gets created and a retry dedups correctly. The synthetic id is visible in the `quotes.email_message_id` field — clear indicator for the quoter. |
| Email has no `From:` header (malformed mail, BCC-only headers) | `Parse Email.from_email: ""` | Handled: WFDASH links the quote to a sentinel "Unknown Sender" customer (`unknown-sender@no-from.wfdash.local`) so the create succeeds. The quote's `email_sender` field stores the literal empty string. The quoter reassigns the customer in the existing quote-editing form if/when the real sender is known. |
| Email has no `Subject:`, no body, no `From:`, no `Message-ID:` (blank RFC 5321 send) | `Parse Email.subject: '(no subject)'`, `body_plain: '(no body)'`, `from_email: ""` | Handled: Parse Email stamps placeholders; WFDASH accepts them and creates the quote (linked to the sentinel customer, no line items). The office triages manually. |
| `upload_attachment_tool` returns `error: binary not found for key attachment_1` (PDF was uploaded, PNG was not) | This was the v3 bug. Fixed in v3.1. | Verify `AI Agent: Create Quote From Email` has `options.passthroughBinaryImages: false`. If a future agent option is added, also verify `upload_attachment_tool`'s IMAP-trigger fallback (`$('IMAP Quote Mailbox').first().binary[binaryKey]`) is still in place. |
| Quote appears with no line items | AI extraction returned empty array (email was a description with no quantities) | **Correct behaviour in v3.1.** Per the user's standing instruction, the workflow does not route unparseable emails to a manual-review queue — the office's quoters fill in the items in the WFDASH UI. Verify by opening the quote. |
| LLM invents line items (e.g. items not in the email body or the PDF text) | System prompt drift | The system prompt says `DO NOT invent items, prices, or quantities. Look BOTH in the email body AND in extracted_text_preview of each attachment.` Verify both source fields are populated in the user message. If the LLM is hallucinating, reduce `temperature` further or change to a more conservative model. |
| PDF text never reaches the LLM | `extracted_text_preview` is empty in the user message | Check that `Extract from File (PDF)` is wired between Split and Aggregate; check its `binaryPropertyName: "attachment"`; check `onError: continueRegularOutput` so PNGs don't kill the run. |
| `AI Agent` returns `quote_number=ERROR items=0 uploaded=0` | DNS failure on the MCP URL | The `MCP: quotes.create_from_email` URL is unreachable (e.g. pinggy tunnel expired). Verify the URL resolves: `nslookup hzbtf-196-39-76-18.run.pinggy-free.link`. Re-establish the tunnel or update `Edit Fields` → `mcp server`. |
| `Bearer Auth account` not wired | `MCP: quotes.append_reply` returns 401 | Manual UI step: open the node → Credentials → pick `Bearer Auth account`. The SDK has no operation for HTTP Request credential auto-assignment. |
---
## Anti-patterns this workflow explicitly rejects
(Adapted from the old Django app. None of these are allowed to creep back in.)
| Old problem | New constraint |
|---|---|
| Four competing schedulers | **One** IMAP trigger. No background processes in Next.js. |
| `cost_price = 0.8 × selling_price` (fabricated) | Cost price left at 0; `included_in_quote = false` until a human sets it. |
| `_needs_review = True` hard-coded | New quotes always have `status: "new"`. AI-extracted items are suggestions, never auto-included. |
| 40% fuzzy word match on company | `findOrCreateCustomer` / `findOrCreateCompany` use exact match on email, then normalised domain match. **Never** fuzzy merge. New records tagged via `customer.description = "Auto-created from email intake"`. |
| `last.split('-').pop() + 1` race | Server-side `generateQuoteNumberServer()` uses the atomic `quote_counters` collection with `last_seq+` increment. |
| Replies as text blob in `notes` | `quote_replies` collection. The reply tool appends a structured row, **never** mutates `notes` directly. |
| `notifications_email = "@wfsales.co.za"` placeholder | No such env var. Errors come from the Error Handler workflow → Telegram. |
| LLM paraphrases / drops required email metadata | The LLM is told in the system prompt to copy metadata byte-for-byte; `create_quote_tool` reads the email fields from upstream `$json` as a backstop, so even if the LLM misformats, the create call has the right data. |
| LLM invents line items | System prompt says `DO NOT invent` and points at `extracted_text_preview` as the source of truth. |
| LLM consumes PNG binary as vision input | `passthroughBinaryImages: false` on the AI Agent (v3.1). PNGs now survive to the upload tool. |
| PDF text never reaches the LLM | New Split → Extract → Aggregate pipeline puts a 2 KB preview of each PDF's text into the user message. |
| Manual-review queue for unparseable emails | **No manual-review queue.** Per the user's standing instruction, if the LLM can't parse the email, it still creates the quote (possibly with empty `extracted_items`) and the office's quoters fill in the items in the WFDASH UI. |
| Non-PDF attachments (PNG) break the Extract from File node | `onError: continueRegularOutput` makes the node pass through; the aggregate shows empty `extracted_text` and the upload still happens. |
| Silent loss on workflow crash | IMAP filter `UNSEEN` + `postProcessAction: "nothing"` means a crash re-tries on the next poll. The WFDASH database row is the success signal. |
| "Marked as read" used as audit signal | Real database rows in PocketBase (`quotes` / `quote_replies`) are the audit signal. n8n does **not** mutate the IMAP mailbox. |
| Subject regex fallback for `company_hint` | Company is left `null` if the AI returns no `company_hint`. A human fills it in. |
| Fabricated `synth-…@wfdash.local` ids pretending to be real | Synth ids are clearly marked with the `synth-` prefix and `@wfdash.local` suffix so they're impossible to confuse with a real RFC 5322 Message-ID. |
---
## Quoter can verify the source email
The quote detail page now has an **Original Email** card between the Items card and the Notes card. The quoter uses it to verify the AI's extracted line items against what the original RFQ email actually said.
Three sub-sections in one Card:
1. **Email metadata** — subject, sender, received_at. Three-column grid with small icons.
2. **Email body** — plain text in a scrollable `<pre>` (max-h-96). This is `quotes.email_body` populated by `Parse Email` / the intake code, so it is the exact text the AI saw.
3. **Attachments** — list of files with kind icon, humanized size, and a download link. Download uses `pb.files.getUrl(attachment, attachment.file, { query: { token: pb.authStore.token } })` — the user's auth token is embedded in the query string because PB's file API won't accept the cookie that the auth provider uses.
The card renders `null` (no card at all) when `email_body` is empty **and** there are no attachments, so manually-created quotes (no email intake) stay clean. The Email Status card at the bottom of the page (Delivered / Opened / Sender / Subject) is unchanged — it remains the structured status summary; the Original Email card is the **content** the AI saw.
HTML email rendering is out of scope for V1 — we throw the HTML body away on intake and keep only `body_plain`. The plain text is what the AI saw, which is what the quoter needs to verify. Add a "Show rendered HTML" toggle if the user later asks for it.
Component: [src/components/quote-email-card.tsx](../src/components/quote-email-card.tsx). Helper: `getQuoteAttachments(quoteId)` in [src/lib/pb-helpers.ts](../src/lib/pb-helpers.ts) (separate `getFullList` against the `quote_attachments` collection rather than a reverse-relation expand, because PB expand syntax doesn't easily cover reverse relations).
---
## Security notes
- **Credentials live in n8n's vault**, never in the Next.js env. The `AGENT_API_KEY` is a long-lived secret — rotate it like any other token.
- The workflow uses `Authorization: Bearer` against WFDASH. The same key also protects the REST API. If you need finer scoping, replace the single `AGENT_API_KEY` with a per-purpose key (e.g. `WFDASH_N8N_KEY`).
- Pin data is **not encrypted** in the n8n DB. Avoid putting real customer PII or attachment contents into pin data when dry-running.
- The IMAP mailbox password is the highest-value secret here. Use a dedicated mailbox password, scoped to the `quotes@` IMAP folder only.
- The Telegram bot token belongs in the `Telegram Alerts` credential. **Do not** put it in env vars or in this repo. Set the chat ID in `TELEGRAM_CHAT_ID` env (a public value — a chat ID is not a secret).
---
## Versioning
- Main workflow ID: `VOrEd4OQKO0KMnJj` (v3.1: langchain AI Agent + tool subnodes, Split → Extract → Aggregate for PDF text, `passthroughBinaryImages: false` to preserve PNG binary)
- Main workflow ID: `8zjoryZvn0OZzToE` (v3: AI Agent v3.1 + Information Extractor pair, two toolCode subnodes — superseded by v3.1, archived)
- Main workflow ID: `DGN5fslHfdkIyLKb` (v2: tolerates empty `Message-ID`, tolerates empty IMAP `uid`, tolerates empty `From:` (sentinel "Unknown Sender"), tolerates empty Subject and empty body (stamps `(no subject)` / `(no body)`), no IMAP Move step — superseded, archived)
- Error workflow ID: `kVylcetxy858PcfP` (no change in v3.1)
- Project: David Els <david.els1@outlook.com> (personal)
- Authored via: [n8n-mcp-setup.md](n8n-mcp-setup.md)
- Last updated: 2026-06-14
Reference in a new issue View git blame Copy permalink