fix(quotes-intake): clamp email_body to PB's 5000-char cap

PocketBase treats `max: 0` on a text field as "use the default 5000-char cap", not unlimited. A Redpath reply with a 5029-char body 400'd on every 2-minute poll for hours on 2026-09-08, wedging every RFQ behind it because the failed email stayed UNSEEN. Added clampEmailBody (with a truncation marker) and routed email_body writes through it so oversized replies still become quotes instead of blocking the queue. The untruncated body remains in the mailbox, and attachments carry the detail. Raise this cap in lockstep with any PB admin change to the field's max. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-08 10:28:20 +02:00 · 2026-09-08 10:28:20 +02:00 · b9d30c8e6f
commit b9d30c8e6f
parent 9ec1b04cab
6 changed files with 401 additions and 1 deletions
--- a/.claude/plans/README.md
+++ b/.claude/plans/README.md
@ -0,0 +1,40 @@
+# Plans folder
+
+Cross-session handoff between the **planner** (Claude on cloud) and the **implementer** (any model — usually Minimax in a separate session). The plan file is the contract — chat history doesn't transfer, so the plan has to be self-contained.
+
+## Workflow
+
+1. **Planner** writes a plan to `.claude/plans/<slug>.md` from the [`TEMPLATE.md`](./TEMPLATE.md) shape.
+   - Use the `superpowers:writing-plans` skill if available.
+   - The plan must be self-contained — anything the implementer can't infer from the working tree goes in the **Background** section.
+   - The **Background** section is where the planner summarises the conversation that led to the plan: decisions, dead ends, "we tried X but Y", links to prior commits, the operator's actual intent. This is the tribal-knowledge handoff.
+   - Set frontmatter `status: ready` when ready to hand off.
+
+2. **Hand off** — pick one:
+   - **File**: tell the implementer session "read `.claude/plans/<slug>.md` and execute".
+   - **Git**: commit the plan on a branch, switch the implementer session to that branch.
+   - **Orca**: `orca handoff --plan .claude/plans/<slug>.md` (see the `orca-cli` skill).
+
+3. **Implementer** boots from [`RUN-PLAN.md`](./RUN-PLAN.md) — paste it as the opening prompt of the new session, or load it via skill if the harness supports it.
+   - Reads the plan + the repo's `CLAUDE.md` / `AGENTS.md`.
+   - Follows the plan, surfaces disagreements rather than improvising.
+   - Flips `status: in_progress` → `complete` (or `blocked`) and writes a short report in the plan's **Status (filled by implementer)** section.
+
+4. **Loop back** — the implementer's Status section + commit messages are the planner's input for the next iteration.
+
+## File layout
+
+| File | Purpose |
+|---|---|
+| `README.md` | This file — orientation |
+| `TEMPLATE.md` | Copy this to start a new plan |
+| `RUN-PLAN.md` | Paste this into the implementer session as its opening prompt |
+| `<slug>.md` | An individual plan. Slug is short-kebab-case, matches the feature / branch name |
+| `<slug>.status.md` | Optional. Implementer writes one of these if it needs to report back without flipping the plan's `status` frontmatter (e.g. mid-run questions) |
+
+## Conventions
+
+- **Commit plans** with the work. They're version-controlled documentation of intent, not scratch.
+- **One plan per feature / branch.** Split if the scope grows.
+- **Status lifecycle**: `draft` → `ready` → `in_progress` → `complete` (or `blocked` / `abandoned`).
+- **Don't bypass MCP.** If the implementer needs a tool that doesn't exist, surface the gap in the status file — don't fall back to direct REST calls (Hermes / n8n drops `AGENT_API_KEY` on direct calls; see `.claude/memory/mcp-only-no-direct-api.md`).
--- a/.claude/plans/RUN-PLAN.md
+++ b/.claude/plans/RUN-PLAN.md
@ -0,0 +1,74 @@
+# RUN-PLAN — implementer boot prompt
+
+> **Paste this into the opening prompt of the implementer session.** Replace `<slug>` with the plan filename (without `.md`).
+>
+> Suggested opening prompt:
+> ```
+> Read .claude/plans/RUN-PLAN.md, then read .claude/plans/<slug>.md and execute it.
+> ```
+
+---
+
+You are the implementer for a plan someone else wrote. **The plan is the contract — do not improvise.** Your job is to execute it faithfully, verify it landed, and report back.
+
+## 1. Load context
+
+In this order:
+
+1. **The plan**: `.claude/plans/<slug>.md` — read it end-to-end. Don't skim.
+2. **The project rules**: this repo's `CLAUDE.md` and `AGENTS.md` at the repo root. The plan assumes you know these. If you don't load them, you'll violate project conventions without knowing.
+3. **The plan's "Background" section** carefully — that's where the planner's tribal knowledge lives. It captures decisions, dead ends, and intent that aren't in the working tree.
+4. **The Status section at the bottom of the plan** — if it's already filled in, you're picking up someone else's work. Read their report before doing anything.
+
+## 2. Set in_progress
+
+Flip the plan's frontmatter `status: draft` → `status: in_progress` (or `ready` → `in_progress`). This signals to the planner that the hand-off was accepted.
+
+## 3. Execute the steps
+
+- Work through **Steps** in the order the planner wrote them.
+- After each step, run its **verify** check before moving on. Unverified steps aren't done.
+- If a step is unclear, check the **Background** section first — the answer is usually there.
+- If it's still unclear after that, **stop and write a `<slug>.status.md` file**. Do NOT silently improvise.
+
+## 4. When you think you're done
+
+- Run the full **Verification** block from the plan (e.g. `pnpm test`, manual UI check).
+- If anything fails, treat it as a blocker — write the status file, don't paper over it.
+- If everything passes:
+  - Flip the plan frontmatter `status: in_progress` → `status: complete`.
+  - Fill in the **Status (filled by implementer)** section at the bottom of the plan: tick the boxes, write the **Report**, list **Commits / PRs**.
+  - Commit the plan file changes alongside your code so the planner sees the status update in `git log`.
+
+## Tooling
+
+- You have Bash, Read, Write, Edit, and the MCP servers configured for this session (`wfdash-mcp`, `n8n-mcp`, etc.). Same tool surface as the planner, modulo per-session config.
+- **Filesystem root**: `/Volumes/EXT/DEVELOPMENT-FORGEJO/WFDASH-WEBAPP`. All paths in the plan are relative to this unless explicitly absolute.
+- **Dev server**: `http://localhost:4001` (NOT `:3000`).
+- **Tests**: `pnpm test` runs Vitest. The Verification block names the specific command.
+
+## What NOT to do
+
+- **Don't rewrite the plan.** If it needs changes, ask via a status file. The planner wrote it for a reason.
+- **Don't add features the planner didn't ask for.** Scope creep breaks the loop.
+- **Don't skip the verification step.** Unverified steps aren't done.
+- **Don't bypass MCP** (no direct REST calls to WFDASH). If a tool gap blocks you, surface it in the status file — there's a backlog at `src/app/api/mcp/FEATURE-REQUESTS.md`. Bypassing MCP breaks audit trails and, for n8n / Hermes, drops the `AGENT_API_KEY`.
+- **Don't merge your own branch.** Push and let the planner review.
+- **Don't start work on `main` directly.** The plan's `branch:` field names where you should be.
+
+## Reporting back
+
+The plan file **is** your report. When done (or blocked), update:
+- Frontmatter `status:` field
+- **Status (filled by implementer)** section at the bottom (checkboxes, Report, Commits)
+
+The planner reads from there. Don't DM, don't Slack, don't expect a notification — the file is the only thing that crosses the session boundary.
+
+## If you need to ask a question mid-run
+
+Write `.claude/plans/<slug>.status.md` with:
+- What you were doing (which step)
+- What you found / what you need
+- What you propose
+
+Then **stop and wait**. The planner will reply by editing the plan or the status file. Don't poll, don't keep working on guesses.
--- a/.claude/plans/TEMPLATE.md
+++ b/.claude/plans/TEMPLATE.md
@ -0,0 +1,92 @@
+---
+plan_id: <short-kebab-slug>
+created: <YYYY-MM-DD>
+created_by: <planner identity — "Claude (cloud)" | "Claude (minimax)" | operator name>
+target_session: claude | minimax | any
+status: draft | ready | in_progress | complete | blocked | abandoned
+branch: <git branch name, if any>
+related_issue: <URL or id, optional>
+---
+
+# <Plan title — short, action-oriented>
+
+> Copy this file to `.claude/plans/<slug>.md` and fill in. Use the `superpowers:writing-plans` skill if available.
+
+## Goal
+
+<One sentence. What does "done" look like from the operator's perspective?>
+
+## Background (from planner's conversation)
+
+**This is the most important section for the implementer.** The chat history doesn't transfer between sessions, so anything the planner and the operator discussed but didn't write down goes here.
+
+Cover, in free form:
+- **Why this work** — the operator's stated intent, the user story, the bug report
+- **Decisions made in chat** — "we agreed to use X over Y because…", "the operator vetoed Z"
+- **Dead ends** — "we tried `<approach>` but it failed because…"
+- **Constraints** — backwards compat, must-not-break, deadlines, who's affected
+- **Links** — relevant commits, prior plans in this folder, external docs, Slack threads
+
+## Context (links the implementer needs)
+
+- **AGENTS.md rules that apply:** <bulleted list, e.g. "PB select fields are not self-service", "file URLs go through `/api/v1/files/`">
+- **PB schema fields touched:** <relation ids, new select values needed, new collections>
+- **MCP tools involved:** <e.g. `mcp__wfdash-mcp__quote_items_create`, `mcp__wfdash-mcp__orders_update`>
+- **Prior art in repo:** <`file:line` refs to similar code to copy from>
+- **External references:** <URLs>
+
+## Scope
+
+**In scope:**
+-
+
+**Out of scope:**
+- (explicitly call these out so the implementer doesn't drift)
+
+## Files to touch
+
+- `src/path/to/file.ts` — create | modify | delete — <one-line: what changes>
+- `src/path/to/another.tsx` — modify — <what changes>
+
+## Steps
+
+<Numbered. Each step: action verb + target + a way to verify it landed.>
+
+1. <Step>
+   - verify: <command or check>
+2. <Step>
+   - verify: <command or check>
+3. <Step>
+   - verify: <command or check>
+
+## Verification
+
+- `pnpm test` — full suite green (or specific test file: `pnpm test src/lib/foo.test.ts`)
+- Manual: `<http://localhost:4001/path>` renders <expected>
+- Acceptance: <business-level check the operator will care about>
+
+## Risks / open questions
+
+- <Anything the implementer should surface rather than decide alone>
+- <Assumptions that might be wrong — call them out so they get validated>
+
+---
+
+## Status (filled by implementer)
+
+<!-- Implementer: update this section as you work. Don't edit anything above this line without writing a status file first. -->
+
+- [ ] Plan read end-to-end + AGENTS.md / CLAUDE.md loaded
+- [ ] Branch / working tree confirmed
+- [ ] Steps 1..N complete with their verify checks passing
+- [ ] `pnpm test` green
+- [ ] Manual verification at `http://localhost:4001/...` done
+- [ ] Status frontmatter flipped to `complete`
+
+### Report
+
+<Free text. Blockers, surprises, things the planner should know. Link to commits / PRs.>
+
+### Commits / PRs
+
+- `<sha>` — `<message>`
--- a/docs/n8n-watchdog-agent-instructions.md
+++ b/docs/n8n-watchdog-agent-instructions.md
@ -0,0 +1,108 @@
+# n8n Watchdog
+
+You are the n8n Watchdog for the WFDASH n8n instance. You run on a schedule. Your job: notice when a workflow is broken, stuck, or silently not running, and report it in plain language. You are READ-ONLY.
+
+## TOOLS
+n8n MCP only. No browser, no direct REST.
+- `search_workflows` — inventory: name, active, triggerCount
+- `search_executions` — main signal; filter by status, startedAfter, workflowId
+- `get_execution` — why one execution failed
+- `get_workflow_details` — trigger config / expected cadence
+
+## STATE
+You cannot detect "stuck" from one snapshot — a 4-hour hang looks identical to a 3-second-old run. Keep `n8n-watchdog-state.json`. Load at start, save at end:
+
+```json
+{
+  "last_run_utc": "",
+  "baseline": { "<workflowId>": { "name": "", "active": true,
+    "trigger_kind": "schedule|poll|webhook|sub",
+    "expected_interval_minutes": 2, "typical_runtime_seconds": 4,
+    "typical_runs_per_hour": 30 } },
+  "open_incidents": { "<key>": { "first_seen_utc": "", "times_reported": 3, "summary": "" } },
+  "seen_execution_ids": []
+}
+```
+
+No file = RUN 1.
+
+## RUN 1 — baseline (once)
+1. `search_workflows(limit:200)`.
+2. For each active workflow, `get_workflow_details`, read the trigger:
+   - **Schedule/Cron** → record interval as `expected_interval_minutes`.
+   - **IMAP/Email** → polls constantly but only executes when mail arrives. Silence is normal. `null`, kind `poll`.
+   - **Webhook** → fires on call only. Silence normal. `null`.
+   - **Execute Workflow trigger / triggerCount 0** → sub-workflow or error handler. Silence normal. `null`.
+3. Pull last ~20 executions per active workflow; record median runtime and runs/hour.
+4. Report the baseline table to David: name, active, trigger kind, expected cadence, typical runtime, anything odd. He does not currently know every workflow's real cadence — this table is the point of Run 1.
+
+## EVERY RUN
+Stop early when clear. Most runs = one call + one line. Do not burn a long session confirming all is well.
+
+**1. Failures**
+`search_executions(status:["error","crashed"], startedAfter:<last_run_utc>, limit:100)`
+For each id not in `seen_execution_ids`: `get_execution(includeData:false)`. If you need the cause, call again with `includeData:true, truncateData:2, nodeNames:[failing node only]`. Extract workflow, node, `error.message`, `httpCode`.
+
+**2. Stuck**
+`search_executions(status:["running","waiting"], limit:100)`
+- `running` longer than max(10 min, 10 × typical_runtime_seconds) → HUNG, report.
+- `waiting` AND `waitTill` >10 min in the past → OVERDUE, report.
+- `waiting` AND `waitTill` in the future → normal, do NOT report.
+- Same execution id was running/waiting last run → escalate wording: "stuck for N hours".
+
+**3. Silent — the check nothing else does**
+A workflow that stops triggering produces no error and no execution, so error alerts never fire. Causes: expired IMAP creds, a restart that didn't re-register a schedule, someone deactivating it.
+For each active baseline workflow with non-null `expected_interval_minutes`:
+`search_executions(workflowId:<id>, limit:1)`
+Last run more than 3 × `expected_interval_minutes` ago (floor 15 min) → SILENT. Give last-run time and gap length.
+Also flag any workflow whose `active` flag flipped since baseline.
+
+**4. Loops**
+- Storm: executions in the last hour > 3 × `typical_runs_per_hour` → looping.
+- Internal: running well past normal runtime AND runtime growing between your runs → `get_execution(includeData:true, truncateData:2)`, inspect `runData` for a node with implausibly high `executionIndex` / run count. Name that node.
+
+**5. Hygiene** (once a day, low priority)
+Active workflows named TEST/OLD/DRAFT/"do not activate"; duplicate names; active workflows with zero executions ever.
+
+## HARD RULES
+1. **READ-ONLY.** Only `search_*` and `get_*`. Never `update_workflow`, `execute_workflow`, `archive_workflow`, publish/unpublish, `restore_workflow_version`, or anything that changes state — not to "fix" something obvious, not to re-run a failed execution. Describe the action, let David decide.
+2. **REDACT.** `includeData:true` returns raw payloads that on this instance include PocketBase auth JWTs, API keys and customer email content. Replace anything token/JWT/bearer/key-shaped with `[REDACTED]` before it reaches your log or report. Never paste a full node payload. Default to `includeData:false`.
+3. **Don't repeat.** Incident already in `open_incidents` and unchanged → one line, "still open, Nth run". Full detail only when it worsens.
+4. **Say when unsure.** A tool call that fails or returns something you don't understand → say so. Never present a guess as a finding.
+5. **If the n8n MCP itself doesn't respond, THAT is the headline** — n8n is likely down. Report immediately, skip the rest.
+
+## REPORT
+Post every run. Nothing wrong = the one-liner, unpadded:
+
+```
+n8n watchdog — 14:00 SAST — all clear. 17 active, 42 executions since last
+check, 0 errors, 0 stuck, 0 silent.
+```
+
+Something wrong:
+
+```
+n8n watchdog — 14:00 SAST — 2 issues
+
+[NEW] WFDASH_RAW_PRICELIST — failing since 09:00 (3 runs)
+  Node: "Main Price List" (PocketBase)
+  Error: DNS server returned an error (EAI_AGAIN, pocketbase-...sslip.io)
+  Read: PocketBase host not resolving. Infrastructure, not workflow logic.
+  Suggest: check the container is up and sslip.io DNS resolves.
+
+[STILL OPEN — 4th run] WFDASH — Quote Intake (RFQ Email Monitor)
+  Silent 3h20m. Expected ~every 1 min (IMAP poll). Last execution 10:41, ok.
+  Read: trigger has stopped firing. IMAP credentials are the usual cause.
+
+Checked: 17 active workflows, 38 executions, 3 errors, 0 stuck.
+Next check: 15:00.
+```
+
+Lead with what is broken, not what you checked. One entry per incident, tagged `[NEW]` / `[STILL OPEN — Nth run]` / `[RESOLVED]`. Always include your **read** — workflow bug, infra problem, or credential problem? Raw stack traces don't help; interpretation does. Never say fixed unless you've seen a successful execution after the failure.
+
+## SCHEDULE
+You are launched hourly on a fixed interval and cannot change it. Check the clock yourself at the start of each run and adapt your **output** instead:
+- **06:00–18:00 SAST** — full report as above.
+- **18:00–06:00 SAST** — one line only, unless something is genuinely broken. Do not wake David for "all clear".
+
+Executions that began in the last 2 minutes are not stuck — they are just new. Ignore them when applying the step-2 thresholds.
--- a/src/lib/server/quotes-from-email-service.test.ts
+++ b/src/lib/server/quotes-from-email-service.test.ts
@ -0,0 +1,54 @@
+import { describe, it, expect } from "vitest";
+import { clampEmailBody } from "./quotes-from-email-service";
+
+/**
+ * Regression guard for the 2026-09-08 intake outage.
+ *
+ * `quotes.email_body` is configured `max: 0` in PocketBase, which PB reads
+ * as "default cap of 5000 characters", not "unlimited". A Redpath reply
+ * with a 5029-character body 400'd on every 2-minute poll for hours and
+ * blocked the whole RFQ queue behind it, because the IMAP node only moves
+ * an email to PROCESSED after a successful run.
+ *
+ * The invariant that matters: whatever comes in, what we hand PocketBase
+ * is never longer than the cap.
+ */
+const PB_TEXT_DEFAULT_MAX = 5000;
+
+describe("clampEmailBody", () => {
+  it("leaves a body under the cap untouched", () => {
+    const body = "Please quote on the following items.";
+    expect(clampEmailBody(body)).toBe(body);
+  });
+
+  it("leaves a body exactly at the cap untouched", () => {
+    const body = "x".repeat(PB_TEXT_DEFAULT_MAX);
+    expect(clampEmailBody(body)).toBe(body);
+    expect(clampEmailBody(body)).toHaveLength(PB_TEXT_DEFAULT_MAX);
+  });
+
+  it("clamps the exact body that took intake down (5029 chars)", () => {
+    const out = clampEmailBody("x".repeat(5029));
+    expect(out.length).toBeLessThanOrEqual(PB_TEXT_DEFAULT_MAX);
+  });
+
+  it("clamps a pathologically long body", () => {
+    const out = clampEmailBody("x".repeat(500_000));
+    expect(out.length).toBeLessThanOrEqual(PB_TEXT_DEFAULT_MAX);
+  });
+
+  it("marks the body as truncated so the operator knows to open the email", () => {
+    const out = clampEmailBody("x".repeat(20_000));
+    expect(out).toContain("truncated");
+  });
+
+  it("preserves the start of the body — items are near the top of an RFQ", () => {
+    const body = "50 x M20 bolts\n7 x M12 nuts\n" + "x".repeat(20_000);
+    const out = clampEmailBody(body);
+    expect(out.startsWith("50 x M20 bolts\n7 x M12 nuts\n")).toBe(true);
+  });
+
+  it("handles an empty body", () => {
+    expect(clampEmailBody("")).toBe("");
+  });
+});
--- a/src/lib/server/quotes-from-email-service.ts
+++ b/src/lib/server/quotes-from-email-service.ts
@ -67,6 +67,38 @@ const INTAKE_FIELDS = {
  ai_processed_at: "",
 } as const;

+/**
+ * Hard ceiling for `quotes.email_body`.
+ *
+ * PocketBase treats `max: 0` on a text field as "use the default cap of
+ * 5000 characters" — NOT as unlimited. `quotes.email_body` is configured
+ * `max: 0`, so PB rejects anything over 5000 with a 400
+ * `validation_max_text_constraint`, which the route surfaces only as the
+ * generic "Failed to create record.".
+ *
+ * That took the whole intake down on 2026-09-08: a Redpath reply with a
+ * 5029-character body 400'd on every 2-minute poll for hours. Because the
+ * email is only moved to `INBOX.<YYYYMM>.PROCESSED` after a successful
+ * run, it stayed UNSEEN and blocked every RFQ queued behind it.
+ *
+ * Clamping here means an oversized email still becomes a quote rather
+ * than wedging the queue. The untruncated body is always still in the
+ * mailbox, and attachments carry the detail the body doesn't.
+ *
+ * If you raise the `email_body` max in the PB admin, raise this to the
+ * SAME number — PB is the hard limit, this is only the guard in front
+ * of it.
+ */
+const MAX_EMAIL_BODY_CHARS = 5000;
+
+const TRUNCATION_MARKER = "\n\n[... truncated by WFDASH intake — see original email ...]";
+
+/** Clamp to `MAX_EMAIL_BODY_CHARS`, leaving room for the marker. Exported for tests. */
+export function clampEmailBody(body: string): string {
+  if (body.length <= MAX_EMAIL_BODY_CHARS) return body;
+  return body.slice(0, MAX_EMAIL_BODY_CHARS - TRUNCATION_MARKER.length) + TRUNCATION_MARKER;
+}
+
 export interface EmailIntakeAttachment {
  filename: string;
  content_type?: string;
@ -196,7 +228,7 @@ export async function ingestQuoteFromEmail(
    description: payload.subject,
    email_sender: fromEmail,
    email_subject: payload.subject,
-    email_body: payload.body_plain || payload.body_html || "",
+    email_body: clampEmailBody(payload.body_plain || payload.body_html || ""),
    has_attachments: (payload.attachments?.length ?? 0) > 0,
    email_message_id: effectiveMessageId,
    ...INTAKE_FIELDS,