WFDASH-WEBAPP/docs/SHIPPED.md

395 lines
20 KiB
Markdown
Raw Normal View History

# WFDASH — What We've Shipped
> **The one-month build log.** 24 May → 22 Jun 2026.
> 15 commits · ~30K lines of real code · ~57K net lines added (incl. lockfile churn). One Next.js 16 CRM that took a legacy Django app and gave it AI, an email pipeline, and a real supplier map.
---
## The Project
**WFDASH** is a business CRM — customers, companies, suppliers, quotes, orders, driver collection pool, delivery notes, repairs. It's a re-platform of an older Django app, rebuilt on a modern stack:
| Layer | Choice |
|---|---|
| Framework | Next.js 16 (App Router) + React 19 + TypeScript |
| UI | shadcn/ui (base-nova style) + Tailwind 4 + Lucide |
| Maps | Leaflet + react-leaflet (clustering, marker edit, fullscreen, dark tiles) |
| Backend | PocketBase (DB + auth + file storage + realtime) |
| Automation | n8n (email monitoring, AI quote rewrite) |
| Agent API | in-app JSON-RPC MCP endpoint + REST `/api/v1/*` |
| Packaging | Multi-stage Docker (Node 20 alpine, non-root) + Nixpacks |
| Tests | Vitest (unit) + Playwright (E2E) |
> **North star:** *"One scheduler, one workflow, one source of truth — no long-running process in the Next.js app, no racy counters, no fabricated cost prices."*
---
## The Timeline At A Glance
```
24 May ─── Foundation sprint (UI kit, PWA, PocketBase collections)
25 May ─── PocketBase APIs + migration docs
27 May ─── Supplier map (Leaflet) ships
11 Jun ─── Test scaffolding (Vitest + Playwright + line endings)
16 Jun ─── Email-to-quote pipeline (v3.1 agentic + PDF text)
22 Jun ─── Big batch: atomic numbering → email cards → AI rewriter → pricing lock → UX polish
```
**By the numbers:**
- **15 commits**, ~57K net lines added (78K insertions / 22K deletions; ~27K of that is real code in `src/`)
- **29,582 lines** currently living in `src/`, `n8n_workfows/`, `scripts/`, `docs/`, `e2e/`
- **5 n8n workflow files** (Quote Email Monitor v3.1 → v4 → v5, Quote Item Rewriter, Error Handler)
- **14 PocketBase collections** (12 from setup + 2 added with the email pipeline)
- **9 REST endpoints** under `/api/v1/*` + **8 MCP tools** under `/api/mcp`
- **15 Vitest unit suites** + **3 Playwright E2E specs**
The MCP tools list (the AI agent uses these):
`orders.list` · `orders.create_from_quote` · `collection_pool.list` · `order_items.update_status` · `quotes.create_from_email` · `quotes.append_reply` · `quotes.find_by_message_id` · `attachments.upload_for_quote`
---
## Phase 1 — Foundation (24 May)
**Commit `363786e` — UI kit shipped**
The bones of the app. shadcn/ui components that the rest of the build rests on:
- `separator`, `sheet`, `sonner`, `switch`, `table`, `tabs`, `textarea`, `tooltip`
- `data-table.tsx` — generic table with sort/filter/pagination (became the workhorse for every list page)
- `line-items-table.tsx` — quote & order line items
**Commit `a828432` — Responsive & accessibility pass**
Dashboard and quotes pages get a proper mobile-card layout, keyboard nav, and ARIA labels.
**Commit `954dc26` — PWA scaffold**
Service worker, offline page, manifest. The app installs to a phone home screen and survives flaky connections in the warehouse.
---
## Phase 2 — Backend Wiring (25 May)
**Commit `fd1571e` — PocketBase collections, APIs, migration docs**
The schema + the agent surface area come online together:
- Base collections: `customers`, `companies`, `suppliers`, `quotes`, `quote_items`, `orders`, `purchase_orders`, `order_items`, `collection_pool`, `delivery_notes`, `delivery_note_items`, `repairs`, `stock_items`
- `scripts/setup-pocketbase.ts` — one-shot setup from `npm run setup`
- MCP endpoint (`/api/mcp`) with `tools/list` + `tools/call`
- REST endpoints under `/api/v1/*` (auth via `Authorization: Bearer AGENT_API_KEY`)
- Migration docs: `core-recheck-matrix.md`, `django-nextjs-parity-backlog.md`, `order-workflow-rules.md`
---
## Phase 3 — Supplier Map (27 May)
**Commits `565b1a6` → `54644c7` → `83935c9` → `a6ac88a`**
A real, interactive supplier map. Five commits in one day because each fix surfaced the next:
| Commit | What landed |
|---|---|
| `565b1a6` | Leaflet + react-leaflet, supplier markers, clustering |
| `54644c7` | Better clustering algorithm + readable labels |
| `83935c9` | Dark tile layer + fullscreen mode + click-to-edit marker coordinates |
| `a6ac88a` | Node 20.9.0 enforced via `engines` + `nixpacks.toml` (so CI builds don't surprise us with Node 18) |
Why it matters: the warehouse team can see supplier locations at a glance, click any pin to fix a wrong coordinate, and the map works in dark mode in the cab at night.
---
## Phase 4 — Test Scaffolding (11 Jun)
**Commit `c4e1c62` — Standardize line endings + full test + config scaffolding**
A boring-but-critical commit:
- Vitest + jsdom + Testing Library, with path aliases matching `tsconfig.json`
- Unit suites for: `cn()`, `quote-status`, `order-item-status` (incl. legacy `driver_status` alias), `agent-auth`, `orders-service`, `pocketbase-admin`, every `/api/v1/*` route, every MCP tool
- Playwright E2E: health probe, agent auth (skips if no API key), login (skips if no creds)
- VSCode workspace + extensions + MCP config examples
- `.gitignore` extended for test artifacts
The "skips if no creds" behaviour is intentional — fresh checkout gets `1 passed, N skipped`, never a red CI.
---
## Phase 5 — Email-to-Quote Pipeline (16 Jun → 22 Jun)
**This is the headline feature.** Three workflow versions, one PocketBase schema, one atomic counter, and a long list of "no, really, it works this time" fixes.
### The flow
```
IMAP poll (quotes@wfsales.co.za, every 5 min, UNSEEN only)
↓
Parse Email + binary preservation
↓
Drop blacklists / auto-replies
↓
Is reply to existing quote?
├── yes → MCP: quotes.append_reply → terminal
└── no → Carry ctx → Split Binary → Extract from File (PDF)
↓
AI Agent (MiniMax) with two tools:
├─ create_quote_tool (MCP: quotes.create_from_email)
└─ upload_attachment_tool (MCP: attachments.upload_for_quote)
↓
PocketBase record + per-attachment rows linked back
```
### Commit `41b5fc4` — Pipeline v3.1 (agentic + PDF text)
Replaces the brittle v3 "Information Extractor → Code" pair with a **langchain AI Agent** that directly calls the MCP tools. PDF attachments go through `Split Binary → Extract from File (PDF) → Aggregate` so the LLM sees the line items as text. Flips `passthroughBinaryImages: false` so PNG attachments survive to the upload tool — the v3 bug where `attachment_1` was eaten by LLM vision is fixed.
Side effects of this commit:
- `pb_quote_attachments.collection.json` + `quote_counters.collection.json` + `quote_replies.collection.json`
- Migration scripts: `create-quote-attachments-and-replies.ts`, `broaden-quote-attachments-mime-types.ts`
- `scripts/n8n-v2-main.ts`, `scripts/n8n-v2-error.ts` — the n8n Code nodes lifted into TypeScript for type-checking
- Smoke scripts: `counter.ts`, `counter-concurrent.ts`, `create-from-email.ts`, `upload-attachment.ts`
- `src/lib/server/quotes-service.ts` — 747 lines of business logic, fully unit-tested
### Commit `3359195` — Atomic numbering + v5 catch-and-create
**The bug:** in v4, by the time the workflow reached "Upload Attachment", the binary had already been stripped → silent upload failure.
**The fix (v5 "Catch and Create"):** upload attachments to PocketBase *immediately* after auth (binary still fresh), then do all the rest using record IDs, then PATCH each attachment to link it to the quote. The old node is renamed `DISABLED - Upload Attachment` so it's a one-click rollback.
Atomic numbering:
- `/api/v1/quotes/next-number` returns `Q-YYYY-MM-NNN`
- Backed by `quote_counters` collection with `findOrCreateByMessageId` dedup
- Concurrent smoke test (`counter-concurrent.ts`) proves the counter doesn't double-issue under load
Files in this drop:
- `n8n_workfows/WFDASH — Quote Email Monitor (v5 catch-and-create).json`
- `pb_quote_attachments.schema.json`
- `scripts/build_v5_workflow.py` (regenerates v5 from v4)
- `scripts/deploy_v5_to_n8n.py` + `scripts/verify_v5.py`
A separate `WFDASH — Error Handler` workflow catches every failure in the main workflow and posts a Telegram alert so nothing falls into a black hole.
---
## Phase 6 — Quote Cards & Color Coding (22 Jun)
**Commit `a18ab0f`**
The first thing a user sees when they open a quote is now genuinely useful:
- **`QuoteEmailCard` overhaul** — proper HTML/plain text body sanitization, automatic linkification, inline PDF/image previews, full metadata display, auth-aware attachment previews via blob URLs
- **`email_received_at`** field added to `quotes` collection + types + setup script
- **`/api/v1/users/lookup`** server endpoint — resolves user IDs to display names
- **User color coding** (`src/lib/user-color.ts`) — stable hash → color so claimed quotes are visually grouped per claimant
- **Quotes list** — tab filters reordered, claimer names shown, rows color-coded per claimant
- **Quote detail page** — claimer info with user icon
- **`DataTable`** — now accepts per-row custom class + style props (powers the color coding)
---
## Phase 7 — AI Quote Item Rewriter (22 Jun)
**Commit `2c6155a`**
The user pastes in a wall of supplier-speak ("1 x ABC pump model XYZ-9000 c/w 2" flanged inlet, 415V, IP55 rated...") and the rewriter turns it into clean, structured line items.
**What landed:**
- **Backend:** `src/app/api/v1/quotes/items/rewrite/route.ts` — 228 lines, calls n8n webhook, validates response, updates PB
- **Frontend:** `src/components/rewrite-quote-modal.tsx` — 716 lines, diff preview, per-item accept/reject
- **n8n:** `WFDASH — Quote Item Rewriter.json` — the workflow definition
- **`user_edited` field** — gates AI rewrites on user-edited content (don't overwrite what the user already cleaned up)
- **Migration:** `scripts/migrations/add-quote-items-user-edited.ts` (idempotent + backfills existing items)
**Bonus features that rode along:**
- **`SupplierSearch` combobox** — searchable combobox using `cmdk`, replaces plain `<Select>` on order + quote pages (216 lines, async load with debounce)
- **`working-days.ts`** utility + aged quote warning badges for overdue quotes
- **Expandable row previews** in the quotes list view
- **DataTable overhaul** — 567 lines changed to support row expansion, custom row styling, server-side state
---
## Phase 8 — Pricing Lock & Quote Polish (22 Jun)
**Commit `86c942a`**
The pricing story gets real:
- **Locked source-of-truth pricing controls** — once locked, cost/selling prices auto-calculate from the source. No more "two people edited the same cell, which wins?"
- Inline cost/selling inputs **removed** from the quote items table — they live in the locked pricing panel now
- **Quote item fields extended:** added `model` and `size`, removed old `type`
- **`/api/v1/quotes/items/rewrite` timeout bumped to 180s** — long AI batches need the headroom
- **n8n rewriter workflow overhauled** — updated prompt + null-safe output handling (won't crash on empty LLM response)
- **Rewrite modal** — sticky headers, better empty chip handling, cleaner styling
---
## Phase 9 — UX Polish (22 Jun)
**Commit `4c0b6b0` — Toasts + hide-completed toggle**
Two quality-of-life wins:
### New Sonner toaster
- `top-center` position with 24px offset (sits below sticky headers, not slammed to the edge)
- Min 360px / max 420px width — toasts have room to breathe
- `rounded-xl border shadow-xl backdrop-blur-md` — looks like a real notification, not a browser alert
- **Brand colors per type** (light + dark variants):
| Type | Light bg | Dark bg | Icon |
|---|---|---|---|
| success | emerald-50 | emerald-900/92% | emerald-600 |
| error | red-50 | red-900/92% | red-600 |
| warning | amber-50 | amber-900/92% | amber-600 |
| info | blue-50 | blue-900/92% | blue-600 |
- Up to 5 visible at once, 4s default duration, action buttons styled with proper tokens
- All 24+ toast call sites get the new look for free — zero migration
### Hide/show completed quote items
- Toggle in the "Quote Items" card header (outline by default, solid indigo when active — deliberately different from the amber Rewrite button and the green done state)
- Incomplete items float to the top, done items sink to the bottom
- **Row numbers stay stable** — done item #5 still says "5.", not "1."
- Celebratory row when everything is done + hidden: `All N items processed` + a Show-completed button
### Memoization
`displayItems` memo handles sort + filter together so the toggle is instant.
---
## The Architecture In One Diagram
```
┌──────────────────── Browser ────────────────────┐
│ Next.js 16 App Router · React 19 · Tailwind 4 │
│ │
│ /quotes /quotes/[id] /quotes/[id]/process │
│ /orders /customers /supplier-map /dashboard │
│ │
│ shadcn/ui · DataTable · Sonner · Leaflet │
└──────────────┬──────────────────┬────────────────┘
│ │
/api/v1/* REST /api/mcp (JSON-RPC)
│ │
┌──────────────▼──────────────────▼────────────────┐
│ PocketBase (single binary) │
│ customers · companies · suppliers · quotes │
│ quote_items · orders · order_items_v2 │
│ collection_pool · delivery_notes · repairs │
│ quote_attachments · quote_replies · counters │
└──────────────▲──────────────────▲────────────────┘
│ │
n8n Code nodes + Admin scripts
AI Agent (MiniMax) (npm run setup)
│ │
┌──────────────┴──────────────────┴────────────────┐
│ n8n Workflows (single scheduler) │
│ • Quote Email Monitor v5 (catch-and-create) │
│ • Quote Item Rewriter (AI batch) │
│ • Error Handler (Telegram alerts) │
│ │
│ Polls IMAP every 5 min · Dedupe by message_id │
└──────────────────────────────────────────────────┘
```
---
## What's Not Done Yet (Parity Backlog)
The Django parity backlog at `docs/django-nextjs-parity-backlog.md` lists the remaining work, in priority order:
| # | Module | Status | What's missing |
|---|---|---|---|
| 1 | Customers | partial/live | Duplicate handling + merge strategy (replace legacy Django mgmt commands) |
| 2 | Companies | partial/live | Address consistency + relation integrity checks |
| 3 | Quotes | partial/live | Email intake **done**; remaining: explicit quote→order safeguards (mitigated by `findOrCreateByMessageId`) |
| 4 | Orders | partial/live | Status board + drv/collection states reflected in UI |
| 5 | Collection pool | partial/live | Assignment UI + filtered work queue |
| 6 | Delivery notes | not migrated | Generate delivery records from delivered/loaded order items |
| 7 | Repairs | not migrated | Rebuild only the useful request/order flow |
| 8 | Stock | intentionally folded | Separate `stock` collection later if needed |
Out of scope (explicit): `stock_orders`, `customer_portal`, `rep_portal`, `internal_stock`, `pos`, `ai_orders`, `system_monitor`. AI / email / monitoring all live in n8n and only talk to the app via API.
---
## The Decisions That Shaped This Build
A few calls we made that you might want to remember for the video:
1. **n8n over a Next.js worker.** Email polling + AI extraction runs in n8n, not in the Next.js process. The app stays stateless; n8n is the single scheduler. Reason: cold-start reliability + the AI Agent node already lives there.
2. **PocketBase over Postgres.** Single binary, no managed cloud, fits Coolify self-hosting. Cost = hosting only.
3. **MCP + REST side-by-side.** REST for humans/scripts (clean auth, clean error codes), MCP for n8n (one endpoint, `tools/list` + `tools/call`). Both go through the same `quotes-service.ts`.
4. **v5 "catch and create" for attachments.** Upload to PB first, link after. The binary is freshest at the top of the workflow; doing it later (when v4 tried) means the buffer is gone. Old node renamed `DISABLED - Upload Attachment` so rollback is one click.
5. **Locked pricing over free-form inputs.** Cost/selling prices live in a locked panel, not on the line item. No more "two people edited the same cell, which wins?"
6. **Vendor ID → stable color hash.** Claimed quotes are color-coded by claimant using a stable hash → Tailwind palette. No "Bob is teal today, indigo tomorrow" drift.
7. **Tests skip cleanly.** Vitest + Playwright both skip what they can't run (no API key → skip, no creds → skip). Fresh checkout gets `1 passed, N skipped`, never a red CI.
---
## Where The Code Lives
```
src/
├── app/
│ ├── (auth)/login/
│ ├── (app)/ # Protected pages
│ │ ├── dashboard/ customers/ companies/
│ │ ├── suppliers/ supplier-map/
│ │ ├── quotes/ # list · [id] · [id]/process · approvals
│ │ ├── orders/ collection-pool/ delivery-notes/ repairs/
│ ├── api/
│ │ ├── mcp/route.ts # JSON-RPC, no auth
│ │ └── v1/ # Bearer auth
│ │ ├── health/ orders/ order-items/ collection-pool/
│ │ └── quotes/ # [id]/pdf · items/rewrite · next-number
│ └── offline.tsx # PWA fallback
├── components/ # data-table, quote-form, supplier-map, etc.
└── lib/
├── pocketbase.ts · pb-helpers.ts · types.ts
├── user-color.ts · quote-status.ts · working-days.ts
└── server/ # agent-auth · orders-service · quotes-service · order-item-status
scripts/ # setup-pocketbase + 4 migrations + 4 smoke tests + 2 v5 deploy scripts
n8n_workfows/ # 5 workflow JSONs + 3 JS snippets + 1 live snapshot
e2e/ # Playwright specs (health, agent-api, auth)
docs/ # 8 design + parity + rollout docs
```
---
## Try It Yourself
```bash
git clone <repo> && cd wfdash
npm install
./pocketbase serve & # in another terminal
cp .env.local.example .env.local # edit ADMIN creds
npm run setup # creates 9 base collections
npm run pb:migrate:orders-v2 # +order_items_v2 +collection_pool
npm run pb:migrate:quote-attachments # +quote_attachments +quote_counters +quote_replies
npm run pb:migrate:quote-items-user-edited # backfills the user_edited field
npm run dev # http://localhost:3000
# Run the tests
npm run test:run # Vitest
npm run e2e # Playwright (skips if no creds)
# Deploy the email pipeline
python scripts/deploy_v5_to_n8n.py # one-shot n8n deploy
python scripts/verify_v5.py # post-deploy sanity check
```
---
*Last updated: 22 Jun 2026 — through commit `4c0b6b0`*