# JEI South Africa — Build Notes

This file captures decisions, surprises, and parser quirks as we build the JEI invoice tracker.

---

## 2026-09-17 — Discovery phase

### Discovered while inspecting uploaded PDFs

1. **One PDF can hold multiple invoices.**
   `1.Invoice_29.05.pdf` has 4 pages and contains 4 separate invoices (161453, 161454, 161611, 161852). The parser must split per page and detect a new "Invoice No" token.

2. **The buyer's legal entity is constant.**
   Every observed invoice shows `K2023742851 SOUTH AFRICA (PTY) LTD` as the legal buyer. Only the T/A line varies. The user mentioned 3 buyer-name variants — these map to T/A values on top of a constant legal entity.

3. **Deliver-To can be a different company.**
   On several invoices the "Deliver To" is `SHIPLEY GLOBAL EVENTS (PTY) LTD, 678 MIRROR CARD ST, GARSFONTEIN, PRETORIA 0042` — a related entity where goods physically land. Captured as `deliver_to_name` + `deliver_to_address`.

4. **36 files in folder, not all are JEI.**
   FedEx shipping docs and credit notes (in ZAR, not GBP) were moved to `invoices/_non_jei/`. Remaining: 33 JEI files (invoices, proforma, 1 quote).

5. **Empty PDFs observed.**
   `Invoice.pdf` and `JEI Supplier Invoice.pdf` have no extractable text (likely failed exports). Will be flagged in `data/review/unparseable.txt` for re-export.

### Supplier details (captured during inspection)

| Field | Value |
|---|---|
| Supplier legal name | `JEI DRILLING & CUTTING SOLUTIONS LTD` |
| Supplier address | Unit 21 Empire Business Park, Enterprise Way, Burnley, Lancashire, BB12 6LT |
| Supplier phone | 01706 229490 |
| Supplier email | sales@jeisolutions.co.uk |
| Supplier VAT | 227257212 |
| Supplier EORI | GB227257212000 |
| Buyer legal name | `K2023742851 SOUTH AFRICA (PTY) LTD` |
| Buyer VAT | 4450314382 |
| Account ref | `JEISA000` (constant) |
| Currency | GBP only |
| Latest invoice observed | 164912 (date 2026-08-20) |
| Earliest invoice observed | 147842 (date 2025-05-21) |

### Documents in scope vs out of scope (this build)

| Document type | Action |
|---|---|
| JEI invoice | Import as `document_type = invoice` |
| JEI proforma (paid in full) | Import as `document_type = proforma` |
| JEI quote | Import as `document_type = quote` |
| FedEx shipping/credit note | Skip (in `_non_jei/`) |

### User decisions captured

- **Deliver-to:** Capture as separate fields.
- **Quotes:** Treat same as invoices; clean up later against JEI statement.
- **Non-JEI files:** Move to `_non_jei/` subfolder; ignore in importer.

### Parser heuristics (planned)

- **Document boundary:** New page containing `Invoice No` or `Quotation No` token.
- **Document type:** `QUOTATION` → quote; `INVOICE` → invoice; proforma detected via filename keyword or stamp on the page.
- **Buyer variant:** Match T/A line against three known strings.
- **Stock code:** Pull from the table column; if it's just `SERIAL` (e.g. for serial-number line items), skip that line — it's not a stock code.
- **Deliver-to:** Read from the `Deliver To :` block at the bottom of the invoice.

### Open questions / things to watch during build

- Two empty PDFs need user attention (re-export from source).
- One large multi-invoice PDF (`JEI Drilling Invoices.pdf`, 6.4 MB) — verify parser doesn't choke on size.
- Some older invoices use 20% VAT column even though Total VAT is shown as £0.00 (reverse-charge mechanism — exporter doesn't charge VAT). Verify parser handles `Total 20% VAT £ 0.00` gracefully.
- One invoice has both "Customer Order No" and "Sales Order No" fields; both are captured (PO = customer's PO; sales order = supplier's internal ref). The user only mentioned the customer one; we capture both for free.

### Duplicate-file check (user-requested)

Searched all 33 JEI filenames for repeated invoice numbers. Found 4 logical duplicates:

| Invoice No | Files | Hash |
|---|---|---|
| 154923 | `154923 Supplier Invoice.pdf` + `JEI 154923 89.10+102.pdf` | Different (different PDF render) |
| 155655 | `155655 Supplier Invoice.pdf` + `JEI 155655.pdf` | Different |
| 156092 | `156092 Supplier Invoice.pdf` + `JEI 156092.pdf` | Different |
| 161453 | `Invoice 161453 from JEI... - 14_04_2026.pdf` + `Invoice 161453.pdf` | Different |

Verified by reading both files for invoice 154923 — same data, different PDF layout (one is flow-text, one is more grid-aligned). Not byte-identical files, but **identical invoice data**.

**Dedup strategy:** the importer relies on the `JEI_invoices.invoice_no + document_type` uniqueness — second occurrence skipped silently. The first one imported wins (its `pdf_file` is kept). User can later delete the duplicate filename's reference via PB admin if needed (the duplicate file's data is merged into the first record).

**Impact:** no schema change needed. Idempotency rule (§5.2) already covers this.

### Additional observations from invoice 154923

- "Customer Order No" field on this invoice reads literally `Email 28.07.25` (looks like a separate "Email [date]" reference the supplier uses). The importer will capture it as-is. User can clean later.
- The invoice is stamped **PAID IN FULL**. Some invoices are pre-paid (proforma-paid or settled-on-delivery). We don't track this in the schema yet (future phase) but it's visible on the original PDF.

## 2026-09-17 — Build phase

### Major wins
- ✅ Next.js 14 + Tailwind app on **http://localhost:6003** (changed from 6002 on 2026-09-18 to avoid clash with Workflow's CIPC service on :6002)
- ✅ PB collections: JEI_invoices (15 fields + id), JEI_invoice_lines (8 fields + relations), JEI_stock_codes (4 fields + id) — 30 fields total
- ✅ PDF extractor parses 38 records from 33 JEI supplier PDFs (skips `_non_jei/`)
- ✅ Importer pushed 24 invoices + 70 stock codes + 118 lines into PB (idempotent — re-runs skip duplicates)
- ✅ All 5 UI pages return 200 with real data

### Critical PB v0.39.8 API quirks discovered

1. **POST /api/collections uses key `fields` not `schema`.** Docs show `schema` but the API silently accepts `schema` and stores under `fields`. Always send `fields: [...]`.

2. **Field options go at the FIELD ROOT, not inside `options`.** This is the opposite of what docs for older PB versions show. e.g.
   - ✅ WORKS: `{ name: "foo", type: "select", maxSelect: 1, values: ["a","b"] }`
   - ❌ FAILS: `{ name: "foo", type: "select", options: { maxSelect: 1, values: ["a","b"] } }` — returns `fields.X.values.code: validation_required, Cannot be blank.` even though the values array is non-empty.
   - Same applies to relation fields: `collectionId` and `cascadeDelete` go at the field root, not in `options`.

3. **GET response hides field options.** After creating a collection, `GET /api/collections/<id>` returns `fields` with `options: {}` (empty) for select/relation fields. The data IS stored — just not returned. PATCH the same field with explicit options to confirm (it succeeds).

4. **PATCH round-trip "wakes up" newly created collections.** Same trick as the existing memory entry — if a collection 403s on read or returns empty fields, PATCHing any rule field (even with the existing value) often fixes it.

### Parser quirks remaining
- 4 unparseable PDFs (empty text — likely failed exports: Invoice.pdf, JEI Supplier Invoice.pdf, JEI SA Invoice 147842.pdf, JEI SA Proforma 148931.pdf).
- Some multi-page PDFs (151778, 151779) have page-boundary issues — the parser creates multiple records from the same physical invoice when it shouldn't.
- "JEISA000" got imported as a stock code (it's actually the buyer's account reference) — minor data issue, can be deleted via admin UI.
- 161453, 161852: lines parsed as 0 (one of the page variants has a different layout that the parser doesn't recognize).

### Token expiry
PB token expires **2026-09-18 19:10 SAST** (~22h from now). Re-grab from DevTools → Application → Local Storage → `pocketbase_auth.token` on the PB admin UI when needed.

### File layout (final)
```
JEI/
├── .env.local                    # PB_URL, PB_TOKEN, NEXT_PUBLIC_PB_URL
├── .gitignore
├── NOTES.md
├── package.json                  # next 14, pocketbase, tailwind, etc.
├── tsconfig.json
├── next.config.js
├── tailwind.config.ts
├── postcss.config.js
├── app/
│   ├── layout.tsx                # header with nav
│   ├── page.tsx                  # dashboard
│   ├── invoices/page.tsx         # list with search
│   ├── invoices/[id]/page.tsx    # detail with PDF link
│   ├── stock-codes/page.tsx      # list with stats
│   ├── stock-codes/[code]/page.tsx  # detail with avg/min/max
│   ├── globals.css
│   └── lib/{pb.ts, types.ts}
├── scripts/
│   ├── create_collections.py     # creates 3 JEI_* collections
│   ├── extract_invoices.py       # PDF → data/review/all.json
│   ├── import_to_pb.py           # all.json → PB (idempotent)
│   └── patch_collections.py      # legacy workaround (not needed anymore)
├── invoices/                     # user drops PDFs here
├── data/review/all.json          # parsed records
└── docs/superpowers/specs/...    # design spec
```

### How to use
1. Server runs on `http://localhost:6003` via `pnpm dev` (or `node_modules/.bin/next.cmd dev -p 6003`).
2. To re-extract: `python scripts/extract_invoices.py` (reads all PDFs in `invoices/`, writes `data/review/all.json`).
3. To re-import: `python scripts/import_to_pb.py` (skips existing invoice_no + document_type pairs).
4. To refresh PB token: visit `https://pb.agent.apphub.co.za/_/`, copy `pocketbase_auth.token` from DevTools → Application → Local Storage, paste into `JEI/.env.local`.