From 97d0339aa64ecfed69b38a85bb9c7f55b9be6b69 Mon Sep 17 00:00:00 2001 From: Mavis Date: Thu, 17 Sep 2026 21:35:07 +0200 Subject: [PATCH] notes: duplicate file findings + dedup strategy --- NOTES.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/NOTES.md b/NOTES.md index 211b60c..e188229 100644 --- a/NOTES.md +++ b/NOTES.md @@ -69,3 +69,25 @@ This file captures decisions, surprises, and parser quirks as we build the JEI i - One large multi-invoice PDF (`JEI Drilling Invoices.pdf`, 6.4 MB) — verify parser doesn't choke on size. - Some older invoices use 20% VAT column even though Total VAT is shown as £0.00 (reverse-charge mechanism — exporter doesn't charge VAT). Verify parser handles `Total 20% VAT £ 0.00` gracefully. - One invoice has both "Customer Order No" and "Sales Order No" fields; both are captured (PO = customer's PO; sales order = supplier's internal ref). The user only mentioned the customer one; we capture both for free. + +### Duplicate-file check (user-requested) + +Searched all 33 JEI filenames for repeated invoice numbers. Found 4 logical duplicates: + +| Invoice No | Files | Hash | +|---|---|---| +| 154923 | `154923 Supplier Invoice.pdf` + `JEI 154923 89.10+102.pdf` | Different (different PDF render) | +| 155655 | `155655 Supplier Invoice.pdf` + `JEI 155655.pdf` | Different | +| 156092 | `156092 Supplier Invoice.pdf` + `JEI 156092.pdf` | Different | +| 161453 | `Invoice 161453 from JEI... - 14_04_2026.pdf` + `Invoice 161453.pdf` | Different | + +Verified by reading both files for invoice 154923 — same data, different PDF layout (one is flow-text, one is more grid-aligned). Not byte-identical files, but **identical invoice data**. + +**Dedup strategy:** the importer relies on the `JEI_invoices.invoice_no + document_type` uniqueness — second occurrence skipped silently. The first one imported wins (its `pdf_file` is kept). User can later delete the duplicate filename's reference via PB admin if needed (the duplicate file's data is merged into the first record). + +**Impact:** no schema change needed. Idempotency rule (§5.2) already covers this. + +### Additional observations from invoice 154923 + +- "Customer Order No" field on this invoice reads literally `Email 28.07.25` (looks like a separate "Email [date]" reference the supplier uses). The importer will capture it as-is. User can clean later. +- The invoice is stamped **PAID IN FULL**. Some invoices are pre-paid (proforma-paid or settled-on-delivery). We don't track this in the schema yet (future phase) but it's visible on the original PDF.