Duplication happens per NOTE, not per batch. The same stack of Vula
invoices was scanned three times in overlapping groupings — 13, 16 and 14
pages with three different file hashes, so upload-time hash dedup could
not see it. Each of those batches mixes notes that promoted, notes
duplicating another batch, and invoices unique to it: vdn074468 alone
carries four that exist nowhere else.
So no batch-level action is correct. Dismissing one wholesale either
discards unique invoices or zeroes a counter for notes already on the
ledger. Adds POST /api/dn-batches/[id]/notes/[noteId]/dismiss, which
stands one note down, refuses to touch a promoted one, and parks the
batch only once nothing is outstanding.
Also fixes the batch dismiss route, which wrote notes_promoted: 0
unconditionally. A batch that had promoted 8 of 13 notes would report
nothing while 8 delivery notes sat in the figures people pay from — the
same board-versus-ledger disagreement already fixed in run.ts. It now
counts what actually reached the ledger and says so in the reason.
929 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Updated the aliasing mechanism to utilize article numbers and barcodes for product identification, addressing issues with ambiguous material codes. The PageItem now requests the article number alongside the product code, ensuring accurate transcription of truncated descriptions. This change improves the resolution process by prioritizing product codes and refining the lookup strategy to reduce ambiguity in purchase orders.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The alias table shipped keyed on material_code alone. Live data says
that cannot work:
- 16 of 24 purchase orders carry ONE material_code across every line.
MEGCPO9239 is "513095" sixteen times.
- Code "580070" spans 24 different products; "513080" spans 16.
An alias keyed on it would have resolved to every line on the order and
been permanently ambiguous — for Builders, the supplier whose truncated
descriptions need aliases most. I verified every PO item HAD a
material_code and never checked whether it discriminated.
Also found, by reading the source scan: the truncation is the supplier's
printer. Builders' form prints descriptions in a fixed ~20-character
column that wraps twice then stops ("Grip Fibreglass Meas" / "uring Tape
Orange a"). Extraction was faithful; nothing upstream can recover it.
But every line also carries an article number and a barcode, which are
never clipped and which we were discarding.
So:
- PageItem gains product_code, and the prompt asks for the Art No /
SKU / Prod Code column plus an explicit instruction to transcribe a
clipped description verbatim rather than guess its ending.
- Aliases record slip_product_code and po_description_key as well.
Lookup tries product code before description; resolution tries the
PO line description, then material_code ONLY when it narrows to a
single line on that order, then falls through to text matching.
- The resolve-item route no longer rejects a PO line for having no
material_code, which was based on the same wrong assumption.
dn_item_aliases had 0 rows, so there was nothing to migrate.
916 tests. Live regression unchanged: 25 matched, 3 ambiguous, 3 none,
zero wrong; 0 of 24 POs vulnerable to nonsense.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Batch 5qzcv6tvgb6cqks promoted 3 of 3 notes and still read needs_attention
because page 4 was a tax invoice. Stray pages become open questions that
can be dismissed, and every detected note must still promote.
Co-Authored-By: MiniMax <noreply@minimax.io>
Five of fourteen blocked batches were tax invoices or collection notes,
which have no delivery note number. Also stops reading a composite
reference like '22433700 / 2025344290' as three separate documents.
Co-Authored-By: MiniMax <noreply@minimax.io>
The script read dn_batch_notes.date, which does not exist on that
collection — the backfill would have silently set nothing while
reporting success. My plan specified the wrong field.
Now sources from delivery_notes.date_received via the batch link (11 of
34), falling back to the scanner-generated filename date (17), leaving 6
empty rather than inventing one. Dry-run by default, and it reports the
source per batch so the numbers can be checked before writing.
Applied: 17 existing fields preserved, uploaded_at added, 28 of 34
batches backfilled. -uploaded_at now sorts newest first.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PocketBase's created is empty on every dn_batches record, making the
board's -created sort a no-op and the tick's oldest-first meaningless.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The earlier commit made the script include these fields for from-scratch
creates, but didn't repair the existing live dn_batch_notes collection.
The repair path (add_fields) covers that case so re-running the script
against an already-deployed PB collection also adds them.
The dn_batches and dn_batch_notes collections were created without
PB's standard 'created' / 'updated' date fields, which broke any
query that sorted on them (the tick route, the inbox page).
Live PB was patched inline by the Task 16 verification; this commit
makes the script that creates these collections include the fields
from the start so a fresh deployment doesn't trip on the same bug.
Rewrites qty_received into the purchase order line's unit, recovering the
slip's own unit from the condition_notes format the OCR callback wrote.
Rows with no recorded unit are left untouched and reported.
The new PWA icon matches the supplied reference: dark background, a
clock face with an orange upper-right arc and hands at 12 and 4
o'clock, sitting on an isometric cube whose two visible faces each
have the MEGA 'M' carved into the background colour, with two small
orange base wedges under the cube.
When the /clock URL is shared (Slack, iMessage, etc.) the static
1200x630 opengraph-image.png is picked up automatically and renders
the icon alongside the 'MEGA Clock' wordmark and tagline, via
openGraph + twitter metadata added to the route.
The migration's docstring pointed at a spec file that was never
written for this feature (the design was worked out via conversation,
not a separate spec doc) — points at the plan instead.
.pyc files are generated, not source -- machine/version-specific,
produce spurious diffs, and a routine __pycache__ cleanup can delete
tracked files as a side effect (hit this earlier in the session). Added
__pycache__/ to .gitignore.
Also fixed a boundary error in the plan's checkbox sweep: Task 17's own
6 steps were skipped when checking off Task 18's range. Task 17 has been
done and independently verified since commit 29ee2cc.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pb-bootstrap-schema.py -- the script a fresh PocketBase instance would
run against -- had fallen ten migrations behind and still described the
pre-restructure world. Rewrote it field-by-field against a live
production dump (17 collections, every field/required-flag/relation/
index verified). Found and fixed two pre-existing gaps unrelated to this
restructure while verifying: fuel_prices and income_claims were entirely
absent from the bootstrap script.
pb-relax-rules.py's collection list updated to match.
No local PocketBase binary is available in this environment, so the
plan's "bootstrap fresh and diff against production" verification step
was replaced with a manual field-by-field comparison against the live
dump -- same goal, different mechanism, documented as a substitution
rather than silently skipped.
Added site scoping to the one agent tool that still lacked it
(list_imports, backed by loadImportBatches) -- every other list_* tool
already had it. Preparation for the future per-user site-allowlist work,
not part of this restructure itself.
Rewrote docs/data-model.md and docs/data-flows.md end to end: the
site-attribution table, the collection reference, the five ingestion
pipelines, and the schema-drift section that was describing the exact
problem this task fixes. Every remaining data_import_batches reference
in both docs is now historical context, not a current description.
This is the last task of the ingestion restructure
(docs/superpowers/plans/2026-09-11-ingestion-restructure.md).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
timesheet_entries, fuel_logs and delivery_notes had no indexes at all --
fine at today's row counts, not once site count grows toward the 18-20
target. Added 8 indexes across fuel_logs, timesheet_entries,
delivery_notes, purchase_orders and plant_hours_log, unioned into each
collection's existing index array rather than replacing it.
Verified both pre-existing UNIQUE indexes (idx_po_number, load-bearing
for the placeholder-PO-number scheme; idx_plant_hours_log_qr_start,
prevents duplicate weekly imports) are byte-for-byte unchanged.
Idempotent: a second run touches nothing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The irreversible step of the ingestion restructure. Guards proved clean
(0 no-site rows across all 8 target collections, PLANT empty, all 27
migrated documents present, no row still depending on the old batch
link), full schema + row-data backups taken to /tmp/pb-backup-2026-09-15
first, then applied.
Dropped project_code from employees, vehicles, fixed_costs,
income_claims, purchase_orders, delivery_notes, fuel_logs and
plant_hours_log. Dropped qty_outstanding from purchase_order_items.
Deleted the PLANT collection (0 rows, 0 code references — a deprecated
alias for vehicles) and DATA_IMPORT_BATCHES (86 rows, superseded by the
five document collections built across Tasks 6-15).
PocketBase refused the first attempt: it will not delete a collection
while any relation still points at it, and fuel_logs, plant_hours_log
and timesheet_entries all still carried a batch_reference relation to
data_import_batches that Task 7's migration made obsolete but never
removed. Confirmed live that nothing in src/ reads or writes that field
any more (only three stale type declarations in pb-types.ts), extended
the drop script to remove it from all three collections, re-ran, and the
deletion succeeded.
Cleaned up the now-dead DataImportBatchRecord interface and BatchType
union from pb-types.ts; BatchStatus survives as the shared type every
document/queue collection's processing_status now uses independently.
Final collection count: 17, down from a peak of 20 mid-restructure (16
original + 3 new document collections - PLANT - data_import_batches -
2 renames counted once each). Every remaining collection's fields match
what code actually reads and writes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
project_code was a text copy of the parent site's code, denormalised
onto 8 child collections that each already have a real site relation.
Written once at insert, never updated -- docs/data-model.md documents
the drift risk. On purchase_orders it never even held a real project
code: production had requisition numbers, site names, and the PO's own
number in that field.
Every read site now resolves the code through site/expand.site/siteById
instead, reusing each page's existing site map where one already
existed. Every create() call stops writing the column. The chat agent's
list_imports tool no longer reads data_import_batches -- it now reads
timesheets, fuel_sheets, plant_hours_sheets, purchase_orders and
delivery_notes filtered by processing_status, which was necessary before
the pool can be dropped (Task 20 would otherwise 404 the agent).
Also closed a Task 17 gap found while doing this: the agent tools' vehicle
and fuel-log handlers still resolved project code from the now-removed
vehicles.project_code column.
Wrote scripts/migrations/2026-09-15-drop-legacy-columns.py, the one
irreversible step in this whole restructure. Its guard against orphaning
fuel_logs rows checked the wrong condition -- "fuel_sheet is empty" --
which produced a false abort: 73 fuel_logs rows have no fuel_sheet
because they predate batch tracking entirely and never had a
batch_reference to begin with, so dropping data_import_batches costs
them nothing. Fixed the guard to check what it actually claims to check:
a row that HAD a batch_reference and was never relinked. Verified live
that no such row exists on any of the three affected collections.
Script run in dry-run only; not yet applied.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tightened site from optional to required on fuel_logs, timesheet_entries,
delivery_notes and purchase_orders now that every route writes it and
every historical row has been backfilled. plant_hours_log was already
required (matches docs/data-model.md's documented invariant, confirmed
against production). timesheets (the document collection) was already
required from Task 6 -- a no-op here, not a regression.
fuel_sheets.site and plant_hours_sheets.site deliberately stay optional:
those sheets can legitimately span multiple sites.
Verified: creating a fuel_logs row with no site now gets a hard 400
(validation_required) from PocketBase itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 13 moved PO creation to upload time and needed a hash-dedup window
matching the spec's "same 30-day dedup window as today" (design doc
section 4.1). purchase_orders has no working created/updated autodate
field on this PocketBase instance, and unlike delivery_notes it had no
other timestamp field either -- the first cut of Task 13 correctly found
this, but responded by dropping the time window entirely rather than
adding one. That's a real behaviour change from spec, not just a
workaround: a PO file legitimately re-sent long after the original would
have been flagged a duplicate forever, with no way to age out.
Added purchase_orders.date_uploaded (additive, mirrors how
delivery_notes already uses its own date_received) and wired it through:
stamped at create, filtered on in the dedup query.
Also brought PurchaseOrderRecord in pb-types.ts in line with what Task 1
already put on the live schema and what the upload route now writes --
processing_status/attempts/last_error/next_retry_at/last_attempt_at/
file_hash/date_uploaded were live in production but absent from the
TypeScript interface. docs/data-model.md section 7 documents this exact
gap as a standing foot-gun; closing it now while it's fresh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Migrates the 27 batches of type timesheets/fuel_logs/plant_hours out of
data_import_batches into the three document collections created in Task
6, preserving every source file and relinking every child row.
Status is derived from actual child count, not copied from the batch
row. Production proved the stored status cannot be trusted: every one of
these batches was written status=processed at creation time, before any
insert ran.
The bug is worse than the plan's own audit found. It predicted 18
phantom-processed batches, all plant_hours. Production has 21 -- three
timesheets batches are also phantom, including two byte-identical
re-uploads of the same wage sheet that both reported success and created
nothing. This hit payroll, not just plant-hour smoke tests.
Also: computed file_hash on migrated documents while the bytes were
already in hand (closes the same dedup gap Task 1 closed on delivery
notes and purchase orders), and split the schema-additive step
(--apply-fields) from the row migration (--apply) so the reversible part
can run independently of the irreversible one. The plan claimed
idempotency via a migrated_from_batch field that doesn't exist; replaced
with an explicit refuse-if-already-populated guard rather than pretend
resumability that isn't there.
27/27 migrated. 6/6 post-migration invariants verified: 3 fuel_sheets,
19 plant_hours_sheets, 5 timesheets, 1367/247/1 children relinked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three document collections replacing data_import_batches for the
spreadsheet importers. Each owns its source file and carries honest
outcome counters (rows_total/created/failed, row_errors) — the counts the
current importers compute and then discard.
site is required on timesheets (a wage sheet belongs to one site) and
optional on the other two, which legitimately span sites since the fuel
importer resolves a site per row. cascadeDelete is false on all three, so
deleting a site cannot destroy its source spreadsheets.
Widened source_file mimeTypes beyond the plan's three entries to match
the pool being replaced. Browsers label CSVs text/csv, application/csv or
text/plain interchangeably and the importers branch on file extension,
not MIME — so the narrower list would have parsed a CSV successfully then
rejected it on write. The 2026-08-24 migration exists because that
already happened once.
Plan corrected: its TypeScript edit added a second `expand` property to
an interface that already had one, which does not compile.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RECEIVED_ITEMS -> DELIVERY_NOTE_ITEMS (55 rows): they are the lines on a
delivery note; the old name described an event, not a thing.
TIMESHEETS -> TIMESHEET_ENTRIES (1367 rows): this collection is the time
log, not a timesheet. The name is freed for the uploaded-document
collection created in Task 6.
PocketBase stores relations by collection id, so every relation survived
untouched; row counts asserted intact on both renames.
Two call sites the plan missed: src/app/api/timesheets/import/route.ts
and src/lib/ocr-result-tier.ts. Deliberately preserved the "timesheets"
string literals in pb-types.ts and the import route — those are
data_import_batches.batch_type select values, not collection names, and
changing them would have violated the select constraint.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The timesheet log carried neither site nor project_code; attribution was
synthesised at read time from employees.site, so moving a worker to
another site silently migrated their entire payroll history with them.
docs/data-model.md flags this as the schema's worst drift risk.
Adds an optional site relation (maxSelect 1, cascadeDelete false so
deleting a site cannot destroy payroll history) and stamps all 1367 rows
from the employee's current site. 1367/1367 written, 0 orphans.
Tightened to required in Task 16, once the import route writes it on
every new row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
147 of 320 fuel_logs had no site relation; their only attribution was the
denormalised project_code text, which Task 18 deletes. Backfilled before
that column goes, resolving each row twice (by project_code and via the
vehicle's own site) and writing only on agreement.
147/147 written, 0 skipped, 0 remaining.
Recorded a caveat the dry run exposed: with one site in production both
resolvers draw from a candidate set of size one, so "both agree" is
guaranteed rather than evidence. The write is still the only possible
correct value, but the guard only bites once more sites exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review found a transient 500 during download landed in the "no stored
file" bucket, so a broken run printed as a clean one and exited 0. Now
counted separately and exits non-zero.
Also: 404 is no longer swallowed on mutations (a PATCH against a deleted
record was counted as written), the target host and mode are printed, the
apply path reports a final written count, and filenames quote with
safe="".
Plan Task 1 Step 4 still carried the superseded copy-from-batch code
block under the rewritten prose, contradicting it. Replaced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Production disproved the copy-from-batch approach in two ways:
- 10 delivery-note batches carry no file_hash at all (they predate
78fa8da), and all 10 sit inside the 30-day dedup window, so copying
would have left a live gap rather than a historical one
- one PO row was produced from two different files with different
hashes via the dedup-by-PO-number branch, so copying gives
last-write-wins on an arbitrary one
Hashing the stored bytes has neither problem and does not depend on the
collection we are about to delete. 56/56 records covered, none skipped.
Immediately surfaced a real duplicate the old path missed: two delivery
notes numbered IN132202 with identical bytes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Updated the operations dashboard to include a rain outlook feature, providing users with a 7-day forecast based on site coordinates. Added latitude and longitude fields to the site form for better weather data integration. Enhanced the fuel log and site management functionalities to utilize dynamic fuel pricing based on historical data. Improved overall data handling and user experience across various dashboard components.
Updated PlantPage to aggregate vehicle data from fuel_logs, including hours worked and total value calculations based on odo readings. Adjusted PlantView to support navigation to vehicle details and improved log display with new data structure. Removed legacy code related to plant_hours_log and refined data handling for better performance and clarity.
The PO upload route (and the delivery-notes and xlsx import
paths) writes the uploaded file into the DATA_IMPORT_BATCHES.
original_file field. That field was bootstrapped with a CSV/Excel
allowlist only:
text/csv, text/plain, application/vnd.ms-excel, application/csv
When the upload route was rewritten on 2026-08-19 (commit f0f7c82)
to write directly to PB instead of forwarding the file to n8n, every
PDF and image upload started failing PB field validation with
validation_invalid_mime_type, surfacing in the API as a 500 with
error='batch_create_failed' and the real cause buried in
pb_data.original_file. The 2026-08-24 migration that should have
added xlsx was also never applied to prod (verified in PB:
xlsx missing too).
Migration: scripts/migrations/2026-08-31-data-import-batches-allow-pdf-images.py
- Strictly additive: unions the new mimeTypes onto the existing
allowlist, exits no-op if already applied (verified idempotent).
- Adds application/pdf, image/jpeg, image/png, image/webp (PDF /
image support for the PO and delivery-note flows) and
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
(xlsx, completing the missed 2026-08-24 migration).
- No code change in the upload routes required - they were already
validating against ALLOWED_MIME in purchase-order-schema.ts and
expecting PB to accept the same set.
Verification on https://pb.mega.infra.apphub.co.za:
Before: POST /api/purchase-orders/upload (PDF) -> HTTP 500,
pb_data.original_file.code = 'validation_invalid_mime_type'.
After: same POST -> HTTP 202 with batch_id, PB row created
(batch_type=purchase_orders, status=pending, file_hash
populated, original_file stored).
Updated the vehicles API to allow PATCH requests with an empty unit_type. This change ensures that existing records with unit_type='' can be updated without validation errors, while maintaining the default value of 'fuel' for new inserts. This adjustment resolves issues with updating pre-existing vehicle records that lack a unit_type field.
53 vehicles predate the unit_type field and have unit_type=''. PB
validates the WHOLE record on any PATCH, so as long as unit_type was
required, no record with an empty value could be updated (Dismiss,
inline-edit, anything calling .update()). Making the field optional
unblocks all 53 immediately.
The default='fuel' still applies to new inserts.
Note: the live PB schema lost its 'default' value during the PATCH
(see review below) — only 'required' was actually changed in the
stored schema. Fresh records will be inserted with unit_type='' (or
null) until the application sets it explicitly. The bootstrap
definition still has default='fuel' so fresh environments get it.
Idempotent migration: scripts/migrations/2026-08-28-vehicles-unit-type-optional.py
applied successfully; re-run is a no-op.
The live PB fixed_costs collection never received the is_recurring field
added in 87bb179 — the bootstrap script is collection-create-only and
errors out on already-existing collections, and no migration added the
field. Smoke-test rows sent is_recurring=false and got it silently
dropped (PB ignores unknown fields on write), then the partition helper
treated the missing value as monthly (its !=== false fallback).
Two changes:
1. Drop 'default: true' from the bootstrap schema definition. A bool
default only applies when the field is absent from the write payload,
so this is belt-and-braces — but it removes any doubt that a default
could ever silently override an explicit false. The partition helper
still treats absent values as monthly, and the backfill migration
patches pre-existing rows to true.
2. Make the backfill migration self-contained: if the field is missing
from the collection, add it (bool, required=false, no default) before
patching rows. Idempotent — no-op if already present.
Co-Authored-By: Claude <noreply@anthropic.com>
Two follow-up fixes from the Task 5 code-quality review:
- scripts/generate-plant-hours-template.ts uses
fileURLToPath(import.meta.url) instead of __dirname. The
project's tsconfig has module: esnext, so the moment the
package flips to ESM the original would throw
ReferenceError: __dirname is not defined.
- src/lib/plant-hours-template.ts documents that the frozen
header row is a best-effort request — SheetJS Community
Edition silently drops the !freeze property (only Pro emits
frozen-pane XML). The function still produces the canonical
13-column workbook, but the doc comment no longer claims a
behaviour that doesn't happen.
buildPlantHoursTemplateWorkbook() emits the canonical 13-column xlsx
with a frozen header row, typed cells (date/time/number), a sample
row using the supplied vehicle, and a Notes column widened for
readability. Used by both /api/plant-hours/template and the seed
script that writes RAW_DOCS/Plant_Hours_Template_2.xlsx.
The plan spec called for default "fuel" on the new select so existing
vehicles rows (which would otherwise fail the required=True
constraint) stay valid. PocketBase applies the default at the field
level when adding it to an existing collection.
Adds the schema needed for the weekly plant-equipment hours log:
- vehicles gets a unit_type select (fuel|plant), defaults to fuel
for backward-compat with existing rows
- plant_hours_log collection with the 13 plant-hours fields plus
vehicle_unit_type_at_import snapshot and composite unique index
(vehicle_qr, start_date)
- data_import_batches.batch_type gains plant_hours
