David/MEGAINFRA-WEBAPP
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(schema): bring bootstrap and data-model docs back to truth

Browse source 
pb-bootstrap-schema.py -- the script a fresh PocketBase instance would
run against -- had fallen ten migrations behind and still described the
pre-restructure world. Rewrote it field-by-field against a live
production dump (17 collections, every field/required-flag/relation/
index verified). Found and fixed two pre-existing gaps unrelated to this
restructure while verifying: fuel_prices and income_claims were entirely
absent from the bootstrap script.

pb-relax-rules.py's collection list updated to match.

No local PocketBase binary is available in this environment, so the
plan's "bootstrap fresh and diff against production" verification step
was replaced with a manual field-by-field comparison against the live
dump -- same goal, different mechanism, documented as a substitution
rather than silently skipped.

Added site scoping to the one agent tool that still lacked it
(list_imports, backed by loadImportBatches) -- every other list_* tool
already had it. Preparation for the future per-user site-allowlist work,
not part of this restructure itself.

Rewrote docs/data-model.md and docs/data-flows.md end to end: the
site-attribution table, the collection reference, the five ingestion
pipelines, and the schema-drift section that was describing the exact
problem this task fixes. Every remaining data_import_batches reference
in both docs is now historical context, not a current description.

This is the last task of the ingestion restructure
(docs/superpowers/plans/2026-09-11-ingestion-restructure.md).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
DEls 2026-09-11 16:17:41 +02:00
parent 530d837bd6
commit 8eb256360b
8 changed files with 611 additions and 345 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

328
docs/data-flows.md
View file

@ -2,21 +2,37 @@
> End-to-end ingestion pipelines with sequence diagrams and the exact site-attribution strategy each one uses. > End-to-end ingestion pipelines with sequence diagrams and the exact site-attribution strategy each one uses.
Every importer writes into a `data_import_batches` row, which carries the Reflects the shape produced by the 2026-09-11 ingestion restructure
original file for the audit trail and the resulting `processed_record_id`. (`docs/superpowers/plans/2026-09-11-ingestion-restructure.md`). The shared
Three of the pipelines are synchronous (the file is parsed and inserted in `data_import_batches` pool is gone. Two pipeline shapes replaced it:
the same request) and two are async (a separate worker, n8n, processes
them and POSTs results back). - **The three sheet imports** (timesheets, fuel logs, plant hours) are
still synchronous, but each now creates its own typed "uploaded document"
row (`timesheets`, `fuel_sheets`, `plant_hours_sheets`) instead of a
generic batch row, and computes its real outcome via
`resolveDocumentOutcome()` (`src/lib/document-status.ts`) once parsing is
done — never hardcoding `status: processed` before a single row exists.
- **Purchase orders** moved from synchronous-upload-then-async-batch to
fully async against `purchase_orders` itself: the row is born at upload
time with a placeholder number and `processing_status: pending`, and an
n8n cron claims, extracts, and fills it in. There is no batch row at any
point in this pipeline any more.
- **Delivery notes** keep their existing shape (site + file stamped at
upload, OCR result posted back by webhook) minus the batch row.
--- ---
## 1. Timesheet import (synchronous) ## 1. Timesheet import (synchronous, document-first)
**Route:** `POST /api/timesheets/import` (`src/app/api/timesheets/import/route.ts`) **Route:** `POST /api/timesheets/import` (`src/app/api/timesheets/import/route.ts`)
**Input:** `multipart/form-data` with `file` (xlsx / xls / csv) plus optional `siteId` and `createMissingEmployees` form fields **Input:** `multipart/form-data` with `file` (xlsx / xls / csv), **required** `siteId`, and optional `createMissingEmployees`
**Output:** `{ batchId, totalRows, imported, skipped, errors, detectedFormat, createdEmployees }` **Output:** `{ documentId, totalRows, imported, skipped, errors, detectedFormat, createdEmployees }`
**Triggers:** `ImportTimesheetsButton` on the Workforce page **Triggers:** `ImportTimesheetsButton` on the Workforce page
Unlike the fuel/plant-hours sheets, a wage sheet always belongs to exactly
one site — `siteId` is required and the upload is rejected before anything
is parsed if it's missing.
**Format auto-detection** (the route walks every sheet, first match wins): **Format auto-detection** (the route walks every sheet, first match wins):
1. **Hills Casual Matrix** — `looksLikeCasualMatrix(rows)` (`src/lib/timesheet-casual-parse.ts`) → one long row per (employee, day). Hours are `HH:MM`, "Absent" cells are skipped, names with leading commas are cleaned. 1. **Hills Casual Matrix** — `looksLikeCasualMatrix(rows)` (`src/lib/timesheet-casual-parse.ts`) → one long row per (employee, day). Hours are `HH:MM`, "Absent" cells are skipped, names with leading commas are cleaned.
@ -30,39 +46,48 @@ sequenceDiagram
participant U as User (browser) participant U as User (browser)
participant R as /api/timesheets/import participant R as /api/timesheets/import
participant P as PocketBase participant P as PocketBase
U->>R: multipart { file, siteId?, createMissingEmployees? } U->>R: multipart { file, siteId (required), createMissingEmployees? }
R->>R: parseFile() → uniform long rows R->>R: parseFile() → uniform long rows
alt createMissingEmployees && siteId R->>P: create(timesheets, site=siteId, source_file, file_hash, processing_status=pending)
alt createMissingEmployees
R->>P: getFullList(employees, fields="id,employee_no") R->>P: getFullList(employees, fields="id,employee_no")
R->>P: create(missing employees, casual, site=siteId) R->>P: create(missing employees, casual, site=siteId)
end end
R->>P: create(data_import_batches, batch_type=timesheets, original_file) R->>P: getFullList(employees, fields="id,employee_no,site")
R->>P: getFullList(employees, fields="id,employee_no")
loop for each parsed row loop for each parsed row
R->>P: create(timesheets, employee=lookup[row.employee_no]) R->>P: create(timesheet_entries, employee=lookup[row.employee_no], site=employeeSite.get(employeeId) || siteId)
end end
R-->>U: { batchId, totalRows, imported, skipped, errors, ... } R->>R: resolveDocumentOutcome({ rowsTotal, rowsCreated, rowErrors }, now)
R->>P: update(timesheets doc, processing_status, rows_total, rows_created, rows_failed, row_errors, last_error)
R-->>U: { documentId, totalRows, imported, skipped, errors, ... }
``` ```
**Site attribution.** Timesheets do not carry a `site` field. The route **Site attribution.** `timesheet_entries` (the collection that used to be
writes the row against the employee record; site attribution happens at called `timesheets` before the rename) now carries a **required** `site`
*read time* in the metrics helpers by joining through relation, stamped from the employee's site at the moment of import — not
`employeeRates.get(timesheet.employee).site`. This is the only joined at read time. This is a deliberate change: the old at-read-time join
collection in the system with no `site` or `project_code` on the row itself. through `employees.site` meant a worker moving sites silently rewrote the
site attribution of every historical hour they'd ever logged. Stamping at
write time freezes each entry's site to what it actually was that day.
**Failure behaviour.** **Failure behaviour.**
- Unknown `employee_no` (and `createMissingEmployees` not set) → row errored, import continues. - Unknown `employee_no` (and `createMissingEmployees` not set) → row errored, import continues.
- Per-row `pb.collection("timesheets").create` failure → row errored, import continues. - Per-row `pb.collection("timesheet_entries").create` failure → row errored, import continues.
- File parse failure → 400 with a top-level `error` and `details`. - File parse failure → 400 with a top-level `error` and `details`.
- `resolveDocumentOutcome()` marks the document `failed` (not `processed`)
if the file had zero data rows, or if parsing produced rows but zero of
them landed — see `src/lib/document-status.ts`. A partial import (some
rows landed, some failed) is `processed` with `rows_failed > 0` and
`last_error` summarising the count; it is not silently reported clean.
--- ---
## 2. Fuel log import (synchronous) ## 2. Fuel log import (synchronous, document-first)
**Route:** `POST /api/fuel-logs/import` (`src/app/api/fuel-logs/import/route.ts`) **Route:** `POST /api/fuel-logs/import` (`src/app/api/fuel-logs/import/route.ts`)
**Input:** `multipart/form-data` with `file` (xlsx or csv) **Input:** `multipart/form-data` with `file` (xlsx or csv), optional `siteId`
**Output:** `{ batchId, totalRows, imported, skipped, errors, qrMatched, descMatched }` **Output:** `{ documentId, totalRows, imported, skipped, errors, qrMatched, descMatched }`
**Triggers:** `ImportFuelLogsButton` on the Fleet page **Triggers:** `ImportFuelLogsButton` on the Fleet page
**Expected columns** (case-insensitive, alias-resolved by `resolveColumn` in `src/lib/fuel-import-parse.ts`): **Expected columns** (case-insensitive, alias-resolved by `resolveColumn` in `src/lib/fuel-import-parse.ts`):
@ -84,51 +109,56 @@ sequenceDiagram
participant U as User (browser) participant U as User (browser)
participant R as /api/fuel-logs/import participant R as /api/fuel-logs/import
participant P as PocketBase participant P as PocketBase
U->>R: multipart { file } U->>R: multipart { file, siteId? }
R->>R: SheetJS parse → rowsRaw R->>R: SheetJS parse → rowsRaw
R->>P: create(data_import_batches, batch_type=fuel_logs, original_file, status=processed) R->>P: create(fuel_sheets, site=siteId?, source_file, file_hash, processing_status=pending)
R->>P: getFullList(vehicles, fields="id,description,project_code,qr_code_id") R->>P: getFullList(vehicles, fields="id,description,site,qr_code_id")
R->>P: getFullList(sites, fields="id,name,project_code") R->>P: getFullList(sites, fields="id,name")
R-->>R: vehicleByQr, vehicleByDesc, siteByName maps R-->>R: vehicleByQr, vehicleByDesc, siteByName maps
loop for each row loop for each row
R->>R: parseDate, toNumber R->>R: parseDate, toNumber
R->>R: vehicle = vehicleByQr[qr] ?? vehicleByDesc[desc] R->>R: vehicle = vehicleByQr[qr] ?? vehicleByDesc[desc]
R->>R: site = siteByName[row.site.toLowerCase()] R->>R: rowSiteId = siteByName[row.site.toLowerCase()] ?? vehicle.site
R->>R: projectCode = site?.projectCode ?? vehicle.projectCode R->>P: create(fuel_logs, site=rowSiteId, vehicle, fuel_sheet=doc.id, ...)
R->>P: create(fuel_logs, site, vehicle, project_code, ...)
end end
R-->>U: { batchId, totalRows, imported, skipped, errors, qrMatched, descMatched } R->>R: resolveDocumentOutcome({ rowsTotal, rowsCreated, rowErrors }, now)
R->>P: update(fuel_sheets doc, processing_status, rows_total, rows_created, rows_failed, row_errors, last_error)
R-->>U: { documentId, totalRows, imported, skipped, errors, qrMatched, descMatched }
``` ```
**Site attribution strategy.** Two-tier match: **Site attribution strategy.** Two-tier match, same shape as before the
restructure, but the destination is a real required `site` relation on
`fuel_logs` instead of a denormalised `project_code` fallback:
1. The row's `Current Site` cell is matched case-insensitive against 1. The row's `Current Site` cell is matched case-insensitive against
`sites.name`. If found, the site relation and its `project_code` are `sites.name`. If found, that site is stamped on the row.
stamped on the row. 2. If the row has no site cell (legacy CSVs) or no match, the row falls
2. If the row has no site cell (legacy CSVs), the row's `project_code` back to the matched vehicle's current `site`.
is set to the matched vehicle's `project_code` and the `site` relation
is left **empty**. The fuel log exists but isn't tied to a site by
relation — the only link is the text `project_code`.
This is the seam where **site attribution goes wrong** as the fleet grows — This is still the seam where **site attribution goes wrong** as the fleet
see [multi-site-assessment §4](./multi-site-assessment.md#4-site-resolution-by-name-string). grows — see [multi-site-assessment §4](./multi-site-assessment.md#4-site-resolution-by-name-string).
Two sites with near-identical names, a renamed site, or a vehicle moved Two sites with near-identical names, a renamed site, or a vehicle moved
between sites all produce silently wrong attribution. between sites all still produce silently wrong attribution; what changed is
that there is no separate `project_code` fallback path to reason about any
more — one relation, one resolution order.
**Failure behaviour.** **Failure behaviour.**
- Unknown vehicle (no QR match, no description match) → row errored, import continues. - Unknown vehicle (no QR match, no description match) → row errored, import continues.
- Invalid date / odo / litres → row errored, import continues. - Invalid date / odo / litres → row errored, import continues.
- PB insert failure → row errored, import continues; the importer extracts - PB insert failure → row errored, import continues; the importer extracts
the field-level validation message via `formatPbError` (lines 65-78). the field-level validation message via `formatPbError`.
- `resolveDocumentOutcome()` governs the document's final status exactly as
in the timesheet pipeline (§1) — zero rows parsed or zero rows landed is
`failed`, never `processed`.
--- ---
## 3. Plant hours import (synchronous) ## 3. Plant hours import (synchronous, document-first)
**Route:** `POST /api/plant-hours/import` (`src/app/api/plant-hours/import/route.ts`) **Route:** `POST /api/plant-hours/import` (`src/app/api/plant-hours/import/route.ts`)
**Input:** `multipart/form-data` with `file` (xlsx or csv, max 10 MB) **Input:** `multipart/form-data` with `file` (xlsx or csv, max 10 MB), optional `siteId`
**Output:** `{ batchId, totalRows, imported, skipped, mismatched, warnings, errors }` **Output:** `{ documentId, totalRows, imported, skipped, mismatched, warnings, errors }`
**Orchestrator:** `importPlantHours` in `src/lib/plant-hours-import.ts` **Orchestrator:** `importPlantHours` in `src/lib/plant-hours-import.ts`
**Triggers:** `ImportPlantHoursButton` on the Plant page **Triggers:** `ImportPlantHoursButton` on the Plant page
@ -140,61 +170,70 @@ sequenceDiagram
participant R as /api/plant-hours/import participant R as /api/plant-hours/import
participant O as importPlantHours (pure) participant O as importPlantHours (pure)
participant P as PocketBase participant P as PocketBase
U->>R: multipart { file } U->>R: multipart { file, siteId? }
R->>R: SheetJS parse R->>R: SheetJS parse
R->>P: create(data_import_batches, batch_type=plant_hours, original_file, status=processed) R->>P: create(plant_hours_sheets, site=siteId?, source_file, file_hash, processing_status=pending)
R->>P: getFullList(vehicles, fields="id,description,project_code,qr_code_id,unit_type,site,cost_per_hour", perPage=5000) R->>P: getFullList(vehicles, fields="id,description,qr_code_id,unit_type,site,cost_per_hour", perPage=5000)
R->>P: getFullList(sites, fields="id,name,project_code", perPage=5000) R->>P: getFullList(sites, fields="id,name", perPage=5000)
R->>O: importPlantHours(rowsRaw, batchId, { vehicles, sites }) R->>O: importPlantHours(rowsRaw, { vehicles, sites })
O->>O: resolve each row by qr_code_id (or desc fallback) O->>O: resolve each row by qr_code_id (or desc fallback)
O->>O: compute hours, value, over_budget, hours_mismatch O->>O: compute hours, value, over_budget, hours_mismatch
O->>O: snapshot vehicle_unit_type_at_import O->>O: snapshot vehicle_unit_type_at_import
loop for each prepped row loop for each prepped row
R->>P: create(plant_hours_log, ...) R->>P: create(plant_hours_log, site=vehicle.site, plant_hours_sheet=doc.id, ...)
alt duplicate (vehicle_qr, start_date) alt duplicate (vehicle_qr, start_date)
P-->>R: 400 validation_not_unique P-->>R: 400 validation_not_unique
R->>R: record as "duplicate: vehicle_qr=X start_date=Y" R->>R: record as "duplicate: vehicle_qr=X start_date=Y"
end end
end end
R-->>U: { batchId, totalRows, imported, skipped, mismatched, warnings, errors } R->>R: resolveDocumentOutcome({ rowsTotal, rowsCreated, rowErrors }, now)
R->>P: update(plant_hours_sheets doc, processing_status, rows_total, rows_created, rows_failed, row_errors, last_error)
R-->>U: { documentId, totalRows, imported, skipped, mismatched, warnings, errors }
``` ```
**Site attribution.** The orchestrator pulls the site from the matched **Site attribution.** The orchestrator pulls the site from the matched
vehicle (`vehicle.site`) and stamps both `site` and `project_code` on vehicle (`vehicle.site`) and stamps it on the row. This remains the
the row. This is the **only collection where `site` is required **only collection where `site` is required (`minSelect=1`)** on the log
(`minSelect=1`)** — every plant_hours_log row has a non-null site. itself — every `plant_hours_log` row has a non-null site.
**Composite unique index.** `idx_plant_hours_log_qr_start` on **Composite unique index.** `idx_plant_hours_log_qr_start` on
`(vehicle_qr, start_date)` blocks accidental double-imports of the same `(vehicle_qr, start_date)` blocks accidental double-imports of the same
week for the same vehicle. The route surfaces the PB-level rejection as week for the same vehicle. The route surfaces the PB-level rejection as
a friendly `duplicate: ...` error per row, not a batch-level failure. a friendly `duplicate: ...` error per row, not a document-level failure.
See [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) for the cross-site implication See [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) for the cross-site implication
of this index. of this index.
**Failure behaviour.** Per-row. The orchestrator collects row-level **Failure behaviour.** Per-row. The orchestrator collects row-level
errors and warnings; the route appends PB insert errors. The batch errors and warnings; the route appends PB insert errors. The document
itself is never aborted by one bad row. itself is never aborted by one bad row — `resolveDocumentOutcome()` decides
the final status the same way as §1/§2.
**Exception report.** `GET /api/plant-hours/exception-report/[batchId]` **Exception report.** `GET /api/plant-hours/exception-report/[sheetId]`
reconstructs a per-row exception CSV by re-parsing the source file reconstructs a per-row exception CSV by re-parsing the source file
from the batch's `original_file` and matching against the imported rows from the document's `source_file` and matching against the imported rows
in `plant_hours_log.filter('batch_reference = "<batchId>"')`. Used to in `plant_hours_log.filter('plant_hours_sheet = "<sheetId>"')`. Used to
re-export a corrected sheet when the operator needs to fix and re-import. re-export a corrected sheet when the operator needs to fix and re-import.
--- ---
## 4. Purchase order upload → ingest (async via n8n) ## 4. Purchase order upload → ingest (async, no batch row)
Three routes cooperate. The browser never talks to n8n directly. Two routes cooperate; the browser never talks to n8n directly. There is no
longer a poll-for-status route, a separate batch collection, or a "move the
file onto the new row" step — the `purchase_orders` row *is* the queue item
from the moment it's created.
**Routes:** **Routes:**
- `POST /api/purchase-orders/upload` — `src/app/api/purchase-orders/upload/route.ts` - `POST /api/purchase-orders/upload` — `src/app/api/purchase-orders/upload/route.ts`
- `POST /api/purchase-orders/process-batch` — called by n8n — `src/app/api/purchase-orders/process-batch/route.ts` - `POST /api/purchase-orders/process-batch` — called by n8n — `src/app/api/purchase-orders/process-batch/route.ts`
- `GET /api/data-import-batches/[id]/status` — poll target for the browser
**Auth model.** All three are server-side; only `process-batch` requires Full implementation detail, including the n8n node map, hardcoded values,
the shared `N8N_INGEST_SECRET` header. The other two run as the and a manual smoke-test script, lives in
PocketBase superuser. `docs/n8n/purchase-order-ingest-implementation.md` (rewritten for this
shape in Task 15 of the restructure) — this section is the short version.
**Auth model.** Both routes are server-side; `process-batch` additionally
requires the shared `N8N_INGEST_SECRET` header.
**Pipeline:** **Pipeline:**
@ -203,66 +242,76 @@ sequenceDiagram
participant U as User participant U as User
participant W as /api/purchase-orders/upload participant W as /api/purchase-orders/upload
participant P as PocketBase participant P as PocketBase
participant N as n8n (PO ingest workflow) participant N as n8n (PO ingest cron, every ~3 min)
participant B as /api/purchase-orders/process-batch participant B as /api/purchase-orders/process-batch
participant B2 as /api/data-import-batches/:id/status U->>W: multipart { file (PDF/image), siteId }
U->>W: multipart { file (PDF/image) }
W->>W: SHA-256 the bytes W->>W: SHA-256 the bytes
W->>P: getFullList(batches, file_hash=X && status=processed && date_uploaded >= 30d_ago) W->>P: getFullList(purchase_orders, file_hash=X && date_uploaded >= 30d_ago)
alt duplicate alt duplicate
W-->>U: 200 { duplicate: true, existing_batch_id, ... } W-->>U: 200 { duplicate: true, existing_id, ... }
else new else new
W->>P: create(data_import_batches, batch_type=purchase_orders, status=pending, file_hash) W->>P: create(purchase_orders, site=siteId, purchase_order_number="PENDING-...", processing_status=pending, file_hash, purchase_order_file)
W-->>U: 202 { batch_id, status: queued } W-->>U: 202 { purchase_order_id, status: queued }
end end
N->>P: poll /api/files/data_import_batches/:id/<filename> (admin auth) N->>P: PATCH purchase_orders/{id}?filter=processing_status="pending" (atomic claim → processing, attempts+1)
N->>N: AI Agent extracts PO fields + line items N->>P: GET /files/purchase_orders/{id}/{purchase_order_file}
N->>B: POST { batch_id, extracted } + x-ingest-secret N->>N: AI Agent extracts PO header + line items
B->>P: getOne(batch) — must be in 'processing' N->>B: POST { purchase_order_id, extracted } + x-ingest-secret
B->>P: getFullList(purchase_orders, filter=po_number=...) — dedup B->>P: getOne(purchase_orders, id) — must still be 'processing'
B->>B: matchSite(extracted.project_code) → siteId B->>P: getFullList(purchase_orders, filter=purchase_order_number=extracted.number) — dedup
B->>P: download batch.original_file via /api/files alt fill in (no existing PO with that number)
B->>P: create(purchase_orders, file=..., site=siteId, raw_ocr_data, is_verified=false) B->>P: update(this row, purchase_order_number=extracted.number, processing_status=processed, raw_ocr_data, is_verified=false)
loop for each line item loop for each line item
B->>P: create(purchase_order_items, purchase_order=poRecord.id) B->>P: create(purchase_order_items, purchase_order=this row.id)
end end
alt dedup branch else merge (a real PO with that number already exists)
B->>P: update(batch, status=processed, processed_record_id=existing.id) // keep original_file B->>P: update(existing PO with any extracted fields it was missing)
else success branch B->>P: delete(this placeholder row)
B->>P: update(batch, status=processed, processed_record_id=poRecord.id, original_file="") else invalid extraction
B->>P: update(this row, processing_status=failed, last_error) // never deleted — stays visible and recoverable
end end
U->>B2: GET /api/data-import-batches/:id/status B-->>N: { ok: true } or 4xx/5xx
B2-->>U: { id, status, processed_record_id, processed_record_label, ... }
``` ```
**Three dedup layers, in order:** **Three-outcome `process-batch` contract.** This is the core of the async
design — one route, three possible endings for the *same* placeholder row,
never a fourth row created:
1. **Fill in.** No existing PO shares the extracted number — the
placeholder row is updated in place with the real number, OCR payload,
and line items. This is the common case.
2. **Merge / dedup / delete placeholder.** A real PO with that number
already exists (a genuine duplicate upload, or two near-simultaneous
uploads of the same document). The existing PO is patched with anything
useful the new extraction found that it was missing, and *this*
placeholder row is deleted. The caller-visible id becomes the winner's.
3. **Mark failed.** The extraction was invalid (no PO number, garbage
output) — the row moves to `processing_status=failed` with `last_error`
set. It is **never deleted** on this path, so it stays visible and
recoverable in the dashboard, and the n8n-side escalation logic can
retry or eventually mark it `dead` after repeated failures.
**Dedup layers, in order:**
1. **Hash dedup** at upload time — SHA-256 of the file bytes, matched 1. **Hash dedup** at upload time — SHA-256 of the file bytes, matched
against batches in the last 30 days with `status=processed`. Returns against `purchase_orders` rows created in the last 30 days. Returns
`200 { duplicate: true }` before any new batch row is written. `200 { duplicate: true }` before any new placeholder row is written.
2. **AI-extracted PO number dedup** at process time — 2. **Extracted-number dedup** at process time — outcome 2 above.
`getFullList(purchase_orders, filter=po_number="...")`. If found,
the existing PO id is recorded on the batch but **the duplicate file
is kept on the batch row** for audit (`clearFile: false`).
3. **PB unique index** `idx_po_number` on 3. **PB unique index** `idx_po_number` on
`purchase_orders(purchase_order_number)` — the final backstop. If `purchase_orders(purchase_order_number)` — the final backstop. If the
the AI dedup misses, this rejects the insert. dedup logic misses, this rejects the insert.
**Site attribution strategy.** `matchSite(pb, payload.project_code)` in **Site attribution strategy.** `matchSite(pb, siteId or project code hint)`
`src/lib/purchase-order-schema.ts`: in `src/lib/purchase-order-schema.ts` resolves the site once, at upload
time, before OCR ever runs — `purchase_orders.site` is a required field, so
there is no unmatched/null-site PO state to reason about downstream the way
there used to be.
1. Exact match on `sites.project_code`. **Failure behaviour.** Per the `process-batch` route's own contract, it
2. Fuzzy match `project_code ~ "X"` (PB's `~` operator does case-insensitive substring). distinguishes `retryable: true` (5xx with a transient signal — n8n retries)
3. If both miss, the PO is stored with `site=null` and a warning is from `retryable: false` (400 — n8n marks the row `dead` after enough
surfaced: `unmatched_project_code: !siteId` is written into attempts and surfaces it in the dashboard). Network blips and PB write
`raw_ocr_data`. The `unmatched_project_code` flag is the operator's conflicts are retryable; validation failures are not.
signal that a manual assignment is needed.
**Failure behaviour.** Per the `process-batch` header comment, the route
distinguishes `retryable: true` (5xx with a transient signal — n8n
retries) from `retryable: false` (400 — n8n marks the batch `dead` and
surfaces it in the dashboard). Network blips and PB write conflicts are
retryable; validation failures are not.
--- ---
@ -287,9 +336,7 @@ sequenceDiagram
participant R as /api/delivery-notes/:id/ocr-result participant R as /api/delivery-notes/:id/ocr-result
participant A as /api/delivery-notes/:id/assign-po participant A as /api/delivery-notes/:id/assign-po
U->>U2: multipart { file, site, delivery_note_number? } U->>U2: multipart { file, site, delivery_note_number? }
U2->>P: getOne(sites, fields="project_code") U2->>P: create(delivery_notes, slip_image=file, site, processing_status=pending, file_hash)
U2->>P: create(delivery_notes, slip_image=file, site, project_code, processing_status=pending)
U2->>P: create(data_import_batches, batch_type=delivery_notes, processed_record_id=dn.id)
U2-->>U: 201 { id, siteId, processingStatus: pending } U2-->>U: 201 { id, siteId, processingStatus: pending }
N->>S: GET (x-ingest-secret) N->>S: GET (x-ingest-secret)
S->>P: getOne(delivery_notes, fields="slip_image") S->>P: getOne(delivery_notes, fields="slip_image")
@ -304,15 +351,14 @@ sequenceDiagram
alt tier = auto alt tier = auto
R->>P: update(delivery_notes, processing_status=auto-verified, is_verified=true, ocr_confidence, ocr_completed_at, purchase_order, raw_ocr_data, ...) R->>P: update(delivery_notes, processing_status=auto-verified, is_verified=true, ocr_confidence, ocr_completed_at, purchase_order, raw_ocr_data, ...)
loop for each item loop for each item
R->>P: create(received_items, delivery_note, purchase_order_item=validatedId, qty_received, condition_notes) R->>P: create(delivery_note_items, delivery_note, purchase_order_item=validatedId, qty_received, condition_notes)
end end
else tier = needs-review else tier = needs-review
R->>P: update(delivery_notes, processing_status=needs-review, is_verified=false, ...) R->>P: update(delivery_notes, processing_status=needs-review, is_verified=false, ...)
R->>P: delete + recreate received_items (idempotent) R->>P: delete + recreate delivery_note_items (idempotent)
else tier = manual else tier = manual
R->>P: update(delivery_notes, processing_status=needs-review, ...) — no received_items R->>P: update(delivery_notes, processing_status=needs-review, ...) — no delivery_note_items
end end
R->>P: update(data_import_batches, status=processed, ...)
opt operator fixes opt operator fixes
U->>A: POST { po: <poId> } (manual fallback) U->>A: POST { po: <poId> } (manual fallback)
A->>P: getOne(purchase_orders, id) A->>P: getOne(purchase_orders, id)
@ -324,11 +370,11 @@ sequenceDiagram
| Confidence | PO resolved | Items verifiable | Tier | Effect | | Confidence | PO resolved | Items verifiable | Tier | Effect |
| --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- |
| ≥ 0.8 | ✅ | ✅ | `auto` | `is_verified=true`, `processing_status=auto-verified`, received_items created with PO links | | ≥ 0.8 | ✅ | ✅ | `auto` | `is_verified=true`, `processing_status=auto-verified`, delivery_note_items created with PO links |
| ≥ 0.8 | ❌ | n/a | `needs-review` (downgraded) | `is_verified=false`, no auto-link — orphan DNs are never auto-verified | | ≥ 0.8 | ❌ | n/a | `needs-review` (downgraded) | `is_verified=false`, no auto-link — orphan DNs are never auto-verified |
| ≥ 0.8 | ✅ | ❌ (local fetch empty but n8n supplied ids) | `needs-review` (downgraded) | We don't claim auto when we can't cross-check the links | | ≥ 0.8 | ✅ | ❌ (local fetch empty but n8n supplied ids) | `needs-review` (downgraded) | We don't claim auto when we can't cross-check the links |
| 0.5..0.79 | any | any | `needs-review` | `is_verified=false`, received_items created without PO links | | 0.5..0.79 | any | any | `needs-review` | `is_verified=false`, delivery_note_items created without PO links |
| < 0.5 | any | any | `manual` | `is_verified=false`, **no received_items** — the operator must capture manually | | < 0.5 | any | any | `manual` | `is_verified=false`, **no delivery_note_items** — the operator must capture manually |
**PO match priority:** exact `purchase_order_number` → normalised **PO match priority:** exact `purchase_order_number` → normalised
(strip non-alnum, lowercase) → Levenshtein (single-char edit, threshold (strip non-alnum, lowercase) → Levenshtein (single-char edit, threshold
@ -338,24 +384,23 @@ so the exception inbox can show "matched by Levenshtein, 0.9" rather
than a silent "no match". than a silent "no match".
**Site attribution strategy.** Site is stamped at upload time from the **Site attribution strategy.** Site is stamped at upload time from the
form field (`src/app/api/delivery-notes/upload/route.ts:26`); the OCR form field (`src/app/api/delivery-notes/upload/route.ts`), a required
callback does not change it. The PO the OCR picks is what links the field on the row; the OCR callback does not change it. For DNs uploaded
delivery to a site indirectly (PO.site). For DNs uploaded without a without a site, the route 400s.
site, the route 400s.
**Idempotency.** If the OCR webhook is replayed, the route **Idempotency.** If the OCR webhook is replayed, the route
**deletes** the existing `received_items` for this DN and recreates them **deletes** the existing `delivery_note_items` for this DN and recreates
within the same call. This avoids stale matches if the OCR rerun finds them within the same call. This avoids stale matches if the OCR rerun finds
different items. The DN update is patch-style so missing fields different items. The DN update is patch-style so missing fields
(delivery_note_number, date_received) are never overwritten by missing (delivery_note_number, date_received) are never overwritten by missing
data. data.
**Failure behaviour.** If the DN update itself fails, the route marks **Failure behaviour.** If the DN update itself fails, the route marks the
the batch `failed` so the operator dashboard reflects the broken state DN's `processing_status` as `failed` so the operator dashboard reflects the
instead of leaving it pending forever. If a single `received_items` broken state instead of leaving it pending forever. If a single
create fails with a 400 (FK race — the PO item was deleted between `delivery_note_items` create fails with a 400 (FK race — the PO item was
fetch and create), the route retries with the link dropped and deleted between fetch and create), the route retries with the link dropped
annotates the row as `link dropped: orphan id`. and annotates the row as `link dropped: orphan id`.
--- ---
@ -370,7 +415,14 @@ annotates the row as `link dropped: orphan id`.
- **PB ID validation.** All `[id]` path params use the PB ID pattern - **PB ID validation.** All `[id]` path params use the PB ID pattern
(`/^[a-z0-9]{15}$/i`) to close the filter-injection sink before the (`/^[a-z0-9]{15}$/i`) to close the filter-injection sink before the
route does `filter: "<field>='<id>'"`. route does `filter: "<field>='<id>'"`.
- **Audit trail.** Every batch keeps the original file via - **Audit trail.** Every document/queue collection (`timesheets`,
`data_import_batches.original_file`. The PO route clears this field `fuel_sheets`, `plant_hours_sheets`, `purchase_orders`) keeps its own
on success (the file is moved to the PO row); it keeps the bytes on original file on its own row via `source_file` (or `purchase_order_file`
the dedup path. The fuel/plant/timesheet routes never clear the file. for POs) — there is no separate batch collection to clear a file from any
more. `file_hash` on each of those, plus `delivery_notes`, drives
upload-time dedup.
- **No `created`/`updated` autodate fields.** This PocketBase instance
never enabled them on any collection. Several routes have comments
noting that sorting or filtering by `created`/`updated` 400s here — use
the collection's own `date_uploaded` / `date_logged` / `date_received`
field instead.

353
docs/data-model.md
View file

@ -3,10 +3,17 @@
> Every PocketBase collection that backs the app, with fields, relations, and > Every PocketBase collection that backs the app, with fields, relations, and
> the site-attribution strategy each one uses. > the site-attribution strategy each one uses.
Source of truth for types: `src/lib/pb-types.ts` (13 collection interfaces). Source of truth for types: `src/lib/pb-types.ts` (16 collection interfaces).
Source of truth for the live schema: `scripts/pb-bootstrap-schema.py`. Source of truth for the live schema: `scripts/pb-bootstrap-schema.py`.
Production instance: `https://pb.mega.infra.apphub.co.za`. Production instance: `https://pb.mega.infra.apphub.co.za`.
This reflects the shape produced by the 2026-09-11 ingestion restructure
(`docs/superpowers/plans/2026-09-11-ingestion-restructure.md`): the shared
`data_import_batches` pool is gone, replaced by typed document collections
(`timesheets`, `fuel_sheets`, `plant_hours_sheets`) and, for purchase orders,
queue fields carried directly on `purchase_orders` itself. `project_code`
denormalisation is gone everywhere except `sites`.
--- ---
## 1. Entity-relationship diagram ## 1. Entity-relationship diagram
@ -21,96 +28,96 @@ erDiagram
SITES ||--o{ DELIVERY_NOTES : "site (relation)" SITES ||--o{ DELIVERY_NOTES : "site (relation)"
SITES ||--o{ FUEL_LOGS : "site (relation)" SITES ||--o{ FUEL_LOGS : "site (relation)"
SITES ||--o{ PLANT_HOURS_LOG : "site (relation)" SITES ||--o{ PLANT_HOURS_LOG : "site (relation)"
SITES ||--o{ TIMESHEET_ENTRIES : "site (relation, stamped at import)"
SITES ||--o{ TIMESHEETS : "site (relation, required)"
SITES ||--o{ FUEL_SHEETS : "site (relation, optional)"
SITES ||--o{ PLANT_HOURS_SHEETS : "site (relation, optional)"
EMPLOYEES ||--o{ TIMESHEETS : "employee (relation)" EMPLOYEES ||--o{ TIMESHEET_ENTRIES : "employee (relation)"
VEHICLES ||--o{ FUEL_LOGS : "vehicle (relation)" VEHICLES ||--o{ FUEL_LOGS : "vehicle (relation)"
VEHICLES ||--o{ PLANT_HOURS_LOG : "vehicle (relation)" VEHICLES ||--o{ PLANT_HOURS_LOG : "vehicle (relation)"
DATA_IMPORT_BATCHES ||--o{ FUEL_LOGS : "batch_reference (relation)" TIMESHEETS ||--o{ TIMESHEET_ENTRIES : "timesheet (relation, optional)"
DATA_IMPORT_BATCHES ||--o{ TIMESHEETS : "batch_reference (relation)" FUEL_SHEETS ||--o{ FUEL_LOGS : "fuel_sheet (relation, optional)"
DATA_IMPORT_BATCHES ||--o{ PLANT_HOURS_LOG : "batch_reference (loose)" PLANT_HOURS_SHEETS ||--o{ PLANT_HOURS_LOG : "plant_hours_sheet (relation, optional)"
PURCHASE_ORDERS ||--|{ PURCHASE_ORDER_ITEMS : "purchase_order (cascade)" PURCHASE_ORDERS ||--|{ PURCHASE_ORDER_ITEMS : "purchase_order (cascade)"
PURCHASE_ORDERS ||--o{ DELIVERY_NOTES : "purchase_order" PURCHASE_ORDERS ||--o{ DELIVERY_NOTES : "purchase_order"
DELIVERY_NOTES ||--|{ RECEIVED_ITEMS : "delivery_note (cascade)" DELIVERY_NOTES ||--|{ DELIVERY_NOTE_ITEMS : "delivery_note (cascade)"
PURCHASE_ORDER_ITEMS ||--o{ RECEIVED_ITEMS : "purchase_order_item" PURCHASE_ORDER_ITEMS ||--o{ DELIVERY_NOTE_ITEMS : "purchase_order_item"
``` ```
The PLANT collection is intentionally absent from the diagram — it is a `data_import_batches` and the deprecated `plant` alias (0 rows, 0 code
deprecated legacy alias for the unified VEHICLES collection references) were deleted entirely as part of this restructure — neither
(`src/lib/pb-types.ts:58-60`). exists in production any more.
--- ---
## 2. Site attribution strategy, per collection ## 2. Site attribution strategy, per collection
This is the table to read first. The question "where does this row belong?" This is the table to read first. Every non-joining collection now carries a
has three different answers depending on which collection you're in. real `site` relation; `project_code` survives only on `sites` itself, as the
one canonical, human-readable label.
| Collection | `site` relation | `project_code` text | Attribution source | Drift risk | | Collection | `site` relation | Attribution source | Drift risk |
| --- | --- | --- | --- | --- | | --- | --- | --- | --- |
| `sites` | n/a | required, **unique** | The site row itself | None — the unique index is authoritative | | `sites` | n/a | The site row itself; `project_code` is required + unique | None — the unique index is authoritative |
| `employees` | ✅ | ✅ | Both, but `site` is canonical for metrics; `project_code` is denormalised at create time | Medium — changing a worker's site later leaves `project_code` stale | | `employees` | ✅ optional | The worker's current site assignment | Low — read directly, nothing denormalised to go stale |
| `vehicles` | ✅ | ✅ | Both, but `site` is the relation; `project_code` is denormalised at create time | High — fuel logs are filtered by `log.site`, not by the current vehicle's site | | `vehicles` | ✅ optional | The vehicle's current site assignment | Low — same |
| `fixed_costs` | ✅ | ✅ | `site` is the relation; `project_code` is denormalised | Low — the import routes always stamp both from the same source | | `fixed_costs` | ✅ optional | Stamped by the entry form | Low |
| `income_claims` | ✅ | ✅ | Same as fixed_costs | Low | | `income_claims` | ✅ optional | Stamped by the entry form | Low |
| `purchase_orders` | ✅ | ✅ | Same as fixed_costs | Low | | `purchase_orders` | ✅ **required** | Resolved by `matchSite` at upload time | Low — required field, no null-site POs |
| `delivery_notes` | ✅ | ✅ | Same as fixed_costs | Low | | `delivery_notes` | ✅ **required** | Stamped at upload time from the form | Low |
| `fuel_logs` | ✅ | ✅ | `site` is filled when the source row has a site; `project_code` is the canonical fallback when the site is missing | Medium — see [multi-site-assessment.md §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) | | `fuel_logs` | ✅ **required** | Resolved per-row during sheet import (site cell, or vehicle's current site as fallback) | Low — required field |
| `plant_hours_log` | ✅ **required** | ✅ **required** | Always set from the vehicle at import time | Lower than fuel_logs because required | | `plant_hours_log` | ✅ **required** | Always set from the matched vehicle's site at import time | Low — required field |
| `timesheets` | ❌ | ❌ | **Joined via `employees.site`** at read time — `timesheets` has no site or project_code of its own | High — site attribution is a per-employee property that can move | | `timesheets` (document) | ✅ **required** | A wage sheet belongs to one site, stamped at upload | None |
| `purchase_order_items` | ❌ | ❌ | Inherited through `purchase_order → site` | None — never queried directly by site | | `timesheet_entries` | ✅ **required** | **Stamped from the employee's site at import time** — not joined at read time | None — this ends the historical drift the old `timesheets` collection had when a worker moved sites after the fact |
| `received_items` | ❌ | ❌ | Inherited through `delivery_note → site` | None — same | | `fuel_sheets` (document) | ✅ optional | A pump sheet can span sites; left empty and resolved per-row against `fuel_logs.site` | Low |
| `data_import_batches` | ❌ | ❌ | Inferred from `processed_record_id` + the processed record's collection | n/a | | `plant_hours_sheets` (document) | ✅ optional | Same reasoning as `fuel_sheets` | Low |
| `purchase_order_items` | ❌ | Inherited through `purchase_order → site` | None — never queried directly by site |
| `delivery_note_items` | ❌ | Inherited through `delivery_note → site` | None — same |
**Reading the table.** Every collection except the two joining-tables and **Reading the table.** `timesheet_entries` (the renamed collection that used
`data_import_batches` carries both a `site` relation and a `project_code` to be called `timesheets`, and used to have no site of its own) is no longer
string. The only collection that carries **neither** is `timesheets` — and the worst drift risk in the system — it now carries a `site` relation that is
that's the collection whose site attribution drifts the most, because it's **stamped once, at import time**, from the employee's site as it was that
synthesised at read time from the employee record. day. A worker moving sites later does not silently rewrite history the way
the old at-read-time join through `employees.site` did. The two join tables
(`purchase_order_items`, `delivery_note_items`) remain the only collections
with no site attribution of their own, by design — they're always reached
through their parent.
--- ---
## 3. The `project_code` denormalisation ## 3. The `project_code` denormalisation — removed
`project_code` is a free-text identifier like `MEG-001` or `HILLS-2026` that Before this restructure, `project_code` (a free-text identifier like
uniquely names a contract. It is the human-readable label that appears on `MEG-001`) was copied onto every child collection as a denormalised string
every report and filter dropdown. at insert time, alongside the `site` relation. That copy was never updated
when the parent site's `project_code` changed, which was the single biggest
drift risk in the schema (see the old §3 in git history for the full
writeup).
The schema carries it on every child collection as a *denormalised copy* of **Current state:** `project_code` now lives in exactly one place —
the parent site's `project_code`. It is **not** a foreign key — it is just a `sites.project_code`, required and unique
string, written at insert time, never updated when the parent site changes. (`idx_sites_project_code`, enforced in `scripts/pb-bootstrap-schema.py`).
Every other collection that needs a human-readable site label resolves it by
expanding or joining the `site` relation at read time
(`src/lib/agent/load.ts`'s `loadSiteRefs()` is the canonical example). There
is no denormalised copy left anywhere to drift.
**Why it exists.** The chat agent and several older reports filter by `scripts/pb-relax-rules.py` and the `PATCH /api/sites/[id]` route
`project_code` (text) rather than expanding the `site` relation (`src/app/api/sites/[id]/route.ts`) still refuse to let you change
(`expand: "site"` would force a join the dashboard pages don't want). It is `project_code` on an existing site — that rule now protects the single
also what the import routes use as the canonical key when a row's `site` source of truth rather than a chain of denormalised copies, but it is
relation is empty. exactly as important as it always was: renaming a site is a new site, not an
edit.
**The drift problem.** If you change a site's `project_code` in PocketBase,
every existing `fuel_logs`, `fixed_costs`, `employees`, `vehicles`,
`purchase_orders`, `delivery_notes`, and `income_claims` row keeps the
*old* string forever. Code that joins through the `site` relation sees the
new value; code that filters on `project_code` text sees the old.
`scripts/pb-relax-rules.py:24` and the `PATCH /api/sites/[id]` route
(`src/app/api/sites/[id]/route.ts:23-28`) deliberately **refuse to let you
change `project_code` on an existing site** — that is exactly the rule
designed to prevent drift. The `POST /api/sites` route also enforces a 64-char
ceiling and uniqueness via the `idx_sites_project_code` index
(`scripts/pb-bootstrap-schema.py:84`).
What this means in practice: the denormalisation is **stable** as long as no
one re-creates a site with the same name and a different code. The biggest
remaining risk is bulk imports that resolve site by name and stamp the
wrong `project_code` onto historical rows — see [multi-site-assessment.md
§4](./multi-site-assessment.md#4-site-resolution-by-name-string).
--- ---
## 4. Collection reference ## 4. Collection reference
Each entry below lists every typed field, the relation direction, and any Each entry below lists every typed field, the relation direction, and any
notable constraints. Optional/legacy fields are marked. notable constraints.
### 4.1 `sites` ### 4.1 `sites`
@ -125,15 +132,15 @@ The site is the unit of business. One row per construction contract.
| `work_hours` | number | optional | Total project work hours | | `work_hours` | number | optional | Total project work hours |
| `hours_per_week` | number | optional | Standard workweek for the site | | `hours_per_week` | number | optional | Standard workweek for the site |
| `wages_hours_per_day` | number | optional | Standard wage day length | | `wages_hours_per_day` | number | optional | Standard wage day length |
| `latitude`, `longitude` | number | optional | Site coordinates |
| `job_start_date` | date | optional | ISO date; the projection helper treats this as the authoritative start when present | | `job_start_date` | date | optional | ISO date; the projection helper treats this as the authoritative start when present |
### 4.2 `employees` ### 4.2 `employees`
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `project_code` | text | Denormalised from `site.project_code` at create time |
| `site` | relation → `sites` | The site the worker is currently assigned to | | `site` | relation → `sites` | The site the worker is currently assigned to |
| `name`, `employee_no`, `position` | text | `employee_no` is the cross-system payroll id; **uppercased** on import (`src/app/api/timesheets/import/route.ts:289`) | | `name`, `employee_no`, `position` | text | `employee_no` is the cross-system payroll id; **uppercased** on import |
| `rate_per_hour` | number | Drives payroll burn | | `rate_per_hour` | number | Drives payroll burn |
| `hours_per_day_standard` | number | 9 by default for auto-created casuals | | `hours_per_day_standard` | number | 9 by default for auto-created casuals |
| `worker_type` | select | `permanent` or `casual` | | `worker_type` | select | `permanent` or `casual` |
@ -142,7 +149,6 @@ The site is the unit of business. One row per construction contract.
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `project_code` | text | Denormalised |
| `site` | relation → `sites` | Current assignment | | `site` | relation → `sites` | Current assignment |
| `description` | text | e.g. "Toyota Hilux 2.4 GD-6" | | `description` | text | e.g. "Toyota Hilux 2.4 GD-6" |
| `standard_work_hours` | text/number | Free-text on schema; numeric in practice. Used by the plant utilisation calc | | `standard_work_hours` | text/number | Free-text on schema; numeric in practice. Used by the plant utilisation calc |
@ -152,8 +158,9 @@ The site is the unit of business. One row per construction contract.
| `unit_type` | select | `fuel` or `plant` (default `fuel`). Splits the bucket between Fleet and Plant pages | | `unit_type` | select | `fuel` or `plant` (default `fuel`). Splits the bucket between Fleet and Plant pages |
| `attention_dismissed_at` | date | When the user dismissed the vehicle's "over fuel limit" alert from the fuel page | | `attention_dismissed_at` | date | When the user dismissed the vehicle's "over fuel limit" alert from the fuel page |
The legacy `plant` collection is a deprecated alias kept around for old The `PLANT` collection (deprecated legacy alias) was deleted in this
importers. New plant items go in `vehicles` with `unit_type="plant"`. restructure — it had 0 rows and 0 code references. All plant items live in
`vehicles` with `unit_type="plant"`.
### 4.4 `fixed_costs` ### 4.4 `fixed_costs`
@ -161,7 +168,6 @@ Monthly or one-off costs. `is_recurring` is the partition key.
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `project_code` | text | Denormalised |
| `site` | relation → `sites` | | | `site` | relation → `sites` | |
| `date_logged` | date | Drives the monthly tile and trend charts | | `date_logged` | date | Drives the monthly tile and trend charts |
| `description`, `notes` | text | | | `description`, `notes` | text | |
@ -173,7 +179,6 @@ Monthly or one-off costs. `is_recurring` is the partition key.
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `claim_number` | text | The contractor's IPN / claim ref | | `claim_number` | text | The contractor's IPN / claim ref |
| `project_code` | text | Denormalised |
| `site` | relation → `sites` | | | `site` | relation → `sites` | |
| `period_start`, `period_end` | date | The period the claim covers | | `period_start`, `period_end` | date | The period the claim covers |
| `gross_amount` | number | Rand | | `gross_amount` | number | Rand |
@ -183,13 +188,27 @@ Monthly or one-off costs. `is_recurring` is the partition key.
| `is_verified` | bool | Verified by a human | | `is_verified` | bool | Verified by a human |
| `notes` | text | | | `notes` | text | |
### 4.6 `purchase_orders` ### 4.6 `fuel_prices`
Unchanged by this restructure. Global (not per-site) pump price history.
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `purchase_order_number` | text | Unique (`idx_po_number`). Normalised to uppercase with "PO" prefix stripped before insert (`src/lib/purchase-order-schema.ts`) | | `effective_from` | date | Required, unique (`idx_fuel_prices_effective_from`) |
| `project_code` | text | Denormalised | | `price_per_litre` | number | Required |
| `site` | relation → `sites` | Resolved by `matchSite` at ingest time | | `notes` | text | optional |
### 4.7 `purchase_orders`
Now carries the queue state `data_import_batches` used to carry — see
`docs/n8n/purchase-order-ingest-implementation.md` for the full async
pipeline. The row is created at upload time with a placeholder number and
`processing_status: pending`; there is no longer a separate batch row.
| Field | Type | Notes |
| --- | --- | --- |
| `purchase_order_number` | text | Unique (`idx_po_number`). Placeholder `PENDING-...` (`src/lib/po-placeholder.ts`) until OCR fills it in |
| `site` | relation → `sites`, **required** | Resolved by `matchSite` at upload time |
| `supplier` | text | | | `supplier` | text | |
| `required_by_date` | date | When the materials are needed | | `required_by_date` | date | When the materials are needed |
| `ordered_by` | text | Who placed it | | `ordered_by` | text | Who placed it |
@ -197,11 +216,20 @@ Monthly or one-off costs. `is_recurring` is the partition key.
| `purchase_order_file` | file | The original PDF/image; mime allowlist enforced by the upload route | | `purchase_order_file` | file | The original PDF/image; mime allowlist enforced by the upload route |
| `raw_ocr_data` | json | Full OCR payload + ingest metadata | | `raw_ocr_data` | json | Full OCR payload + ingest metadata |
| `is_verified` | bool | Human-verified flag | | `is_verified` | bool | Human-verified flag |
| `file_hash` | text | SHA-256 of the original file; drives the upload-time dedup window |
| `processing_status` | select | `pending` / `processing` / `processed` / `failed` / `dead` |
| `attempts` | number | Incremented on each claim |
| `last_error` | text | Last failure reason |
| `next_retry_at`, `last_attempt_at` | date | Backoff scheduling for the n8n cron |
| `date_uploaded` | date | |
### 4.7 `purchase_order_items` Index: `idx_po_site_status` on `(site, processing_status)` — the shape the
n8n cron's "fetch pending" query and the dashboard's per-site pending count
both need.
Joins POs to line items. `cascadeDelete: true` on the parent relation ### 4.8 `purchase_order_items`
(`scripts/pb-bootstrap-schema.py:132`).
Joins POs to line items. `cascadeDelete: true` on the parent relation.
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
@ -209,35 +237,81 @@ Joins POs to line items. `cascadeDelete: true` on the parent relation
| `material_code`, `cost_code` | text | | | `material_code`, `cost_code` | text | |
| `description` | text | The free-text the OCR matches against in the DN line | | `description` | text | The free-text the OCR matches against in the DN line |
| `qty` | number | Ordered quantity | | `qty` | number | Ordered quantity |
| `qty_outstanding` | number | Decreased as received_items roll up — but the canonical fulfilment % is computed from `received_items` directly (see `/dashboard/purchasing` page and the orphan-id note in `src/app/dashboard/purchasing/page.tsx:52-72`) |
| `unit`, `rate`, `disc_percent`, `vat_percent`, `amount` | | The amount field is the **computed** total, not stored separately | | `unit`, `rate`, `disc_percent`, `vat_percent`, `amount` | | The amount field is the **computed** total, not stored separately |
### 4.8 `timesheets` `qty_outstanding` no longer exists as a stored field. Outstanding quantity
is computed at read time from `delivery_note_items` via
`src/lib/po-fulfilment.ts` (`computeOutstanding`) — there is nothing left to
drift out of sync with the actual delivery history.
The collection with **no site** and **no project_code**. Site is resolved ### 4.9 `timesheets` (uploaded document)
through `employeeRates` at every read site.
New collection. One row per uploaded wage sheet — the file itself, plus
queue/outcome fields shared with `fuel_sheets` and `plant_hours_sheets`.
| Field | Type | Notes |
| --- | --- | --- |
| `site` | relation → `sites`, **required** | A wage sheet belongs to one site, unlike the fuel/plant sheets |
| `source_file` | file | The uploaded bytes |
| `content_type` | text | MIME of the original file |
| `file_hash` | text | SHA-256 |
| `date_uploaded`, `uploaded_by` | date/text | |
| `processing_status` | select | `pending` / `processing` / `processed` / `failed` / `dead` |
| `processed_at` | date | |
| `attempts`, `last_error` | number/text | |
| `rows_total`, `rows_created`, `rows_failed` | number | Set by `resolveDocumentOutcome()` (`src/lib/document-status.ts`) — never hardcoded to "processed" before rows are actually inserted |
| `row_errors` | json | Up to `MAX_ROW_ERRORS` (200) per-row failure reasons |
### 4.10 `timesheet_entries` (renamed from `timesheets`)
The actual time log: one row per (employee, date, hours). This is the
collection that used to be called `timesheets` before the restructure —
watch for stale references to the old name in anything not covered by this
rewrite.
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `employee` | relation → `employees` | | | `employee` | relation → `employees` | |
| `batch_reference` | relation → `data_import_batches` | |
| `date_logged` | date | | | `date_logged` | date | |
| `hours_worked` | number | Decimal | | `hours_worked` | number | Decimal |
| `site` | relation → `sites`, **required** | Stamped from the employee's site at import time — not joined at read time (see §2) |
| `timesheet` | relation → `timesheets`, optional | Empty means captured directly by QR scan, not from an uploaded sheet — this stays optional forever |
### 4.9 `fuel_logs` Indexes: `idx_ts_entries_site_date` on `(site, date_logged)`;
`idx_ts_entries_emp_date` on `(employee, date_logged)`.
### 4.11 `fuel_sheets` (uploaded document)
Same shape as `timesheets` (§4.9) except `site` is optional — a pump sheet
can span multiple sites, resolved per-row when the actual `fuel_logs` rows
are created.
Indexes: `idx_fuel_sheets_hash` on `file_hash`; `idx_fuel_sheets_status` on
`(processing_status, date_uploaded)`.
### 4.12 `fuel_logs`
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `project_code` | text | Denormalised, falls back to `vehicle.project_code` when the source row has no site | | `site` | relation → `sites`, **required** | Resolved per-row at import time |
| `site` | relation → `sites` | Optional — empty when the source row had no site column |
| `vehicle` | relation → `vehicles` | Required by the schema but the importer tolerates a missing relation and falls back to QR | | `vehicle` | relation → `vehicles` | Required by the schema but the importer tolerates a missing relation and falls back to QR |
| `batch_reference` | relation → `data_import_batches` | |
| `date_logged` | date | ISO datetime; importer anchors at UTC midnight | | `date_logged` | date | ISO datetime; importer anchors at UTC midnight |
| `litres` | number | Required | | `litres` | number | Required |
| `odo_reading` | number | Required — used for variance calculation | | `odo_reading` | number | Required — used for variance calculation |
| `latest_consumption` | number | Optional. Empty cells preserved as `undefined` rather than 0 (intentional, see comment at `src/lib/pb-types.ts:173-181`) | | `latest_consumption` | number | Optional. Empty cells preserved as `undefined` rather than 0 (intentional) |
| `fuel_sheet` | relation → `fuel_sheets`, optional | Which uploaded sheet produced this row, if any |
### 4.10 `plant_hours_log` Indexes: `idx_fuel_logs_site_date` on `(site, date_logged)`;
`idx_fuel_logs_vehicle_date` on `(vehicle, date_logged)`.
### 4.13 `plant_hours_sheets` (uploaded document)
Identical shape to `fuel_sheets` (§4.11), same optional-site reasoning.
Indexes: `idx_plant_hours_sheets_hash` on `file_hash`;
`idx_plant_hours_sheets_status` on `(processing_status, date_uploaded)`.
### 4.14 `plant_hours_log`
The most disciplined collection. Every site-attribution field is required. The most disciplined collection. Every site-attribution field is required.
@ -247,37 +321,39 @@ The most disciplined collection. Every site-attribution field is required.
| `vehicle_qr` | text | The QR text — part of the **composite unique index** below | | `vehicle_qr` | text | The QR text — part of the **composite unique index** below |
| `vehicle` | relation → `vehicles` | Required (minSelect=1) | | `vehicle` | relation → `vehicles` | Required (minSelect=1) |
| `site` | relation → `sites` | Required (minSelect=1) | | `site` | relation → `sites` | Required (minSelect=1) |
| `project_code` | text | Required |
| `start_hours`, `end_hours`, `hours` | number | | | `start_hours`, `end_hours`, `hours` | number | |
| `hour_meter_budget`, `cost_per_hour`, `value` | number | `value = hours × cost_per_hour` | | `hour_meter_budget`, `cost_per_hour`, `value` | number | `value = hours × cost_per_hour` |
| `over_budget`, `hours_mismatch` | bool | Computed at import time | | `over_budget`, `hours_mismatch` | bool | Computed at import time |
| `vehicle_unit_type_at_import` | select | Snapshot of the vehicle's `unit_type` at the moment of import — see the note in [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) about why this discipline should extend to `site` | | `vehicle_unit_type_at_import` | select | Snapshot of the vehicle's `unit_type` at the moment of import — see the note in [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) about why this discipline should extend to `site` |
| `batch_reference` | relation | Optional | | `plant_hours_sheet` | relation → `plant_hours_sheets`, optional | Which uploaded sheet produced this row, if any |
**Unique index: `(vehicle_qr, start_date)`.** Prevents accidental **Unique index: `(vehicle_qr, start_date)`.** Prevents accidental
double-imports of the same week for the same vehicle double-imports of the same week for the same vehicle. Surfaced as a friendly
(`scripts/pb-bootstrap-schema.py:257`). Surfaced as a friendly
`"duplicate: vehicle_qr=X start_date=Y"` error in the import response. `"duplicate: vehicle_qr=X start_date=Y"` error in the import response.
Additional index: `idx_ph_log_vehicle_start` on `(vehicle, start_date)`.
### 4.11 `delivery_notes` ### 4.15 `delivery_notes`
| Field | Type | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `processing_status` | select | `pending` → `ocr-complete` → `auto-verified` / `needs-review` / `manual-verified` / `failed` |
| `delivery_note_number` | text | Optional; OCR may fill it in | | `delivery_note_number` | text | Optional; OCR may fill it in |
| `project_code` | text | Denormalised | | `site` | relation → `sites`, **required** | Resolved at upload time |
| `site` | relation → `sites` | Resolved at upload time |
| `purchase_order` | relation → `purchase_orders` | Filled by OCR (`exact → normalised → Levenshtein`) or by manual `assign-po` route | | `purchase_order` | relation → `purchase_orders` | Filled by OCR (`exact → normalised → Levenshtein`) or by manual `assign-po` route |
| `date_received` | date | | | `date_received` | date | |
| `received_by` | text | The uploader's identity — currently always `"web-upload"` | | `received_by` | text | The uploader's identity — currently always `"web-upload"` |
| `slip_image` | file | The photo/PDF, served via `/api/delivery-notes/:id/slip` to n8n | | `slip_image` | file | The photo/PDF, served via `/api/delivery-notes/:id/slip` to n8n |
| `raw_ocr_data` | json | Full OCR payload + match metadata | | `raw_ocr_data` | json | Full OCR payload + match metadata |
| `processing_status` | select | `pending` → `ocr-complete` → `auto-verified` / `needs-review` / `manual-verified` / `failed` | | `is_verified` | bool | Final flip from `auto-verified` or `manual-verified` |
| `ocr_confidence` | number | 0..100 integer | | `ocr_confidence` | number | 0..100 integer |
| `ocr_completed_at` | date | | | `ocr_completed_at` | date | |
| `received_by_name` | text | Signature OCR'd from the slip | | `received_by_name` | text | Signature OCR'd from the slip |
| `is_verified` | bool | Final flip from `auto-verified` or `manual-verified` | | `file_hash` | text | SHA-256, used for upload-time dedup |
### 4.12 `received_items` Indexes: `idx_dn_site_date` on `(site, date_received)`; `idx_dn_status` on
`processing_status`.
### 4.16 `delivery_note_items` (renamed from `received_items`)
Joins DNs to PO line items. `cascadeDelete: true` on the parent DN. Joins DNs to PO line items. `cascadeDelete: true` on the parent DN.
@ -288,41 +364,29 @@ Joins DNs to PO line items. `cascadeDelete: true` on the parent DN.
| `qty_received` | number | | | `qty_received` | number | |
| `condition_notes` | text | Free-text audit trail (e.g. `"OCR 92% · desc: Cement 50kg · unit: bag"`) | | `condition_notes` | text | Free-text audit trail (e.g. `"OCR 92% · desc: Cement 50kg · unit: bag"`) |
### 4.13 `data_import_batches`
The audit-trail spine. Every uploaded file — Excel, CSV, PDF, image — has a
batch row that tracks its lifecycle.
| Field | Type | Notes |
| --- | --- | --- |
| `batch_type` | select | `timesheets` / `fuel_logs` / `purchase_orders` / `delivery_notes` / `plant_hours` |
| `content_type` | text | MIME of the original file |
| `original_file` | file | The uploaded bytes; mime-restricted per `batch_type` |
| `date_uploaded` | date | |
| `uploaded_by` | text | Currently always `"web-import"` or `"web-upload"` |
| `status` | select | `pending` / `processing` / `processed` / `failed` / `dead` |
| `attempts` | number | Incremented on each retry |
| `last_error` | text | Last failure reason |
| `last_attempt_at` | date | |
| `next_retry_at` | date | |
| `processed_at` | date | When the batch reached `processed` |
| `processed_record_id` | text | The id of the record this batch created (PO id, DN id, etc.) |
| `file_hash` | text | SHA-256 of the original file; the dedup window for PO re-uploads uses this index |
--- ---
## 5. Indexes and constraints summary ## 5. Indexes and constraints summary
| Index | Where | Purpose | | Index | Where | Purpose |
| --- | --- | --- | | --- | --- | --- |
| `idx_sites_project_code` | `sites(project_code)` UNIQUE | Prevents duplicate site codes; enforced on the POST /api/sites route via the `idx_sites_project_code` name match (`src/app/api/sites/route.ts:64`) | | `idx_sites_project_code` | `sites(project_code)` UNIQUE | Prevents duplicate site codes; enforced on the POST /api/sites route |
| `idx_po_number` | `purchase_orders(purchase_order_number)` UNIQUE | Prevents duplicate PO numbers — works alongside the AI extraction dedup by `purchase_order_number` | | `idx_po_number` | `purchase_orders(purchase_order_number)` UNIQUE | Prevents duplicate PO numbers — final backstop behind the app-level dedup in `process-batch` |
| `idx_plant_hours_log_qr_start` | `plant_hours_log(vehicle_qr, start_date)` UNIQUE | Prevents accidental double-imports of the same week for the same vehicle; surfaced as `duplicate:` reason in the importer | | `idx_po_site_status` | `purchase_orders(site, processing_status)` | Per-site pending-PO counts and the n8n cron's fetch-pending query |
| `idx_batches_file_hash` | `data_import_batches(file_hash, status)` | Used by the PO upload dedup window (`src/app/api/purchase-orders/upload/route.ts:80-84`) | | `idx_plant_hours_log_qr_start` | `plant_hours_log(vehicle_qr, start_date)` UNIQUE | Prevents accidental double-imports of the same week for the same vehicle |
| `idx_ph_log_vehicle_start` | `plant_hours_log(vehicle, start_date)` | Per-vehicle history queries |
| `idx_fuel_prices_effective_from` | `fuel_prices(effective_from)` UNIQUE | One price per effective date |
| `idx_dn_site_date` | `delivery_notes(site, date_received)` | Per-site DN listing |
| `idx_dn_status` | `delivery_notes(processing_status)` | Pending-verification dashboards |
| `idx_fuel_logs_site_date`, `idx_fuel_logs_vehicle_date` | `fuel_logs` | Per-site / per-vehicle fuel history |
| `idx_ts_entries_site_date`, `idx_ts_entries_emp_date` | `timesheet_entries` | Per-site / per-employee timesheet queries |
| `idx_timesheets_hash`, `idx_timesheets_status` | `timesheets` (document) | Upload dedup + pending-import dashboards |
| `idx_fuel_sheets_hash`, `idx_fuel_sheets_status` | `fuel_sheets` | Same, for fuel sheet uploads |
| `idx_plant_hours_sheets_hash`, `idx_plant_hours_sheets_status` | `plant_hours_sheets` | Same, for plant hours sheet uploads |
No collection outside the above has a server-enforced unique constraint. In No collection outside the above has a server-enforced unique constraint. In
particular, `vehicles.qr_code_id` and `vehicles.project_code` are particular, `vehicles.qr_code_id` is **not** unique — see
**not** unique — see [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites). [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites).
--- ---
@ -335,7 +399,7 @@ collection:
authenticated user) authenticated user)
- `updateRule`, `deleteRule` → `null` (superuser only) - `updateRule`, `deleteRule` → `null` (superuser only)
The collection list is the full set of 13 — there is no collection where The collection list is the full set of 16 — there is no collection where
clients can update or delete records through the API. The clients can update or delete records through the API. The
`/api/records/[collection]/[id]/verify` route hand-picks two `/api/records/[collection]/[id]/verify` route hand-picks two
collections (`purchase_orders`, `delivery_notes`) for the `is_verified` collections (`purchase_orders`, `delivery_notes`) for the `is_verified`
@ -351,16 +415,29 @@ unused) browser-side `pb` client and the n8n file-proxy route
## 7. Schema drift ## 7. Schema drift
The `pb-bootstrap-schema.py` script is the canonical schema definition. The `pb-bootstrap-schema.py` is the canonical schema definition — a fresh
following migrations exist on disk and are not represented in the script: PocketBase must come up in the exact shape production is in. That script
had drifted ten-plus migrations behind production before the 2026-09-11
restructure's final task (Task 20) brought it back to truth: it was missing
`fuel_prices` and `income_claims` entirely, still described the deleted
`data_import_batches` and `plant` collections, still carried `project_code`
on eight collections that no longer have it, still had `qty_outstanding` on
`purchase_order_items`, and was missing every index added across the
restructure.
| Migration | Purpose | That drift is exactly the failure mode this section exists to name, so the
| --- | --- | rule going forward is unchanged from before: **any time the schema changes,
| `migrate-delivery-notes-ocr.py` | Adds the `processing_status` / `ocr_*` fields and the OCR webhook hooks to existing DNs | the new field must be added to both `pb-bootstrap-schema.py`** (so a fresh
| `migrate-plant-hours-backfill-vehicle.py` | Backfills `plant_hours_log.vehicle` relations from `vehicle_qr` for pre-relation rows | PB starts in the right shape) **and `src/lib/pb-types.ts`** (so the
| `scripts/migrations/*` (other) | One-off patches for specific schema or data issues | TypeScript code agrees), in the same change that adds it to production.
Forgetting either side is a known foot-gun — it is what produced this
section's rewrite in the first place. `scripts/pb-relax-rules.py`'s
collection list is the third thing that must move in lockstep; it drifted
just as far (missing `fuel_prices`, `income_claims` and `plant_hours_log`
entirely, on top of the same renames/deletions).
Any time the schema changes, the new field must be added to **both** One-off data-migration scripts under `scripts/migrations/*` are expected to
`pb-bootstrap-schema.py` (so a fresh PB starts in the right shape) **and** exist and are not a drift signal by themselves — they are how the schema
`src/lib/pb-types.ts` (so the TypeScript code agrees). Forgetting either *got* to its current shape. The drift signal is `pb-bootstrap-schema.py` (or
side is a known foot-gun. `pb-relax-rules.py`) disagreeing with what a migration already did to
production.

BIN
scripts/__pycache__/pb-bootstrap-schema.cpython-314.pyc
View file

Binary file not shown.

BIN
scripts/__pycache__/pb-relax-rules.cpython-314.pyc Normal file
View file

Binary file not shown.

222
scripts/pb-bootstrap-schema.py
View file

@ -4,6 +4,25 @@
PocketBase v0.39 JSON shape: type-specific options (values, maxSelect, PocketBase v0.39 JSON shape: type-specific options (values, maxSelect,
mimeTypes, collectionId, cascadeDelete, min, max) live at the field mimeTypes, collectionId, cascadeDelete, min, max) live at the field
top level — NOT nested under "options". top level — NOT nested under "options".
Reflects the post-ingestion-restructure schema (2026-09):
- `data_import_batches` (the shared queue pool) and the deprecated
`plant` alias for `vehicles` are gone.
- `timesheets`, `fuel_sheets` and `plant_hours_sheets` are the
uploaded-document collections; each carries its own
`processing_status` queue fields directly (no separate batch row).
- `timesheets` (old) was renamed `timesheet_entries` — the actual
per-employee-per-day time log.
- `received_items` was renamed `delivery_note_items`.
- `purchase_orders` absorbed the queue fields `data_import_batches`
used to carry, plus `file_hash` and `date_uploaded`.
- `project_code` was removed from every collection except `sites`
(the one place it is canonical); relations to `sites` carry site
attribution everywhere else.
- `qty_outstanding` was removed from `purchase_order_items` — it is
now computed at read time via src/lib/po-fulfilment.ts.
- `fuel_prices` and `income_claims` were always live collections but
were missing from this bootstrap script; both are included now.
""" """
import json import json
import sys import sys
@ -52,21 +71,36 @@ def field(name, type_, **kw):
return f return f
DEFS = {} # Shared field set for the three "uploaded document" collections
# (timesheets, fuel_sheets, plant_hours_sheets) — each is a
DEFS["data_import_batches"] = { # resumable/idempotent import queue item in its own right, not a row
"name": "DATA_IMPORT_BATCHES", "type": "base", # in a shared pool.
"fields": [ def sheet_fields(site_required):
field("batch_type", "select", required=True, return [
values=["timesheets", "fuel_logs", "purchase_orders", "delivery_notes", "plant_hours"], maxSelect=1), field("site", "relation", required=site_required,
field("original_file", "file", required=False, collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("source_file", "file", required=False,
maxSelect=1, maxSize=20971520, maxSelect=1, maxSize=20971520,
mimeTypes=["text/csv", "text/plain", "application/vnd.ms-excel", "application/csv"]), mimeTypes=["text/csv", "text/plain", "application/vnd.ms-excel",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/csv"]),
field("content_type", "text", required=False),
field("file_hash", "text", required=False),
field("date_uploaded", "date", required=False), field("date_uploaded", "date", required=False),
field("uploaded_by", "text", required=False), field("uploaded_by", "text", required=False),
], field("processing_status", "select", required=False,
"indexes": [], values=["pending", "processing", "processed", "failed", "dead"], maxSelect=1),
} field("processed_at", "date", required=False),
field("attempts", "number", required=False),
field("last_error", "text", required=False),
field("rows_total", "number", required=False),
field("rows_created", "number", required=False),
field("rows_failed", "number", required=False),
field("row_errors", "json", required=False),
]
DEFS = {}
DEFS["sites"] = { DEFS["sites"] = {
"name": "SITES", "type": "base", "name": "SITES", "type": "base",
@ -78,6 +112,8 @@ DEFS["sites"] = {
field("work_hours", "number", required=False), field("work_hours", "number", required=False),
field("hours_per_week", "number", required=False), field("hours_per_week", "number", required=False),
field("wages_hours_per_day", "number", required=False), field("wages_hours_per_day", "number", required=False),
field("latitude", "number", required=False),
field("longitude", "number", required=False),
field("job_start_date", "date", required=False), field("job_start_date", "date", required=False),
], ],
"indexes": [ "indexes": [
@ -88,7 +124,6 @@ DEFS["sites"] = {
DEFS["employees"] = { DEFS["employees"] = {
"name": "EMPLOYEES", "type": "base", "name": "EMPLOYEES", "type": "base",
"fields": [ "fields": [
field("project_code", "text", required=False),
field("site", "relation", required=False, field("site", "relation", required=False,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1), collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("name", "text", required=False), field("name", "text", required=False),
@ -106,8 +141,7 @@ DEFS["purchase_orders"] = {
"name": "PURCHASE_ORDERS", "type": "base", "name": "PURCHASE_ORDERS", "type": "base",
"fields": [ "fields": [
field("purchase_order_number", "text", required=True, min=1, max=128), field("purchase_order_number", "text", required=True, min=1, max=128),
field("project_code", "text", required=False), field("site", "relation", required=True,
field("site", "relation", required=False,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1), collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("supplier", "text", required=False), field("supplier", "text", required=False),
field("required_by_date", "date", required=False), field("required_by_date", "date", required=False),
@ -119,9 +153,18 @@ DEFS["purchase_orders"] = {
mimeTypes=["application/pdf", "image/jpeg", "image/png", "image/webp"]), mimeTypes=["application/pdf", "image/jpeg", "image/png", "image/webp"]),
field("raw_ocr_data", "json", required=False), field("raw_ocr_data", "json", required=False),
field("is_verified", "bool", required=False), field("is_verified", "bool", required=False),
field("file_hash", "text", required=False),
field("processing_status", "select", required=False,
values=["pending", "processing", "processed", "failed", "dead"], maxSelect=1),
field("attempts", "number", required=False),
field("last_error", "text", required=False),
field("next_retry_at", "date", required=False),
field("last_attempt_at", "date", required=False),
field("date_uploaded", "date", required=False),
], ],
"indexes": [ "indexes": [
"CREATE UNIQUE INDEX `idx_po_number` ON `purchase_orders` (`purchase_order_number`)", "CREATE UNIQUE INDEX `idx_po_number` ON `purchase_orders` (`purchase_order_number`)",
"CREATE INDEX `idx_po_site_status` ON `purchase_orders` (`site`, `processing_status`)",
], ],
} }
@ -134,7 +177,6 @@ DEFS["purchase_order_items"] = {
field("cost_code", "text", required=False), field("cost_code", "text", required=False),
field("description", "text", required=False), field("description", "text", required=False),
field("qty", "number", required=False), field("qty", "number", required=False),
field("qty_outstanding", "number", required=False),
field("unit", "text", required=False), field("unit", "text", required=False),
field("rate", "number", required=False), field("rate", "number", required=False),
field("disc_percent", "number", required=False), field("disc_percent", "number", required=False),
@ -147,7 +189,6 @@ DEFS["purchase_order_items"] = {
DEFS["fixed_costs"] = { DEFS["fixed_costs"] = {
"name": "FIXED_COSTS", "type": "base", "name": "FIXED_COSTS", "type": "base",
"fields": [ "fields": [
field("project_code", "text", required=False),
field("site", "relation", required=False, field("site", "relation", required=False,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1), collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("date_logged", "date", required=False), field("date_logged", "date", required=False),
@ -166,63 +207,104 @@ DEFS["fixed_costs"] = {
"indexes": [], "indexes": [],
} }
DEFS["plant"] = {
"name": "PLANT", "type": "base",
"fields": [
field("project_code", "text", required=False),
field("site", "relation", required=False,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("description", "text", required=False),
field("standard_work_hours", "text", required=False),
field("cost_per_hour", "number", required=False),
],
"indexes": [],
}
DEFS["vehicles"] = { DEFS["vehicles"] = {
"name": "VEHICLES", "type": "base", "name": "VEHICLES", "type": "base",
"fields": [ "fields": [
field("project_code", "text", required=False),
field("site", "relation", required=False, field("site", "relation", required=False,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1), collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("description", "text", required=False), field("description", "text", required=False),
field("standard_work_hours", "text", required=False), field("standard_work_hours", "text", required=False),
field("fuel_limits", "text", required=False), field("fuel_limits", "text", required=False),
field("qr_code_id", "text", required=False),
field("cost_per_hour", "number", required=False),
field("attention_dismissed_at", "date", required=False),
field("unit_type", "select", required=False, field("unit_type", "select", required=False,
values=["fuel", "plant"], maxSelect=1, default="fuel"), values=["fuel", "plant"], maxSelect=1, default="fuel"),
field("attention_dismissed_at", "date", required=False),
], ],
"indexes": [], "indexes": [],
} }
DEFS["fuel_prices"] = {
"name": "FUEL_PRICES", "type": "base",
"fields": [
field("effective_from", "date", required=True),
field("price_per_litre", "number", required=True),
field("notes", "text", required=False),
],
"indexes": [
"CREATE UNIQUE INDEX `idx_fuel_prices_effective_from` ON `fuel_prices` (`effective_from`)",
],
}
DEFS["timesheets"] = { DEFS["timesheets"] = {
"name": "TIMESHEETS", "type": "base", "name": "TIMESHEETS", "type": "base",
"fields": sheet_fields(site_required=True),
"indexes": [
"CREATE INDEX `idx_timesheets_hash` ON `timesheets` (`file_hash`)",
"CREATE INDEX `idx_timesheets_status` ON `timesheets` (`processing_status`, `date_uploaded`)",
],
}
DEFS["timesheet_entries"] = {
"name": "TIMESHEET_ENTRIES", "type": "base",
"fields": [ "fields": [
field("employee", "relation", required=False, field("employee", "relation", required=False,
collectionId="__EMPLOYEES_ID__", cascadeDelete=False, maxSelect=1), collectionId="__EMPLOYEES_ID__", cascadeDelete=False, maxSelect=1),
field("batch_reference", "relation", required=False,
collectionId="__DATA_IMPORT_BATCHES_ID__", cascadeDelete=False, maxSelect=1),
field("date_logged", "date", required=False), field("date_logged", "date", required=False),
field("hours_worked", "number", required=False), field("hours_worked", "number", required=False),
# Stamped from the employee's site at import time — NOT joined at
# read time. This ends historical drift when a worker moves sites.
field("site", "relation", required=True,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
# Optional forever: empty means captured directly by QR scan, not
# from an uploaded sheet.
field("timesheet", "relation", required=False,
collectionId="__TIMESHEETS_ID__", cascadeDelete=False, maxSelect=1),
],
"indexes": [
"CREATE INDEX `idx_ts_entries_site_date` ON `timesheet_entries` (`site`, `date_logged`)",
"CREATE INDEX `idx_ts_entries_emp_date` ON `timesheet_entries` (`employee`, `date_logged`)",
],
}
DEFS["fuel_sheets"] = {
"name": "FUEL_SHEETS", "type": "base",
# site is optional: a pump sheet can span sites, resolved per-row.
"fields": sheet_fields(site_required=False),
"indexes": [
"CREATE INDEX `idx_fuel_sheets_hash` ON `fuel_sheets` (`file_hash`)",
"CREATE INDEX `idx_fuel_sheets_status` ON `fuel_sheets` (`processing_status`, `date_uploaded`)",
], ],
"indexes": [],
} }
DEFS["fuel_logs"] = { DEFS["fuel_logs"] = {
"name": "FUEL_LOGS", "type": "base", "name": "FUEL_LOGS", "type": "base",
"fields": [ "fields": [
field("project_code", "text", required=False), field("site", "relation", required=True,
field("site", "relation", required=False,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1), collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("vehicle", "relation", required=False, field("vehicle", "relation", required=False,
collectionId="__VEHICLES_ID__", cascadeDelete=False, maxSelect=1), collectionId="__VEHICLES_ID__", cascadeDelete=False, maxSelect=1),
field("batch_reference", "relation", required=False,
collectionId="__DATA_IMPORT_BATCHES_ID__", cascadeDelete=False, maxSelect=1),
field("date_logged", "date", required=False), field("date_logged", "date", required=False),
field("litres", "number", required=False), field("litres", "number", required=False),
field("odo_reading", "number", required=False), field("odo_reading", "number", required=False),
field("latest_consumption", "number", required=False),
field("fuel_sheet", "relation", required=False,
collectionId="__FUEL_SHEETS_ID__", cascadeDelete=False, maxSelect=1),
],
"indexes": [
"CREATE INDEX `idx_fuel_logs_site_date` ON `fuel_logs` (`site`, `date_logged`)",
"CREATE INDEX `idx_fuel_logs_vehicle_date` ON `fuel_logs` (`vehicle`, `date_logged`)",
],
}
DEFS["plant_hours_sheets"] = {
"name": "PLANT_HOURS_SHEETS", "type": "base",
# site is optional, same reasoning as fuel_sheets.
"fields": sheet_fields(site_required=False),
"indexes": [
"CREATE INDEX `idx_plant_hours_sheets_hash` ON `plant_hours_sheets` (`file_hash`)",
"CREATE INDEX `idx_plant_hours_sheets_status` ON `plant_hours_sheets` (`processing_status`, `date_uploaded`)",
], ],
"indexes": [],
} }
DEFS["plant_hours_log"] = { DEFS["plant_hours_log"] = {
@ -234,7 +316,6 @@ DEFS["plant_hours_log"] = {
collectionId="__VEHICLES_ID__", cascadeDelete=False, maxSelect=1, minSelect=1), collectionId="__VEHICLES_ID__", cascadeDelete=False, maxSelect=1, minSelect=1),
field("site", "relation", required=True, field("site", "relation", required=True,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1, minSelect=1), collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1, minSelect=1),
field("project_code", "text", required=True),
field("start_date", "date", required=True), field("start_date", "date", required=True),
field("end_date", "date", required=True), field("end_date", "date", required=True),
field("start_time", "text", required=False), field("start_time", "text", required=False),
@ -246,24 +327,46 @@ DEFS["plant_hours_log"] = {
field("hour_meter_budget", "number", required=True), field("hour_meter_budget", "number", required=True),
field("cost_per_hour", "number", required=True), field("cost_per_hour", "number", required=True),
field("value", "number", required=True), field("value", "number", required=True),
field("over_budget", "bool", required=True), field("over_budget", "bool", required=False),
field("hours_mismatch", "bool", required=True), field("hours_mismatch", "bool", required=False),
field("vehicle_unit_type_at_import", "select", required=True, field("vehicle_unit_type_at_import", "select", required=True,
values=["fuel", "plant"], maxSelect=1), values=["fuel", "plant"], maxSelect=1),
field("batch_reference", "relation", required=False, field("plant_hours_sheet", "relation", required=False,
collectionId="__DATA_IMPORT_BATCHES_ID__", cascadeDelete=False, maxSelect=1, minSelect=0), collectionId="__PLANT_HOURS_SHEETS_ID__", cascadeDelete=False, maxSelect=1, minSelect=0),
], ],
"indexes": [ "indexes": [
"CREATE UNIQUE INDEX `idx_plant_hours_log_qr_start` ON `plant_hours_log` (`vehicle_qr`, `start_date`)", "CREATE UNIQUE INDEX `idx_plant_hours_log_qr_start` ON `plant_hours_log` (`vehicle_qr`, `start_date`)",
"CREATE INDEX `idx_ph_log_vehicle_start` ON `plant_hours_log` (`vehicle`, `start_date`)",
], ],
} }
DEFS["income_claims"] = {
"name": "INCOME_CLAIMS", "type": "base",
"fields": [
field("claim_number", "text", required=True),
field("site", "relation", required=False,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("period_start", "date", required=False),
field("period_end", "date", required=False),
field("gross_amount", "number", required=False),
field("certified_by", "text", required=False),
field("paid_on", "date", required=False),
field("status", "select", required=False,
values=["draft", "certified", "paid"], maxSelect=1),
field("is_verified", "bool", required=False),
field("notes", "text", required=False),
],
"indexes": [],
}
DEFS["delivery_notes"] = { DEFS["delivery_notes"] = {
"name": "DELIVERY_NOTES", "type": "base", "name": "DELIVERY_NOTES", "type": "base",
"fields": [ "fields": [
field("processing_status", "select", required=False,
values=["pending", "ocr-complete", "auto-verified",
"needs-review", "manual-verified", "failed"], maxSelect=1),
field("delivery_note_number", "text", required=False), field("delivery_note_number", "text", required=False),
field("project_code", "text", required=False), field("site", "relation", required=True,
field("site", "relation", required=False,
collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1), collectionId="__SITES_ID__", cascadeDelete=False, maxSelect=1),
field("purchase_order", "relation", required=False, field("purchase_order", "relation", required=False,
collectionId="__PURCHASE_ORDERS_ID__", cascadeDelete=False, maxSelect=1), collectionId="__PURCHASE_ORDERS_ID__", cascadeDelete=False, maxSelect=1),
@ -274,18 +377,19 @@ DEFS["delivery_notes"] = {
mimeTypes=["image/jpeg", "image/png", "image/webp", "application/pdf"]), mimeTypes=["image/jpeg", "image/png", "image/webp", "application/pdf"]),
field("raw_ocr_data", "json", required=False), field("raw_ocr_data", "json", required=False),
field("is_verified", "bool", required=False), field("is_verified", "bool", required=False),
field("processing_status", "select", required=False,
values=["pending", "ocr-complete", "auto-verified",
"needs-review", "manual-verified", "failed"], maxSelect=1),
field("ocr_confidence", "number", required=False), field("ocr_confidence", "number", required=False),
field("ocr_completed_at", "date", required=False), field("ocr_completed_at", "date", required=False),
field("received_by_name", "text", required=False), field("received_by_name", "text", required=False),
field("file_hash", "text", required=False),
],
"indexes": [
"CREATE INDEX `idx_dn_site_date` ON `delivery_notes` (`site`, `date_received`)",
"CREATE INDEX `idx_dn_status` ON `delivery_notes` (`processing_status`)",
], ],
"indexes": [],
} }
DEFS["received_items"] = { DEFS["delivery_note_items"] = {
"name": "RECEIVED_ITEMS", "type": "base", "name": "DELIVERY_NOTE_ITEMS", "type": "base",
"fields": [ "fields": [
field("delivery_note", "relation", required=False, field("delivery_note", "relation", required=False,
collectionId="__DELIVERY_NOTES_ID__", cascadeDelete=True, maxSelect=1), collectionId="__DELIVERY_NOTES_ID__", cascadeDelete=True, maxSelect=1),
@ -297,20 +401,26 @@ DEFS["received_items"] = {
"indexes": [], "indexes": [],
} }
# Dependency order: a collection referencing another via `relation` must be
# created after the collection it points to, so its `collectionId`
# placeholder can be resolved.
ORDER = [ ORDER = [
"data_import_batches",
"sites", "sites",
"employees", "employees",
"purchase_orders", "purchase_orders",
"purchase_order_items", "purchase_order_items",
"fixed_costs", "fixed_costs",
"plant",
"vehicles", "vehicles",
"fuel_prices",
"timesheets", "timesheets",
"timesheet_entries",
"fuel_sheets",
"fuel_logs", "fuel_logs",
"plant_hours_sheets",
"plant_hours_log", "plant_hours_log",
"income_claims",
"delivery_notes", "delivery_notes",
"received_items", "delivery_note_items",
] ]
ids = {} ids = {}

12
scripts/pb-relax-rules.py
View file

@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Relax API rules on the 12 MEGA INFRASTRUCTURE collections. """Relax API rules on the 16 MEGA INFRASTRUCTURE collections.
Pattern: Pattern:
list/view/create → any authenticated user list/view/create → any authenticated user
@ -20,18 +20,22 @@ AUTH_RULE = "@request.auth.id != ''" # any logged-in user
NONE_RULE = None # superuser only NONE_RULE = None # superuser only
RULES = { RULES = {
"DATA_IMPORT_BATCHES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"SITES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "SITES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"EMPLOYEES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "EMPLOYEES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"PURCHASE_ORDERS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "PURCHASE_ORDERS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"PURCHASE_ORDER_ITEMS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "PURCHASE_ORDER_ITEMS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"FIXED_COSTS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "FIXED_COSTS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"PLANT": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"VEHICLES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "VEHICLES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"FUEL_PRICES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"TIMESHEETS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "TIMESHEETS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"TIMESHEET_ENTRIES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"FUEL_SHEETS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"FUEL_LOGS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "FUEL_LOGS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"PLANT_HOURS_SHEETS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"PLANT_HOURS_LOG": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"INCOME_CLAIMS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"DELIVERY_NOTES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "DELIVERY_NOTES": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
"RECEIVED_ITEMS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE), "DELIVERY_NOTE_ITEMS": (AUTH_RULE, AUTH_RULE, AUTH_RULE, NONE_RULE, NONE_RULE),
} }

25
src/lib/agent/load.ts
View file

@ -344,36 +344,47 @@ export interface ImportItem {
* PurchaseOrderRecord.processing_status / DeliveryNoteRecord.processing_status * PurchaseOrderRecord.processing_status / DeliveryNoteRecord.processing_status
* in src/lib/pb-types.ts) — there is no separate batch row for either. * in src/lib/pb-types.ts) — there is no separate batch row for either.
*/ */
export async function loadImportBatches(args: { status?: string }): Promise<ImportItem[]> { export async function loadImportBatches(args: {
status?: string;
siteId?: string;
}): Promise<ImportItem[]> {
const pb = await pbAdmin(); const pb = await pbAdmin();
const statusFilter = args.status ? `processing_status='${args.status}'` : ""; const clauses: string[] = [];
if (args.status) clauses.push(`processing_status='${args.status}'`);
// fuel_sheets and plant_hours_sheets carry an optional `site` — a sheet
// with no site set can span sites and is resolved per-row, so it is
// deliberately excluded when scoping to one site rather than included
// by default.
const siteClause = args.siteId ? `site='${args.siteId}'` : "";
const filterWith = (extra: string) =>
[...clauses, ...(extra ? [extra] : [])].join(" && ") || undefined;
const docFields = const docFields =
"id,processing_status,date_uploaded,uploaded_by,attempts,last_error,processed_at"; "id,processing_status,date_uploaded,uploaded_by,attempts,last_error,processed_at";
const [timesheets, fuelSheets, plantHoursSheets, purchaseOrders, deliveryNotes] = const [timesheets, fuelSheets, plantHoursSheets, purchaseOrders, deliveryNotes] =
await Promise.all([ await Promise.all([
pb.collection("timesheets").getFullList({ pb.collection("timesheets").getFullList({
...(statusFilter ? { filter: statusFilter } : {}), filter: filterWith(siteClause),
sort: "-date_uploaded", sort: "-date_uploaded",
fields: docFields, fields: docFields,
}), }),
pb.collection("fuel_sheets").getFullList({ pb.collection("fuel_sheets").getFullList({
...(statusFilter ? { filter: statusFilter } : {}), filter: filterWith(siteClause),
sort: "-date_uploaded", sort: "-date_uploaded",
fields: docFields, fields: docFields,
}), }),
pb.collection("plant_hours_sheets").getFullList({ pb.collection("plant_hours_sheets").getFullList({
...(statusFilter ? { filter: statusFilter } : {}), filter: filterWith(siteClause),
sort: "-date_uploaded", sort: "-date_uploaded",
fields: docFields, fields: docFields,
}), }),
pb.collection("purchase_orders").getFullList({ pb.collection("purchase_orders").getFullList({
...(statusFilter ? { filter: statusFilter } : {}), filter: filterWith(siteClause),
sort: "-date_uploaded", sort: "-date_uploaded",
fields: "id,processing_status,date_uploaded,ordered_by,attempts,last_error", fields: "id,processing_status,date_uploaded,ordered_by,attempts,last_error",
}), }),
pb.collection("delivery_notes").getFullList({ pb.collection("delivery_notes").getFullList({
...(statusFilter ? { filter: statusFilter } : {}), filter: filterWith(siteClause),
sort: "-date_received", sort: "-date_received",
fields: "id,processing_status,date_received,received_by,received_by_name", fields: "id,processing_status,date_received,received_by,received_by_name",
}), }),

12
src/lib/agent/tools.ts
View file

@ -871,6 +871,7 @@ export const tools: AgentTool[] = [
description: description:
"Document import batches — timesheets, fuel logs, purchase orders, delivery notes and plant hours — with their processing status, attempt count and any error. Use for 'what uploads failed', 'did my timesheet import work', 'what is stuck'.", "Document import batches — timesheets, fuel logs, purchase orders, delivery notes and plant hours — with their processing status, attempt count and any error. Use for 'what uploads failed', 'did my timesheet import work', 'what is stuck'.",
schema: { schema: {
site: z.string().optional().describe("Site name, project code, or id"),
status: z status: z
.enum(["pending", "processing", "processed", "failed", "dead"]) .enum(["pending", "processing", "processed", "failed", "dead"])
.optional() .optional()
@ -878,12 +879,23 @@ export const tools: AgentTool[] = [
limit: z.number().optional().describe("Defaults to 25"), limit: z.number().optional().describe("Defaults to 25"),
}, },
handler: async (args) => { handler: async (args) => {
let siteId: string | undefined;
let scope = "all sites";
if (args.site) {
const found = await requireSite(String(args.site));
if (!found.ok) return { error: found.error };
siteId = found.site.id;
scope = found.site.name;
}
const batches = await loadImportBatches({ const batches = await loadImportBatches({
status: args.status as string | undefined, status: args.status as string | undefined,
siteId,
}); });
const limit = typeof args.limit === "number" ? args.limit : 25; const limit = typeof args.limit === "number" ? args.limit : 25;
return { return {
scope,
total: batches.length, total: batches.length,
showing: Math.min(limit, batches.length), showing: Math.min(limit, batches.length),
failed: batches.filter((b) => b.status === "failed" || b.status === "dead").length, failed: batches.filter((b) => b.status === "failed" || b.status === "dead").length,