MEGAINFRA-WEBAPP/docs/data-model.md

30 KiB
Raw Blame History

Data model

Every PocketBase collection that backs the app, with fields, relations, and the site-attribution strategy each one uses.

Source of truth for types: src/lib/pb-types.ts (16 collection interfaces). Source of truth for the live schema: scripts/pb-bootstrap-schema.py. Production instance: https://pb.mega.infra.apphub.co.za.

This reflects the shape produced by the 2026-09-11 ingestion restructure (docs/superpowers/plans/2026-09-11-ingestion-restructure.md): the shared data_import_batches pool is gone, replaced by typed document collections (timesheets, fuel_sheets, plant_hours_sheets) and, for purchase orders, queue fields carried directly on purchase_orders itself. project_code denormalisation is gone everywhere except sites.


1. Entity-relationship diagram

erDiagram
    SITES ||--o{ EMPLOYEES : "site (relation)"
    SITES ||--o{ VEHICLES : "site (relation)"
    SITES ||--o{ FIXED_COSTS : "site (relation)"
    SITES ||--o{ INCOME_CLAIMS : "site (relation)"
    SITES ||--o{ PURCHASE_ORDERS : "site (relation)"
    SITES ||--o{ DELIVERY_NOTES : "site (relation)"
    SITES ||--o{ FUEL_LOGS : "site (relation)"
    SITES ||--o{ PLANT_HOURS_LOG : "site (relation)"
    SITES ||--o{ TIMESHEET_ENTRIES : "site (relation, stamped at import)"
    SITES ||--o{ TIMESHEETS : "site (relation, required)"
    SITES ||--o{ FUEL_SHEETS : "site (relation, optional)"
    SITES ||--o{ PLANT_HOURS_SHEETS : "site (relation, optional)"

    EMPLOYEES ||--o{ TIMESHEET_ENTRIES : "employee (relation)"
    VEHICLES ||--o{ FUEL_LOGS : "vehicle (relation)"
    VEHICLES ||--o{ PLANT_HOURS_LOG : "vehicle (relation)"
    TIMESHEETS ||--o{ TIMESHEET_ENTRIES : "timesheet (relation, optional)"
    FUEL_SHEETS ||--o{ FUEL_LOGS : "fuel_sheet (relation, optional)"
    PLANT_HOURS_SHEETS ||--o{ PLANT_HOURS_LOG : "plant_hours_sheet (relation, optional)"

    PURCHASE_ORDERS ||--|{ PURCHASE_ORDER_ITEMS : "purchase_order (cascade)"
    PURCHASE_ORDERS ||--o{ DELIVERY_NOTES : "purchase_order"
    DELIVERY_NOTES ||--|{ DELIVERY_NOTE_ITEMS : "delivery_note (cascade)"
    PURCHASE_ORDER_ITEMS ||--o{ DELIVERY_NOTE_ITEMS : "purchase_order_item"

data_import_batches and the deprecated plant alias (0 rows, 0 code references) were deleted entirely as part of this restructure — neither exists in production any more.


2. Site attribution strategy, per collection

This is the table to read first. Every non-joining collection now carries a real site relation; project_code survives only on sites itself, as the one canonical, human-readable label.

Collection site relation Attribution source Drift risk
sites n/a The site row itself; project_code is required + unique None — the unique index is authoritative
employees ✅ optional The worker's current site assignment Low — read directly, nothing denormalised to go stale
vehicles ✅ optional The vehicle's current site assignment Low — same
fixed_costs ✅ optional Stamped by the entry form Low
income_claims ✅ optional Stamped by the entry form Low
purchase_orders ✅ required Resolved by matchSite at upload time Low — required field, no null-site POs
delivery_notes ✅ required Stamped at upload time from the form Low
fuel_logs ✅ required Resolved per-row during sheet import (site cell, or vehicle's current site as fallback) Low — required field
plant_hours_log ✅ required Always set from the matched vehicle's site at import time Low — required field
timesheets (document) ✅ required A wage sheet belongs to one site, stamped at upload None
timesheet_entries ✅ required Stamped from the employee's site at import time — not joined at read time None — this ends the historical drift the old timesheets collection had when a worker moved sites after the fact
fuel_sheets (document) ✅ optional A pump sheet can span sites; left empty and resolved per-row against fuel_logs.site Low
plant_hours_sheets (document) ✅ optional Same reasoning as fuel_sheets Low
purchase_order_items ❌ Inherited through purchase_order → site None — never queried directly by site
delivery_note_items ❌ Inherited through delivery_note → site None — same

Reading the table. timesheet_entries (the renamed collection that used to be called timesheets, and used to have no site of its own) is no longer the worst drift risk in the system — it now carries a site relation that is stamped once, at import time, from the employee's site as it was that day. A worker moving sites later does not silently rewrite history the way the old at-read-time join through employees.site did. The two join tables (purchase_order_items, delivery_note_items) remain the only collections with no site attribution of their own, by design — they're always reached through their parent.


3. The project_code denormalisation — removed

Before this restructure, project_code (a free-text identifier like MEG-001) was copied onto every child collection as a denormalised string at insert time, alongside the site relation. That copy was never updated when the parent site's project_code changed, which was the single biggest drift risk in the schema (see the old §3 in git history for the full writeup).

Current state: project_code now lives in exactly one place — sites.project_code, required and unique (idx_sites_project_code, enforced in scripts/pb-bootstrap-schema.py). Every other collection that needs a human-readable site label resolves it by expanding or joining the site relation at read time (src/lib/agent/load.ts's loadSiteRefs() is the canonical example). There is no denormalised copy left anywhere to drift.

scripts/pb-relax-rules.py and the PATCH /api/sites/[id] route (src/app/api/sites/[id]/route.ts) still refuse to let you change project_code on an existing site — that rule now protects the single source of truth rather than a chain of denormalised copies, but it is exactly as important as it always was: renaming a site is a new site, not an edit.


4. Collection reference

Each entry below lists every typed field, the relation direction, and any notable constraints.

4.1 sites

The site is the unit of business. One row per construction contract.

Field Type Required Notes
project_code text ✅ Unique (idx_sites_project_code). Locked after creation.
name text optional Human label, e.g. "The Hills"
duration text optional Free-text duration ("8 months", "52 weeks") — parsed by parseDurationMonths
contract_value number optional Rand value, no currency field
work_hours number optional Total project work hours
hours_per_week number optional Standard workweek for the site
wages_hours_per_day number optional Standard wage day length
latitude, longitude number optional Site coordinates
job_start_date date optional ISO date; the projection helper treats this as the authoritative start when present

4.2 employees

Field Type Notes
site relation → sites The site the worker is currently assigned to
name, employee_no, position text employee_no is the cross-system payroll id; uppercased on import
rate_per_hour number Drives payroll burn
hours_per_day_standard number 9 by default for auto-created casuals
worker_type select permanent or casual

Worker status

employment_status is an optional select (active, inactive). Blank status is also treated as active for compatibility with older imports. inactive_from is an optional date recording when the worker stopped working. The status takes effect immediately; future inactive dates are rejected by the app.

The workforce Needs attention rows offer Mark inactive beside Add rate. Worker detail offers the same control and Reactivate worker. The worker list hides inactive workers by default and provides Include inactive workers. Inactive workers are excluded from current headcounts, missing-rate alerts, attendance alerts and new clock-ins. A pending same-day clock-out can still be completed. Recorded timesheets, rate history and historical payroll costs stay available. Reactivation clears inactive_from; it does not change rates or hours.

The live migration initialized all 156 existing workers as active. No worker is automatically inactivated because they are absent from a wage list. Run scripts/pb-add-worker-status.py to add the schema fields and initialize blank statuses in another environment; it preserves existing inactive statuses.

Employee rate history: employee_rates

Created by scripts/pb-add-employee-rates.py. One row per employee per effective calendar day; the next effective date implicitly ends the previous period. There is no fixed limit on annual changes.

Field Type Notes
employee required relation → employees Existing worker record; deletion does not cascade
effective_from required date Inclusive business date, written at UTC midnight like date_logged
rate_per_hour required number Positive hourly rate; PocketBase's required number rejects zero
notes text Optional source or reason for change
created, updated autodate Automatic record timestamps

Unique index on employee and the calendar-day portion of effective_from prevents duplicate changes even when timestamps differ within that day. A second employee/date index supports dated lookups. Read rules inherit employee visibility; create/update/delete are superuser-only.

Rollout status: 169 September/October history records are imported; existing employee baseline rates are preserved. The workforce overview uses dated history and provides a missing-rate dialog. Other cost calculations still use employees.rate_per_hour; connect them to dated history before updating that legacy baseline. The imported payroll lists supply previous and new rates, with new rates effective 2026-10-01. Previous rates need an effective date covering their actual known period; do not invent rates for earlier years. Missing historical coverage must be flagged, never silently priced at the newest rate. Imports match employee numbers globally, preserve existing site assignments, and place only missing/unassigned workers at Mega Head Office. History lookups use the date worked, not the upload date or a site's reporting-month boundaries.

4.3 vehicles (unified — includes plant)

Field Type Notes
site relation → sites Current assignment
description text e.g. "Toyota Hilux 2.4 GD-6"
standard_work_hours text/number Free-text on schema; numeric in practice. Used by the plant utilisation calc
fuel_limits text Free-form like "8.5 L/h", "6.5 km/L" — parsed by parseFuelLimit (src/lib/fuel-variance.ts)
qr_code_id text The clean identifier the fuel importer matches against. NOT unique in schema — see multi-site-assessment §5
cost_per_hour number Drives plant utilisation value
unit_type select fuel or plant (default fuel). Splits the bucket between Fleet and Plant pages
attention_dismissed_at date When the user dismissed the vehicle's "over fuel limit" alert from the fuel page
last_latitude number Latest GPS fix from a MEGA Clock Fleet scan. Decimal degrees. Null until first scan
last_longitude number Decimal degrees. Null until first scan
last_scanned_at date UTC midnight anchor of the most recent scan, same convention as fuel_logs.date_logged. Indexed via idx_vehicles_last_scanned
last_location_accuracy_m number Position.coords.accuracy from the kiosk phone, in metres. Diagnostic

The PLANT collection (deprecated legacy alias) was deleted in this restructure — it had 0 rows and 0 code references. All plant items live in vehicles with unit_type="plant".

4.4 fixed_costs

Monthly or one-off costs. is_recurring is the partition key.

Field Type Notes
site relation → sites
date_logged date Drives the monthly tile and trend charts
description, notes text
cost number Rand value
is_recurring bool true (or missing) → monthly bucket; false → one-off bucket. Partitioned by partitionFixedCosts (src/lib/partition-fixed-costs.ts)

4.5 income_claims

Field Type Notes
claim_number text The contractor's IPN / claim ref
site relation → sites
period_start, period_end date The period the claim covers
gross_amount number Rand
certified_by text Signature line on the cert
paid_on date Empty until the claim is actually paid — drives the income series
status select draft / certified / paid
is_verified bool Verified by a human
notes text

4.6 fuel_prices

Unchanged by this restructure. Global (not per-site) pump price history.

Field Type Notes
effective_from date Required, unique (idx_fuel_prices_effective_from)
price_per_litre number Required
notes text optional

4.7 purchase_orders

Now carries the queue state data_import_batches used to carry — see docs/n8n/purchase-order-ingest-implementation.md for the full async pipeline. The row is created at upload time with a placeholder number and processing_status: pending; there is no longer a separate batch row.

Field Type Notes
purchase_order_number text Unique (idx_po_number). Placeholder PENDING-... (src/lib/po-placeholder.ts) until OCR fills it in
site relation → sites, required Resolved by matchSite at upload time
supplier text
required_by_date date When the materials are needed
ordered_by text Who placed it
items_count, total_value number Denormalised roll-ups for table views
purchase_order_file file The original PDF/image; mime allowlist enforced by the upload route
raw_ocr_data json Full OCR payload + ingest metadata
is_verified bool Human-verified flag
file_hash text SHA-256 of the original file; drives the upload-time dedup window
processing_status select pending / processing / processed / failed / dead
attempts number Incremented on each claim
last_error text Last failure reason
next_retry_at, last_attempt_at date Backoff scheduling for the n8n cron
date_uploaded date

Index: idx_po_site_status on (site, processing_status) — the shape the n8n cron's "fetch pending" query and the dashboard's per-site pending count both need.

4.8 purchase_order_items

Joins POs to line items. cascadeDelete: true on the parent relation.

Field Type Notes
purchase_order relation → purchase_orders Required parent
material_code, cost_code text
description text The free-text the OCR matches against in the DN line
qty number Ordered quantity
unit, rate, disc_percent, vat_percent, amount The amount field is the computed total, not stored separately

qty_outstanding no longer exists as a stored field. Outstanding quantity is computed at read time from delivery_note_items via src/lib/po-fulfilment.ts (computeOutstanding) — there is nothing left to drift out of sync with the actual delivery history.

4.9 timesheets (uploaded document)

New collection. One row per uploaded wage sheet — the file itself, plus queue/outcome fields shared with fuel_sheets and plant_hours_sheets.

Field Type Notes
site relation → sites, required A wage sheet belongs to one site, unlike the fuel/plant sheets
source_file file The uploaded bytes
content_type text MIME of the original file
file_hash text SHA-256
date_uploaded, uploaded_by date/text
processing_status select pending / processing / processed / failed / dead
processed_at date
attempts, last_error number/text
rows_total, rows_created, rows_failed number Set by resolveDocumentOutcome() (src/lib/document-status.ts) — never hardcoded to "processed" before rows are actually inserted
row_errors json Up to MAX_ROW_ERRORS (200) per-row failure reasons

4.10 timesheet_entries (renamed from timesheets)

The actual time log: one row per (employee, date, hours). This is the collection that used to be called timesheets before the restructure — watch for stale references to the old name in anything not covered by this rewrite.

Field Type Notes
employee relation → employees
date_logged date
hours_worked number Decimal
site relation → sites, required Stamped from the employee's site at import time — not joined at read time (see §2)
timesheet relation → timesheets, optional Empty means captured directly by QR scan, not from an uploaded sheet — this stays optional forever

Indexes: idx_ts_entries_site_date on (site, date_logged); idx_ts_entries_emp_date on (employee, date_logged).

4.11 fuel_sheets (uploaded document)

Same shape as timesheets (§4.9) except site is optional — a pump sheet can span multiple sites, resolved per-row when the actual fuel_logs rows are created.

Indexes: idx_fuel_sheets_hash on file_hash; idx_fuel_sheets_status on (processing_status, date_uploaded).

4.12 fuel_logs

Field Type Notes
site relation → sites, required Resolved per-row at import time
vehicle relation → vehicles Required by the schema but the importer tolerates a missing relation and falls back to QR
date_logged date ISO datetime; importer anchors at UTC midnight
litres number Required
odo_reading number Required — used for variance calculation
latest_consumption number Optional. Empty cells preserved as undefined rather than 0 (intentional)
latitude, longitude number Optional. Captured by the MEGA Clock Fleet scanner (/api/clock/vehicle-scan); the spreadsheet import path has no GPS context and leaves them null. Decimal degrees. Range validation lives in the route
location_accuracy_m number Optional. Position.coords.accuracy in metres at scan time. Diagnostic only
fuel_sheet relation → fuel_sheets, optional Which uploaded sheet produced this row, if any

Indexes: idx_fuel_logs_site_date on (site, date_logged); idx_fuel_logs_vehicle_date on (vehicle, date_logged).

4.13 plant_hours_sheets (uploaded document)

Identical shape to fuel_sheets (§4.11), same optional-site reasoning.

Indexes: idx_plant_hours_sheets_hash on file_hash; idx_plant_hours_sheets_status on (processing_status, date_uploaded).

4.14 plant_hours_log

The most disciplined collection. Every site-attribution field is required.

Field Type Notes
date_logged, start_date, end_date date
vehicle_qr text The QR text — part of the composite unique index below
vehicle relation → vehicles Required (minSelect=1)
site relation → sites Required (minSelect=1)
start_hours, end_hours, hours number
hour_meter_budget, cost_per_hour, value number value = hours × cost_per_hour
over_budget, hours_mismatch bool Computed at import time
vehicle_unit_type_at_import select Snapshot of the vehicle's unit_type at the moment of import — see the note in multi-site-assessment §5 about why this discipline should extend to site
plant_hours_sheet relation → plant_hours_sheets, optional Which uploaded sheet produced this row, if any

Unique index: (vehicle_qr, start_date). Prevents accidental double-imports of the same week for the same vehicle. Surfaced as a friendly "duplicate: vehicle_qr=X start_date=Y" error in the import response. Additional index: idx_ph_log_vehicle_start on (vehicle, start_date).

4.15 delivery_notes

Field Type Notes
processing_status select pending → ocr-complete → auto-verified / needs-review / manual-verified / failed
delivery_note_number text Optional; OCR may fill it in
site relation → sites, required Resolved at upload time
purchase_order relation → purchase_orders Filled by OCR (exact → normalised → Levenshtein) or by manual assign-po route
date_received date
received_by text The uploader's identity — currently always "web-upload"
slip_image file The photo/PDF, served via /api/delivery-notes/:id/slip to n8n
raw_ocr_data json Full OCR payload + match metadata
is_verified bool Final flip from auto-verified or manual-verified
ocr_confidence number 0..100 integer
ocr_completed_at date
received_by_name text Signature OCR'd from the slip
file_hash text SHA-256, used for upload-time dedup

Indexes: idx_dn_site_date on (site, date_received); idx_dn_status on processing_status.

4.16 delivery_note_items (renamed from received_items)

Joins DNs to PO line items. cascadeDelete: true on the parent DN.

Field Type Notes
delivery_note relation → delivery_notes Required parent
purchase_order_item relation → purchase_order_items Optional — can be empty for items the OCR couldn't match, or for orphans where the PO item was deleted between fetch and create
qty_received number
condition_notes text Free-text audit trail (e.g. "OCR 92% · desc: Cement 50kg · unit: bag")

4.17 ask_threads, ask_messages, ask_log (Ask agent actions)

Additive, created by scripts/pb-add-ask-collections.py (--apply is a human step). All rules null — superuser only; the Next.js server is the only client. See modules.md §7.1 for how these back threads, memory, actions and change requests.

ask_threads — one row per conversation.

Field Type Notes
title text autoTitle() of the first message, or renamed by the user
owner relation → users, required The picked actor (x-ask-user)
archived bool
last_message_at date Bumped on every appendMessage; a lost race here is harmless

ask_messages — one row per message, attachment or generated image. Append-only: created once, never rewritten, so concurrent tabs/users can't overwrite each other.

Field Type Notes
thread relation → ask_threads, required, cascade delete
owner relation → users
kind select user / assistant / attachment / image
text text (max 100 000) The message; for an attachment, its summary; for an image, the prompt
display text The slash command as typed, when applicable
charts json ChartSpec[] the answer placed or left unplaced
action_ids, attachment_ids, tools json Arrays of ids / tool names
file file (≤25 MB; xlsx/xls/csv/pdf/png/jpg/webp/txt) Attachment or generated image
parsed json (≤5 MB) Attachment preview (capped 2 000 rows / 50 000 chars — read_attachment re-parses the stored file beyond that), or { pdf: {...} } for a report turn
parse_error text

ask_log — one row per thing Ask keeps track of, by type.

Field Type Notes
type select change (proposed data edit, for Undo) / memory (lasting fact) / request (app change request)
owner relation → users
shared bool Memories only — visible to everyone when true
thread relation → ask_threads (optional, no cascade) The log outlives a deleted thread
status select change: pending/applied/rejected/failed/undone. memory: applied (active) / undone (forgotten). request: open/planned/done/declined
text text (max 2 000) Change summary / memory text / request title
detail text Request detail
site relation → sites (optional) A memory about one site
ops json The planned AgentOp[], $refs intact
before, after json Snapshots for Undo. before: one entry per op. after: while pending, the proposed values per op; once applied/failed, { results, confirmed_by, undone_by? }
error text Set on applied_action → failed
applied_at, undone_at date

Indexes: ask_threads(owner, archived, last_message_at), ask_messages(thread, created), ask_log(type, status), ask_log(owner, type), ask_log(thread).


5. Indexes and constraints summary

Index Where Purpose
idx_sites_project_code sites(project_code) UNIQUE Prevents duplicate site codes; enforced on the POST /api/sites route
idx_po_number purchase_orders(purchase_order_number) UNIQUE Prevents duplicate PO numbers — final backstop behind the app-level dedup in process-batch
idx_po_site_status purchase_orders(site, processing_status) Per-site pending-PO counts and the n8n cron's fetch-pending query
idx_plant_hours_log_qr_start plant_hours_log(vehicle_qr, start_date) UNIQUE Prevents accidental double-imports of the same week for the same vehicle
idx_ph_log_vehicle_start plant_hours_log(vehicle, start_date) Per-vehicle history queries
idx_fuel_prices_effective_from fuel_prices(effective_from) UNIQUE One price per effective date
idx_dn_site_date delivery_notes(site, date_received) Per-site DN listing
idx_dn_status delivery_notes(processing_status) Pending-verification dashboards
idx_fuel_logs_site_date, idx_fuel_logs_vehicle_date fuel_logs Per-site / per-vehicle fuel history
idx_vehicles_last_scanned vehicles(last_scanned_at) "Scanned in the last X hours" filters on the map page and the future attention rail
idx_ts_entries_site_date, idx_ts_entries_emp_date timesheet_entries Per-site / per-employee timesheet queries
idx_timesheets_hash, idx_timesheets_status timesheets (document) Upload dedup + pending-import dashboards
idx_fuel_sheets_hash, idx_fuel_sheets_status fuel_sheets Same, for fuel sheet uploads
idx_plant_hours_sheets_hash, idx_plant_hours_sheets_status plant_hours_sheets Same, for plant hours sheet uploads

No collection outside the above has a server-enforced unique constraint. In particular, vehicles.qr_code_id is not unique — see multi-site-assessment §5.


6. PocketBase API rules

scripts/pb-relax-rules.py writes the ruleset used in production. For every collection:

  • listRule, viewRule, createRule → @request.auth.id != '' (any authenticated user)
  • updateRule, deleteRule → null (superuser only)

The collection list is the full set of 16 — there is no collection where clients can update or delete records through the API. The /api/records/[collection]/[id]/verify route hand-picks two collections (purchase_orders, delivery_notes) for the is_verified toggle, but it runs server-side as pbAdmin(), so it is not constrained by these rules.

The only routes that respect the API rules at all are the (currently unused) browser-side pb client and the n8n file-proxy route /api/delivery-notes/:id/slip, which uses the admin token in its Authorization header. The PB layer never blocks the Next.js server.


7. Schema drift

pb-bootstrap-schema.py is the canonical schema definition — a fresh PocketBase must come up in the exact shape production is in. That script had drifted ten-plus migrations behind production before the 2026-09-11 restructure's final task (Task 20) brought it back to truth: it was missing fuel_prices and income_claims entirely, still described the deleted data_import_batches and plant collections, still carried project_code on eight collections that no longer have it, still had qty_outstanding on purchase_order_items, and was missing every index added across the restructure.

That drift is exactly the failure mode this section exists to name, so the rule going forward is unchanged from before: any time the schema changes, the new field must be added to both pb-bootstrap-schema.py (so a fresh PB starts in the right shape) and src/lib/pb-types.ts (so the TypeScript code agrees), in the same change that adds it to production. Forgetting either side is a known foot-gun — it is what produced this section's rewrite in the first place. scripts/pb-relax-rules.py's collection list is the third thing that must move in lockstep; it drifted just as far (missing fuel_prices, income_claims and plant_hours_log entirely, on top of the same renames/deletions).

One-off data-migration scripts under scripts/migrations/* are expected to exist and are not a drift signal by themselves — they are how the schema got to its current shape. The drift signal is pb-bootstrap-schema.py (or pb-relax-rules.py) disagreeing with what a migration already did to production.