MEGAINFRA-WEBAPP/docs/data-model.md

Data model

Every PocketBase collection that backs the app, with fields, relations, and the site-attribution strategy each one uses.

Source of truth for types: src/lib/pb-types.ts (13 collection interfaces). Source of truth for the live schema: scripts/pb-bootstrap-schema.py. Production instance: https://pb.mega.infra.apphub.co.za.

---

1. Entity-relationship diagram

erDiagram
    SITES ||--o{ EMPLOYEES : "site (relation)"
    SITES ||--o{ VEHICLES : "site (relation)"
    SITES ||--o{ FIXED_COSTS : "site (relation)"
    SITES ||--o{ INCOME_CLAIMS : "site (relation)"
    SITES ||--o{ PURCHASE_ORDERS : "site (relation)"
    SITES ||--o{ DELIVERY_NOTES : "site (relation)"
    SITES ||--o{ FUEL_LOGS : "site (relation)"
    SITES ||--o{ PLANT_HOURS_LOG : "site (relation)"

    EMPLOYEES ||--o{ TIMESHEETS : "employee (relation)"
    VEHICLES ||--o{ FUEL_LOGS : "vehicle (relation)"
    VEHICLES ||--o{ PLANT_HOURS_LOG : "vehicle (relation)"
    DATA_IMPORT_BATCHES ||--o{ FUEL_LOGS : "batch_reference (relation)"
    DATA_IMPORT_BATCHES ||--o{ TIMESHEETS : "batch_reference (relation)"
    DATA_IMPORT_BATCHES ||--o{ PLANT_HOURS_LOG : "batch_reference (loose)"

    PURCHASE_ORDERS ||--|{ PURCHASE_ORDER_ITEMS : "purchase_order (cascade)"
    PURCHASE_ORDERS ||--o{ DELIVERY_NOTES : "purchase_order"
    DELIVERY_NOTES ||--|{ RECEIVED_ITEMS : "delivery_note (cascade)"
    PURCHASE_ORDER_ITEMS ||--o{ RECEIVED_ITEMS : "purchase_order_item"

The PLANT collection is intentionally absent from the diagram — it is a deprecated legacy alias for the unified VEHICLES collection (src/lib/pb-types.ts:58-60).

---

2. Site attribution strategy, per collection

This is the table to read first. The question "where does this row belong?" has three different answers depending on which collection you're in.

Collection	site relation	project_code text	Attribution source	Drift risk
sites	n/a	required, unique	The site row itself	None — the unique index is authoritative
employees	✅	✅	Both, but site is canonical for metrics; project_code is denormalised at create time	Medium — changing a worker's site later leaves project_code stale
vehicles	✅	✅	Both, but site is the relation; project_code is denormalised at create time	High — fuel logs are filtered by log.site, not by the current vehicle's site
fixed_costs	✅	✅	site is the relation; project_code is denormalised	Low — the import routes always stamp both from the same source
income_claims	✅	✅	Same as fixed_costs	Low
purchase_orders	✅	✅	Same as fixed_costs	Low
delivery_notes	✅	✅	Same as fixed_costs	Low
fuel_logs	✅	✅	site is filled when the source row has a site; project_code is the canonical fallback when the site is missing	Medium — see multi-site-assessment.md §5
plant_hours_log	✅ required	✅ required	Always set from the vehicle at import time	Lower than fuel_logs because required
timesheets	❌	❌	Joined via employees.site at read time — timesheets has no site or project_code of its own	High — site attribution is a per-employee property that can move
purchase_order_items	❌	❌	Inherited through purchase_order → site	None — never queried directly by site
received_items	❌	❌	Inherited through delivery_note → site	None — same
data_import_batches	❌	❌	Inferred from processed_record_id + the processed record's collection	n/a

Reading the table. Every collection except the two joining-tables and data_import_batches carries both a site relation and a project_code string. The only collection that carries neither is timesheets — and that's the collection whose site attribution drifts the most, because it's synthesised at read time from the employee record.

---

3. The project_code denormalisation

project_code is a free-text identifier like MEG-001 or HILLS-2026 that uniquely names a contract. It is the human-readable label that appears on every report and filter dropdown.

The schema carries it on every child collection as a denormalised copy of the parent site's project_code. It is not a foreign key — it is just a string, written at insert time, never updated when the parent site changes.

Why it exists. The chat agent and several older reports filter by project_code (text) rather than expanding the site relation (expand: "site" would force a join the dashboard pages don't want). It is also what the import routes use as the canonical key when a row's site relation is empty.

The drift problem. If you change a site's project_code in PocketBase, every existing fuel_logs, fixed_costs, employees, vehicles, purchase_orders, delivery_notes, and income_claims row keeps the old string forever. Code that joins through the site relation sees the new value; code that filters on project_code text sees the old.

scripts/pb-relax-rules.py:24 and the PATCH /api/sites/[id] route (src/app/api/sites/[id]/route.ts:23-28) deliberately refuse to let you change project_code on an existing site — that is exactly the rule designed to prevent drift. The POST /api/sites route also enforces a 64-char ceiling and uniqueness via the idx_sites_project_code index (scripts/pb-bootstrap-schema.py:84).

What this means in practice: the denormalisation is stable as long as no one re-creates a site with the same name and a different code. The biggest remaining risk is bulk imports that resolve site by name and stamp the wrong project_code onto historical rows — see multi-site-assessment.md §4.

---

4. Collection reference

Each entry below lists every typed field, the relation direction, and any notable constraints. Optional/legacy fields are marked.

4.1 sites

The site is the unit of business. One row per construction contract.

Field	Type	Required	Notes
project_code	text	✅	Unique (idx_sites_project_code). Locked after creation.
name	text	optional	Human label, e.g. "The Hills"
duration	text	optional	Free-text duration ("8 months", "52 weeks") — parsed by parseDurationMonths
contract_value	number	optional	Rand value, no currency field
work_hours	number	optional	Total project work hours
hours_per_week	number	optional	Standard workweek for the site
wages_hours_per_day	number	optional	Standard wage day length
job_start_date	date	optional	ISO date; the projection helper treats this as the authoritative start when present

4.2 employees

Field	Type	Notes
project_code	text	Denormalised from site.project_code at create time
site	relation → sites	The site the worker is currently assigned to
name, employee_no, position	text	employee_no is the cross-system payroll id; uppercased on import (src/app/api/timesheets/import/route.ts:289)
rate_per_hour	number	Drives payroll burn
hours_per_day_standard	number	9 by default for auto-created casuals
worker_type	select	permanent or casual

4.3 vehicles (unified — includes plant)

Field	Type	Notes
project_code	text	Denormalised
site	relation → sites	Current assignment
description	text	e.g. "Toyota Hilux 2.4 GD-6"
standard_work_hours	text/number	Free-text on schema; numeric in practice. Used by the plant utilisation calc
fuel_limits	text	Free-form like "8.5 L/h", "6.5 km/L" — parsed by parseFuelLimit (src/lib/fuel-variance.ts)
qr_code_id	text	The clean identifier the fuel importer matches against. NOT unique in schema — see multi-site-assessment §5
cost_per_hour	number	Drives plant utilisation value
unit_type	select	fuel or plant (default fuel). Splits the bucket between Fleet and Plant pages
attention_dismissed_at	date	When the user dismissed the vehicle's "over fuel limit" alert from the fuel page

The legacy plant collection is a deprecated alias kept around for old importers. New plant items go in vehicles with unit_type="plant".

4.4 fixed_costs

Monthly or one-off costs. is_recurring is the partition key.

Field	Type	Notes
project_code	text	Denormalised
site	relation → sites
date_logged	date	Drives the monthly tile and trend charts
description, notes	text
cost	number	Rand value
is_recurring	bool	true (or missing) → monthly bucket; false → one-off bucket. Partitioned by partitionFixedCosts (src/lib/partition-fixed-costs.ts)

4.5 income_claims

Field	Type	Notes
claim_number	text	The contractor's IPN / claim ref
project_code	text	Denormalised
site	relation → sites
period_start, period_end	date	The period the claim covers
gross_amount	number	Rand
certified_by	text	Signature line on the cert
paid_on	date	Empty until the claim is actually paid — drives the income series
status	select	draft / certified / paid
is_verified	bool	Verified by a human
notes	text

4.6 purchase_orders

Field	Type	Notes
purchase_order_number	text	Unique (idx_po_number). Normalised to uppercase with "PO" prefix stripped before insert (src/lib/purchase-order-schema.ts)
project_code	text	Denormalised
site	relation → sites	Resolved by matchSite at ingest time
supplier	text
required_by_date	date	When the materials are needed
ordered_by	text	Who placed it
items_count, total_value	number	Denormalised roll-ups for table views
purchase_order_file	file	The original PDF/image; mime allowlist enforced by the upload route
raw_ocr_data	json	Full OCR payload + ingest metadata
is_verified	bool	Human-verified flag

4.7 purchase_order_items

Joins POs to line items. cascadeDelete: true on the parent relation (scripts/pb-bootstrap-schema.py:132).

Field	Type	Notes
purchase_order	relation → purchase_orders	Required parent
material_code, cost_code	text
description	text	The free-text the OCR matches against in the DN line
qty	number	Ordered quantity
qty_outstanding	number	Decreased as received_items roll up — but the canonical fulfilment % is computed from received_items directly (see /dashboard/purchasing page and the orphan-id note in src/app/dashboard/purchasing/page.tsx:52-72)
unit, rate, disc_percent, vat_percent, amount		The amount field is the computed total, not stored separately

4.8 timesheets

The collection with no site and no project_code. Site is resolved through employeeRates at every read site.

Field	Type	Notes
employee	relation → employees
batch_reference	relation → data_import_batches
date_logged	date
hours_worked	number	Decimal

4.9 fuel_logs

Field	Type	Notes
project_code	text	Denormalised, falls back to vehicle.project_code when the source row has no site
site	relation → sites	Optional — empty when the source row had no site column
vehicle	relation → vehicles	Required by the schema but the importer tolerates a missing relation and falls back to QR
batch_reference	relation → data_import_batches
date_logged	date	ISO datetime; importer anchors at UTC midnight
litres	number	Required
odo_reading	number	Required — used for variance calculation
latest_consumption	number	Optional. Empty cells preserved as undefined rather than 0 (intentional, see comment at src/lib/pb-types.ts:173-181)

4.10 plant_hours_log

The most disciplined collection. Every site-attribution field is required.

Field	Type	Notes
date_logged, start_date, end_date	date
vehicle_qr	text	The QR text — part of the composite unique index below
vehicle	relation → vehicles	Required (minSelect=1)
site	relation → sites	Required (minSelect=1)
project_code	text	Required
start_hours, end_hours, hours	number
hour_meter_budget, cost_per_hour, value	number	value = hours × cost_per_hour
over_budget, hours_mismatch	bool	Computed at import time
vehicle_unit_type_at_import	select	Snapshot of the vehicle's unit_type at the moment of import — see the note in multi-site-assessment §5 about why this discipline should extend to site
batch_reference	relation	Optional

Unique index: (vehicle_qr, start_date). Prevents accidental double-imports of the same week for the same vehicle (scripts/pb-bootstrap-schema.py:257). Surfaced as a friendly "duplicate: vehicle_qr=X start_date=Y" error in the import response.

4.11 delivery_notes

Field	Type	Notes
delivery_note_number	text	Optional; OCR may fill it in
project_code	text	Denormalised
site	relation → sites	Resolved at upload time
purchase_order	relation → purchase_orders	Filled by OCR (exact → normalised → Levenshtein) or by manual assign-po route
date_received	date
received_by	text	The uploader's identity — currently always "web-upload"
slip_image	file	The photo/PDF, served via /api/delivery-notes/:id/slip to n8n
raw_ocr_data	json	Full OCR payload + match metadata
processing_status	select	pending → ocr-complete → auto-verified / needs-review / manual-verified / failed
ocr_confidence	number	0..100 integer
ocr_completed_at	date
received_by_name	text	Signature OCR'd from the slip
is_verified	bool	Final flip from auto-verified or manual-verified

4.12 received_items

Joins DNs to PO line items. cascadeDelete: true on the parent DN.

Field	Type	Notes
delivery_note	relation → delivery_notes	Required parent
purchase_order_item	relation → purchase_order_items	Optional — can be empty for items the OCR couldn't match, or for orphans where the PO item was deleted between fetch and create
qty_received	number
condition_notes	text	Free-text audit trail (e.g. "OCR 92% · desc: Cement 50kg · unit: bag")

4.13 data_import_batches

The audit-trail spine. Every uploaded file — Excel, CSV, PDF, image — has a batch row that tracks its lifecycle.

Field	Type	Notes
batch_type	select	timesheets / fuel_logs / purchase_orders / delivery_notes / plant_hours
content_type	text	MIME of the original file
original_file	file	The uploaded bytes; mime-restricted per batch_type
date_uploaded	date
uploaded_by	text	Currently always "web-import" or "web-upload"
status	select	pending / processing / processed / failed / dead
attempts	number	Incremented on each retry
last_error	text	Last failure reason
last_attempt_at	date
next_retry_at	date
processed_at	date	When the batch reached processed
processed_record_id	text	The id of the record this batch created (PO id, DN id, etc.)
file_hash	text	SHA-256 of the original file; the dedup window for PO re-uploads uses this index

---

5. Indexes and constraints summary

Index	Where	Purpose
idx_sites_project_code	sites(project_code) UNIQUE	Prevents duplicate site codes; enforced on the POST /api/sites route via the idx_sites_project_code name match (src/app/api/sites/route.ts:64)
idx_po_number	purchase_orders(purchase_order_number) UNIQUE	Prevents duplicate PO numbers — works alongside the AI extraction dedup by purchase_order_number
idx_plant_hours_log_qr_start	plant_hours_log(vehicle_qr, start_date) UNIQUE	Prevents accidental double-imports of the same week for the same vehicle; surfaced as duplicate: reason in the importer
idx_batches_file_hash	data_import_batches(file_hash, status)	Used by the PO upload dedup window (src/app/api/purchase-orders/upload/route.ts:80-84)

No collection outside the above has a server-enforced unique constraint. In particular, vehicles.qr_code_id and vehicles.project_code are not unique — see multi-site-assessment §5.

---

6. PocketBase API rules

scripts/pb-relax-rules.py writes the ruleset used in production. For every collection:

• listRule, viewRule, createRule → @request.auth.id != '' (any authenticated user)
• updateRule, deleteRule → null (superuser only)

The collection list is the full set of 13 — there is no collection where clients can update or delete records through the API. The /api/records/[collection]/[id]/verify route hand-picks two collections (purchase_orders, delivery_notes) for the is_verified toggle, but it runs server-side as pbAdmin(), so it is not constrained by these rules.

The only routes that respect the API rules at all are the (currently unused) browser-side pb client and the n8n file-proxy route /api/delivery-notes/:id/slip, which uses the admin token in its Authorization header. The PB layer never blocks the Next.js server.

---

7. Schema drift

The pb-bootstrap-schema.py script is the canonical schema definition. The following migrations exist on disk and are not represented in the script:

Migration	Purpose
migrate-delivery-notes-ocr.py	Adds the processing_status / ocr_* fields and the OCR webhook hooks to existing DNs
migrate-plant-hours-backfill-vehicle.py	Backfills plant_hours_log.vehicle relations from vehicle_qr for pre-relation rows
scripts/migrations/* (other)	One-off patches for specific schema or data issues

Any time the schema changes, the new field must be added to both pb-bootstrap-schema.py (so a fresh PB starts in the right shape) and src/lib/pb-types.ts (so the TypeScript code agrees). Forgetting either side is a known foot-gun.