MEGAINFRA-WEBAPP/docs/data-model.md

# Data model

> Every PocketBase collection that backs the app, with fields, relations, and
> the site-attribution strategy each one uses.

Source of truth for types: `src/lib/pb-types.ts` (16 collection interfaces).
Source of truth for the live schema: `scripts/pb-bootstrap-schema.py`.
Production instance: `https://pb.mega.infra.apphub.co.za`.

This reflects the shape produced by the 2026-09-11 ingestion restructure
(`docs/superpowers/plans/2026-09-11-ingestion-restructure.md`): the shared
`data_import_batches` pool is gone, replaced by typed document collections
(`timesheets`, `fuel_sheets`, `plant_hours_sheets`) and, for purchase orders,
queue fields carried directly on `purchase_orders` itself. `project_code`
denormalisation is gone everywhere except `sites`.

---

## 1. Entity-relationship diagram

```mermaid
erDiagram
    SITES ||--o{ EMPLOYEES : "site (relation)"
    SITES ||--o{ VEHICLES : "site (relation)"
    SITES ||--o{ FIXED_COSTS : "site (relation)"
    SITES ||--o{ INCOME_CLAIMS : "site (relation)"
    SITES ||--o{ PURCHASE_ORDERS : "site (relation)"
    SITES ||--o{ DELIVERY_NOTES : "site (relation)"
    SITES ||--o{ FUEL_LOGS : "site (relation)"
    SITES ||--o{ PLANT_HOURS_LOG : "site (relation)"
    SITES ||--o{ TIMESHEET_ENTRIES : "site (relation, stamped at import)"
    SITES ||--o{ TIMESHEETS : "site (relation, required)"
    SITES ||--o{ FUEL_SHEETS : "site (relation, optional)"
    SITES ||--o{ PLANT_HOURS_SHEETS : "site (relation, optional)"

    EMPLOYEES ||--o{ TIMESHEET_ENTRIES : "employee (relation)"
    VEHICLES ||--o{ FUEL_LOGS : "vehicle (relation)"
    VEHICLES ||--o{ PLANT_HOURS_LOG : "vehicle (relation)"
    TIMESHEETS ||--o{ TIMESHEET_ENTRIES : "timesheet (relation, optional)"
    FUEL_SHEETS ||--o{ FUEL_LOGS : "fuel_sheet (relation, optional)"
    PLANT_HOURS_SHEETS ||--o{ PLANT_HOURS_LOG : "plant_hours_sheet (relation, optional)"

    PURCHASE_ORDERS ||--|{ PURCHASE_ORDER_ITEMS : "purchase_order (cascade)"
    PURCHASE_ORDERS ||--o{ DELIVERY_NOTES : "purchase_order"
    DELIVERY_NOTES ||--|{ DELIVERY_NOTE_ITEMS : "delivery_note (cascade)"
    PURCHASE_ORDER_ITEMS ||--o{ DELIVERY_NOTE_ITEMS : "purchase_order_item"
```

`data_import_batches` and the deprecated `plant` alias (0 rows, 0 code
references) were deleted entirely as part of this restructure — neither
exists in production any more.

---

## 2. Site attribution strategy, per collection

This is the table to read first. Every non-joining collection now carries a
real `site` relation; `project_code` survives only on `sites` itself, as the
one canonical, human-readable label.

| Collection | `site` relation | Attribution source | Drift risk |
| --- | --- | --- | --- |
| `sites` | n/a | The site row itself; `project_code` is required + unique | None — the unique index is authoritative |
| `employees` | ✅ optional | The worker's current site assignment | Low — read directly, nothing denormalised to go stale |
| `vehicles` | ✅ optional | The vehicle's current site assignment | Low — same |
| `fixed_costs` | ✅ optional | Stamped by the entry form | Low |
| `income_claims` | ✅ optional | Stamped by the entry form | Low |
| `purchase_orders` | ✅ **required** | Resolved by `matchSite` at upload time | Low — required field, no null-site POs |
| `delivery_notes` | ✅ **required** | Stamped at upload time from the form | Low |
| `fuel_logs` | ✅ **required** | Resolved per-row during sheet import (site cell, or vehicle's current site as fallback) | Low — required field |
| `plant_hours_log` | ✅ **required** | Always set from the matched vehicle's site at import time | Low — required field |
| `timesheets` (document) | ✅ **required** | A wage sheet belongs to one site, stamped at upload | None |
| `timesheet_entries` | ✅ **required** | **Stamped from the employee's site at import time** — not joined at read time | None — this ends the historical drift the old `timesheets` collection had when a worker moved sites after the fact |
| `fuel_sheets` (document) | ✅ optional | A pump sheet can span sites; left empty and resolved per-row against `fuel_logs.site` | Low |
| `plant_hours_sheets` (document) | ✅ optional | Same reasoning as `fuel_sheets` | Low |
| `purchase_order_items` | ❌ | Inherited through `purchase_order → site` | None — never queried directly by site |
| `delivery_note_items` | ❌ | Inherited through `delivery_note → site` | None — same |

**Reading the table.** `timesheet_entries` (the renamed collection that used
to be called `timesheets`, and used to have no site of its own) is no longer
the worst drift risk in the system — it now carries a `site` relation that is
**stamped once, at import time**, from the employee's site as it was that
day. A worker moving sites later does not silently rewrite history the way
the old at-read-time join through `employees.site` did. The two join tables
(`purchase_order_items`, `delivery_note_items`) remain the only collections
with no site attribution of their own, by design — they're always reached
through their parent.

---

## 3. The `project_code` denormalisation — removed

Before this restructure, `project_code` (a free-text identifier like
`MEG-001`) was copied onto every child collection as a denormalised string
at insert time, alongside the `site` relation. That copy was never updated
when the parent site's `project_code` changed, which was the single biggest
drift risk in the schema (see the old §3 in git history for the full
writeup).

**Current state:** `project_code` now lives in exactly one place —
`sites.project_code`, required and unique
(`idx_sites_project_code`, enforced in `scripts/pb-bootstrap-schema.py`).
Every other collection that needs a human-readable site label resolves it by
expanding or joining the `site` relation at read time
(`src/lib/agent/load.ts`'s `loadSiteRefs()` is the canonical example). There
is no denormalised copy left anywhere to drift.

`scripts/pb-relax-rules.py` and the `PATCH /api/sites/[id]` route
(`src/app/api/sites/[id]/route.ts`) still refuse to let you change
`project_code` on an existing site — that rule now protects the single
source of truth rather than a chain of denormalised copies, but it is
exactly as important as it always was: renaming a site is a new site, not an
edit.

---

## 4. Collection reference

Each entry below lists every typed field, the relation direction, and any
notable constraints.

### 4.1 `sites`

The site is the unit of business. One row per construction contract.

| Field | Type | Required | Notes |
| --- | --- | --- | --- |
| `project_code` | text | ✅ | Unique (`idx_sites_project_code`). Locked after creation. |
| `name` | text | optional | Human label, e.g. "The Hills" |
| `duration` | text | optional | Free-text duration ("8 months", "52 weeks") — parsed by `parseDurationMonths` |
| `contract_value` | number | optional | Rand value, no currency field |
| `work_hours` | number | optional | Total project work hours |
| `hours_per_week` | number | optional | Standard workweek for the site |
| `wages_hours_per_day` | number | optional | Standard wage day length |
| `latitude`, `longitude` | number | optional | Site coordinates |
| `job_start_date` | date | optional | ISO date; the projection helper treats this as the authoritative start when present |

### 4.2 `employees`

| Field | Type | Notes |
| --- | --- | --- |
| `site` | relation → `sites` | The site the worker is currently assigned to |
| `name`, `employee_no`, `position` | text | `employee_no` is the cross-system payroll id; **uppercased** on import |
| `rate_per_hour` | number | Drives payroll burn |
| `hours_per_day_standard` | number | 9 by default for auto-created casuals |
| `worker_type` | select | `permanent` or `casual` |

### 4.3 `vehicles` (unified — includes plant)

| Field | Type | Notes |
| --- | --- | --- |
| `site` | relation → `sites` | Current assignment |
| `description` | text | e.g. "Toyota Hilux 2.4 GD-6" |
| `standard_work_hours` | text/number | Free-text on schema; numeric in practice. Used by the plant utilisation calc |
| `fuel_limits` | text | Free-form like `"8.5 L/h"`, `"6.5 km/L"` — parsed by `parseFuelLimit` (`src/lib/fuel-variance.ts`) |
| `qr_code_id` | text | The clean identifier the fuel importer matches against. NOT unique in schema — see [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) |
| `cost_per_hour` | number | Drives plant utilisation value |
| `unit_type` | select | `fuel` or `plant` (default `fuel`). Splits the bucket between Fleet and Plant pages |
| `attention_dismissed_at` | date | When the user dismissed the vehicle's "over fuel limit" alert from the fuel page |

The `PLANT` collection (deprecated legacy alias) was deleted in this
restructure — it had 0 rows and 0 code references. All plant items live in
`vehicles` with `unit_type="plant"`.

### 4.4 `fixed_costs`

Monthly or one-off costs. `is_recurring` is the partition key.

| Field | Type | Notes |
| --- | --- | --- |
| `site` | relation → `sites` | |
| `date_logged` | date | Drives the monthly tile and trend charts |
| `description`, `notes` | text | |
| `cost` | number | Rand value |
| `is_recurring` | bool | `true` (or missing) → monthly bucket; `false` → one-off bucket. Partitioned by `partitionFixedCosts` (`src/lib/partition-fixed-costs.ts`) |

### 4.5 `income_claims`

| Field | Type | Notes |
| --- | --- | --- |
| `claim_number` | text | The contractor's IPN / claim ref |
| `site` | relation → `sites` | |
| `period_start`, `period_end` | date | The period the claim covers |
| `gross_amount` | number | Rand |
| `certified_by` | text | Signature line on the cert |
| `paid_on` | date | Empty until the claim is actually paid — drives the income series |
| `status` | select | `draft` / `certified` / `paid` |
| `is_verified` | bool | Verified by a human |
| `notes` | text | |

### 4.6 `fuel_prices`

Unchanged by this restructure. Global (not per-site) pump price history.

| Field | Type | Notes |
| --- | --- | --- |
| `effective_from` | date | Required, unique (`idx_fuel_prices_effective_from`) |
| `price_per_litre` | number | Required |
| `notes` | text | optional |

### 4.7 `purchase_orders`

Now carries the queue state `data_import_batches` used to carry — see
`docs/n8n/purchase-order-ingest-implementation.md` for the full async
pipeline. The row is created at upload time with a placeholder number and
`processing_status: pending`; there is no longer a separate batch row.

| Field | Type | Notes |
| --- | --- | --- |
| `purchase_order_number` | text | Unique (`idx_po_number`). Placeholder `PENDING-...` (`src/lib/po-placeholder.ts`) until OCR fills it in |
| `site` | relation → `sites`, **required** | Resolved by `matchSite` at upload time |
| `supplier` | text | |
| `required_by_date` | date | When the materials are needed |
| `ordered_by` | text | Who placed it |
| `items_count`, `total_value` | number | Denormalised roll-ups for table views |
| `purchase_order_file` | file | The original PDF/image; mime allowlist enforced by the upload route |
| `raw_ocr_data` | json | Full OCR payload + ingest metadata |
| `is_verified` | bool | Human-verified flag |
| `file_hash` | text | SHA-256 of the original file; drives the upload-time dedup window |
| `processing_status` | select | `pending` / `processing` / `processed` / `failed` / `dead` |
| `attempts` | number | Incremented on each claim |
| `last_error` | text | Last failure reason |
| `next_retry_at`, `last_attempt_at` | date | Backoff scheduling for the n8n cron |
| `date_uploaded` | date | |

Index: `idx_po_site_status` on `(site, processing_status)` — the shape the
n8n cron's "fetch pending" query and the dashboard's per-site pending count
both need.

### 4.8 `purchase_order_items`

Joins POs to line items. `cascadeDelete: true` on the parent relation.

| Field | Type | Notes |
| --- | --- | --- |
| `purchase_order` | relation → `purchase_orders` | Required parent |
| `material_code`, `cost_code` | text | |
| `description` | text | The free-text the OCR matches against in the DN line |
| `qty` | number | Ordered quantity |
| `unit`, `rate`, `disc_percent`, `vat_percent`, `amount` | | The amount field is the **computed** total, not stored separately |

`qty_outstanding` no longer exists as a stored field. Outstanding quantity
is computed at read time from `delivery_note_items` via
`src/lib/po-fulfilment.ts` (`computeOutstanding`) — there is nothing left to
drift out of sync with the actual delivery history.

### 4.9 `timesheets` (uploaded document)

New collection. One row per uploaded wage sheet — the file itself, plus
queue/outcome fields shared with `fuel_sheets` and `plant_hours_sheets`.

| Field | Type | Notes |
| --- | --- | --- |
| `site` | relation → `sites`, **required** | A wage sheet belongs to one site, unlike the fuel/plant sheets |
| `source_file` | file | The uploaded bytes |
| `content_type` | text | MIME of the original file |
| `file_hash` | text | SHA-256 |
| `date_uploaded`, `uploaded_by` | date/text | |
| `processing_status` | select | `pending` / `processing` / `processed` / `failed` / `dead` |
| `processed_at` | date | |
| `attempts`, `last_error` | number/text | |
| `rows_total`, `rows_created`, `rows_failed` | number | Set by `resolveDocumentOutcome()` (`src/lib/document-status.ts`) — never hardcoded to "processed" before rows are actually inserted |
| `row_errors` | json | Up to `MAX_ROW_ERRORS` (200) per-row failure reasons |

### 4.10 `timesheet_entries` (renamed from `timesheets`)

The actual time log: one row per (employee, date, hours). This is the
collection that used to be called `timesheets` before the restructure —
watch for stale references to the old name in anything not covered by this
rewrite.

| Field | Type | Notes |
| --- | --- | --- |
| `employee` | relation → `employees` | |
| `date_logged` | date | |
| `hours_worked` | number | Decimal |
| `site` | relation → `sites`, **required** | Stamped from the employee's site at import time — not joined at read time (see §2) |
| `timesheet` | relation → `timesheets`, optional | Empty means captured directly by QR scan, not from an uploaded sheet — this stays optional forever |

Indexes: `idx_ts_entries_site_date` on `(site, date_logged)`;
`idx_ts_entries_emp_date` on `(employee, date_logged)`.

### 4.11 `fuel_sheets` (uploaded document)

Same shape as `timesheets` (§4.9) except `site` is optional — a pump sheet
can span multiple sites, resolved per-row when the actual `fuel_logs` rows
are created.

Indexes: `idx_fuel_sheets_hash` on `file_hash`; `idx_fuel_sheets_status` on
`(processing_status, date_uploaded)`.

### 4.12 `fuel_logs`

| Field | Type | Notes |
| --- | --- | --- |
| `site` | relation → `sites`, **required** | Resolved per-row at import time |
| `vehicle` | relation → `vehicles` | Required by the schema but the importer tolerates a missing relation and falls back to QR |
| `date_logged` | date | ISO datetime; importer anchors at UTC midnight |
| `litres` | number | Required |
| `odo_reading` | number | Required — used for variance calculation |
| `latest_consumption` | number | Optional. Empty cells preserved as `undefined` rather than 0 (intentional) |
| `fuel_sheet` | relation → `fuel_sheets`, optional | Which uploaded sheet produced this row, if any |

Indexes: `idx_fuel_logs_site_date` on `(site, date_logged)`;
`idx_fuel_logs_vehicle_date` on `(vehicle, date_logged)`.

### 4.13 `plant_hours_sheets` (uploaded document)

Identical shape to `fuel_sheets` (§4.11), same optional-site reasoning.

Indexes: `idx_plant_hours_sheets_hash` on `file_hash`;
`idx_plant_hours_sheets_status` on `(processing_status, date_uploaded)`.

### 4.14 `plant_hours_log`

The most disciplined collection. Every site-attribution field is required.

| Field | Type | Notes |
| --- | --- | --- |
| `date_logged`, `start_date`, `end_date` | date | |
| `vehicle_qr` | text | The QR text — part of the **composite unique index** below |
| `vehicle` | relation → `vehicles` | Required (minSelect=1) |
| `site` | relation → `sites` | Required (minSelect=1) |
| `start_hours`, `end_hours`, `hours` | number | |
| `hour_meter_budget`, `cost_per_hour`, `value` | number | `value = hours × cost_per_hour` |
| `over_budget`, `hours_mismatch` | bool | Computed at import time |
| `vehicle_unit_type_at_import` | select | Snapshot of the vehicle's `unit_type` at the moment of import — see the note in [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) about why this discipline should extend to `site` |
| `plant_hours_sheet` | relation → `plant_hours_sheets`, optional | Which uploaded sheet produced this row, if any |

**Unique index: `(vehicle_qr, start_date)`.** Prevents accidental
double-imports of the same week for the same vehicle. Surfaced as a friendly
`"duplicate: vehicle_qr=X start_date=Y"` error in the import response.
Additional index: `idx_ph_log_vehicle_start` on `(vehicle, start_date)`.

### 4.15 `delivery_notes`

| Field | Type | Notes |
| --- | --- | --- |
| `processing_status` | select | `pending` → `ocr-complete` → `auto-verified` / `needs-review` / `manual-verified` / `failed` |
| `delivery_note_number` | text | Optional; OCR may fill it in |
| `site` | relation → `sites`, **required** | Resolved at upload time |
| `purchase_order` | relation → `purchase_orders` | Filled by OCR (`exact → normalised → Levenshtein`) or by manual `assign-po` route |
| `date_received` | date | |
| `received_by` | text | The uploader's identity — currently always `"web-upload"` |
| `slip_image` | file | The photo/PDF, served via `/api/delivery-notes/:id/slip` to n8n |
| `raw_ocr_data` | json | Full OCR payload + match metadata |
| `is_verified` | bool | Final flip from `auto-verified` or `manual-verified` |
| `ocr_confidence` | number | 0..100 integer |
| `ocr_completed_at` | date | |
| `received_by_name` | text | Signature OCR'd from the slip |
| `file_hash` | text | SHA-256, used for upload-time dedup |

Indexes: `idx_dn_site_date` on `(site, date_received)`; `idx_dn_status` on
`processing_status`.

### 4.16 `delivery_note_items` (renamed from `received_items`)

Joins DNs to PO line items. `cascadeDelete: true` on the parent DN.

| Field | Type | Notes |
| --- | --- | --- |
| `delivery_note` | relation → `delivery_notes` | Required parent |
| `purchase_order_item` | relation → `purchase_order_items` | Optional — can be empty for items the OCR couldn't match, or for orphans where the PO item was deleted between fetch and create |
| `qty_received` | number | |
| `condition_notes` | text | Free-text audit trail (e.g. `"OCR 92% · desc: Cement 50kg · unit: bag"`) |

---

## 5. Indexes and constraints summary

| Index | Where | Purpose |
| --- | --- | --- |
| `idx_sites_project_code` | `sites(project_code)` UNIQUE | Prevents duplicate site codes; enforced on the POST /api/sites route |
| `idx_po_number` | `purchase_orders(purchase_order_number)` UNIQUE | Prevents duplicate PO numbers — final backstop behind the app-level dedup in `process-batch` |
| `idx_po_site_status` | `purchase_orders(site, processing_status)` | Per-site pending-PO counts and the n8n cron's fetch-pending query |
| `idx_plant_hours_log_qr_start` | `plant_hours_log(vehicle_qr, start_date)` UNIQUE | Prevents accidental double-imports of the same week for the same vehicle |
| `idx_ph_log_vehicle_start` | `plant_hours_log(vehicle, start_date)` | Per-vehicle history queries |
| `idx_fuel_prices_effective_from` | `fuel_prices(effective_from)` UNIQUE | One price per effective date |
| `idx_dn_site_date` | `delivery_notes(site, date_received)` | Per-site DN listing |
| `idx_dn_status` | `delivery_notes(processing_status)` | Pending-verification dashboards |
| `idx_fuel_logs_site_date`, `idx_fuel_logs_vehicle_date` | `fuel_logs` | Per-site / per-vehicle fuel history |
| `idx_ts_entries_site_date`, `idx_ts_entries_emp_date` | `timesheet_entries` | Per-site / per-employee timesheet queries |
| `idx_timesheets_hash`, `idx_timesheets_status` | `timesheets` (document) | Upload dedup + pending-import dashboards |
| `idx_fuel_sheets_hash`, `idx_fuel_sheets_status` | `fuel_sheets` | Same, for fuel sheet uploads |
| `idx_plant_hours_sheets_hash`, `idx_plant_hours_sheets_status` | `plant_hours_sheets` | Same, for plant hours sheet uploads |

No collection outside the above has a server-enforced unique constraint. In
particular, `vehicles.qr_code_id` is **not** unique — see
[multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites).

---

## 6. PocketBase API rules

`scripts/pb-relax-rules.py` writes the ruleset used in production. For every
collection:

- `listRule`, `viewRule`, `createRule` → `@request.auth.id != ''` (any
  authenticated user)
- `updateRule`, `deleteRule` → `null` (superuser only)

The collection list is the full set of 16 — there is no collection where
clients can update or delete records through the API. The
`/api/records/[collection]/[id]/verify` route hand-picks two
collections (`purchase_orders`, `delivery_notes`) for the `is_verified`
toggle, but it runs server-side as `pbAdmin()`, so it is not constrained by
these rules.

The only routes that respect the API rules at all are the (currently
unused) browser-side `pb` client and the n8n file-proxy route
`/api/delivery-notes/:id/slip`, which uses the admin token in its
`Authorization` header. The PB layer never blocks the Next.js server.

---

## 7. Schema drift

`pb-bootstrap-schema.py` is the canonical schema definition — a fresh
PocketBase must come up in the exact shape production is in. That script
had drifted ten-plus migrations behind production before the 2026-09-11
restructure's final task (Task 20) brought it back to truth: it was missing
`fuel_prices` and `income_claims` entirely, still described the deleted
`data_import_batches` and `plant` collections, still carried `project_code`
on eight collections that no longer have it, still had `qty_outstanding` on
`purchase_order_items`, and was missing every index added across the
restructure.

That drift is exactly the failure mode this section exists to name, so the
rule going forward is unchanged from before: **any time the schema changes,
the new field must be added to both `pb-bootstrap-schema.py`** (so a fresh
PB starts in the right shape) **and `src/lib/pb-types.ts`** (so the
TypeScript code agrees), in the same change that adds it to production.
Forgetting either side is a known foot-gun — it is what produced this
section's rewrite in the first place. `scripts/pb-relax-rules.py`'s
collection list is the third thing that must move in lockstep; it drifted
just as far (missing `fuel_prices`, `income_claims` and `plant_hours_log`
entirely, on top of the same renames/deletions).

One-off data-migration scripts under `scripts/migrations/*` are expected to
exist and are not a drift signal by themselves — they are how the schema
*got* to its current shape. The drift signal is `pb-bootstrap-schema.py` (or
`pb-relax-rules.py`) disagreeing with what a migration already did to
production.