MEGAINFRA-WEBAPP/docs/data-model.md

# Data model

> Every PocketBase collection that backs the app, with fields, relations, and
> the site-attribution strategy each one uses.

Source of truth for types: `src/lib/pb-types.ts` (13 collection interfaces).
Source of truth for the live schema: `scripts/pb-bootstrap-schema.py`.
Production instance: `https://pb.mega.infra.apphub.co.za`.

---

## 1. Entity-relationship diagram

```mermaid
erDiagram
    SITES ||--o{ EMPLOYEES : "site (relation)"
    SITES ||--o{ VEHICLES : "site (relation)"
    SITES ||--o{ FIXED_COSTS : "site (relation)"
    SITES ||--o{ INCOME_CLAIMS : "site (relation)"
    SITES ||--o{ PURCHASE_ORDERS : "site (relation)"
    SITES ||--o{ DELIVERY_NOTES : "site (relation)"
    SITES ||--o{ FUEL_LOGS : "site (relation)"
    SITES ||--o{ PLANT_HOURS_LOG : "site (relation)"

    EMPLOYEES ||--o{ TIMESHEETS : "employee (relation)"
    VEHICLES ||--o{ FUEL_LOGS : "vehicle (relation)"
    VEHICLES ||--o{ PLANT_HOURS_LOG : "vehicle (relation)"
    DATA_IMPORT_BATCHES ||--o{ FUEL_LOGS : "batch_reference (relation)"
    DATA_IMPORT_BATCHES ||--o{ TIMESHEETS : "batch_reference (relation)"
    DATA_IMPORT_BATCHES ||--o{ PLANT_HOURS_LOG : "batch_reference (loose)"

    PURCHASE_ORDERS ||--|{ PURCHASE_ORDER_ITEMS : "purchase_order (cascade)"
    PURCHASE_ORDERS ||--o{ DELIVERY_NOTES : "purchase_order"
    DELIVERY_NOTES ||--|{ RECEIVED_ITEMS : "delivery_note (cascade)"
    PURCHASE_ORDER_ITEMS ||--o{ RECEIVED_ITEMS : "purchase_order_item"
```

The PLANT collection is intentionally absent from the diagram — it is a
deprecated legacy alias for the unified VEHICLES collection
(`src/lib/pb-types.ts:58-60`).

---

## 2. Site attribution strategy, per collection

This is the table to read first. The question "where does this row belong?"
has three different answers depending on which collection you're in.

| Collection | `site` relation | `project_code` text | Attribution source | Drift risk |
| --- | --- | --- | --- | --- |
| `sites` | n/a | required, **unique** | The site row itself | None — the unique index is authoritative |
| `employees` | ✅ | ✅ | Both, but `site` is canonical for metrics; `project_code` is denormalised at create time | Medium — changing a worker's site later leaves `project_code` stale |
| `vehicles` | ✅ | ✅ | Both, but `site` is the relation; `project_code` is denormalised at create time | High — fuel logs are filtered by `log.site`, not by the current vehicle's site |
| `fixed_costs` | ✅ | ✅ | `site` is the relation; `project_code` is denormalised | Low — the import routes always stamp both from the same source |
| `income_claims` | ✅ | ✅ | Same as fixed_costs | Low |
| `purchase_orders` | ✅ | ✅ | Same as fixed_costs | Low |
| `delivery_notes` | ✅ | ✅ | Same as fixed_costs | Low |
| `fuel_logs` | ✅ | ✅ | `site` is filled when the source row has a site; `project_code` is the canonical fallback when the site is missing | Medium — see [multi-site-assessment.md §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) |
| `plant_hours_log` | ✅ **required** | ✅ **required** | Always set from the vehicle at import time | Lower than fuel_logs because required |
| `timesheets` | ❌ | ❌ | **Joined via `employees.site`** at read time — `timesheets` has no site or project_code of its own | High — site attribution is a per-employee property that can move |
| `purchase_order_items` | ❌ | ❌ | Inherited through `purchase_order → site` | None — never queried directly by site |
| `received_items` | ❌ | ❌ | Inherited through `delivery_note → site` | None — same |
| `data_import_batches` | ❌ | ❌ | Inferred from `processed_record_id` + the processed record's collection | n/a |

**Reading the table.** Every collection except the two joining-tables and
`data_import_batches` carries both a `site` relation and a `project_code`
string. The only collection that carries **neither** is `timesheets` — and
that's the collection whose site attribution drifts the most, because it's
synthesised at read time from the employee record.

---

## 3. The `project_code` denormalisation

`project_code` is a free-text identifier like `MEG-001` or `HILLS-2026` that
uniquely names a contract. It is the human-readable label that appears on
every report and filter dropdown.

The schema carries it on every child collection as a *denormalised copy* of
the parent site's `project_code`. It is **not** a foreign key — it is just a
string, written at insert time, never updated when the parent site changes.

**Why it exists.** The chat agent and several older reports filter by
`project_code` (text) rather than expanding the `site` relation
(`expand: "site"` would force a join the dashboard pages don't want). It is
also what the import routes use as the canonical key when a row's `site`
relation is empty.

**The drift problem.** If you change a site's `project_code` in PocketBase,
every existing `fuel_logs`, `fixed_costs`, `employees`, `vehicles`,
`purchase_orders`, `delivery_notes`, and `income_claims` row keeps the
*old* string forever. Code that joins through the `site` relation sees the
new value; code that filters on `project_code` text sees the old.

`scripts/pb-relax-rules.py:24` and the `PATCH /api/sites/[id]` route
(`src/app/api/sites/[id]/route.ts:23-28`) deliberately **refuse to let you
change `project_code` on an existing site** — that is exactly the rule
designed to prevent drift. The `POST /api/sites` route also enforces a 64-char
ceiling and uniqueness via the `idx_sites_project_code` index
(`scripts/pb-bootstrap-schema.py:84`).

What this means in practice: the denormalisation is **stable** as long as no
one re-creates a site with the same name and a different code. The biggest
remaining risk is bulk imports that resolve site by name and stamp the
wrong `project_code` onto historical rows — see [multi-site-assessment.md
§4](./multi-site-assessment.md#4-site-resolution-by-name-string).

---

## 4. Collection reference

Each entry below lists every typed field, the relation direction, and any
notable constraints. Optional/legacy fields are marked.

### 4.1 `sites`

The site is the unit of business. One row per construction contract.

| Field | Type | Required | Notes |
| --- | --- | --- | --- |
| `project_code` | text | ✅ | Unique (`idx_sites_project_code`). Locked after creation. |
| `name` | text | optional | Human label, e.g. "The Hills" |
| `duration` | text | optional | Free-text duration ("8 months", "52 weeks") — parsed by `parseDurationMonths` |
| `contract_value` | number | optional | Rand value, no currency field |
| `work_hours` | number | optional | Total project work hours |
| `hours_per_week` | number | optional | Standard workweek for the site |
| `wages_hours_per_day` | number | optional | Standard wage day length |
| `job_start_date` | date | optional | ISO date; the projection helper treats this as the authoritative start when present |

### 4.2 `employees`

| Field | Type | Notes |
| --- | --- | --- |
| `project_code` | text | Denormalised from `site.project_code` at create time |
| `site` | relation → `sites` | The site the worker is currently assigned to |
| `name`, `employee_no`, `position` | text | `employee_no` is the cross-system payroll id; **uppercased** on import (`src/app/api/timesheets/import/route.ts:289`) |
| `rate_per_hour` | number | Drives payroll burn |
| `hours_per_day_standard` | number | 9 by default for auto-created casuals |
| `worker_type` | select | `permanent` or `casual` |

### 4.3 `vehicles` (unified — includes plant)

| Field | Type | Notes |
| --- | --- | --- |
| `project_code` | text | Denormalised |
| `site` | relation → `sites` | Current assignment |
| `description` | text | e.g. "Toyota Hilux 2.4 GD-6" |
| `standard_work_hours` | text/number | Free-text on schema; numeric in practice. Used by the plant utilisation calc |
| `fuel_limits` | text | Free-form like `"8.5 L/h"`, `"6.5 km/L"` — parsed by `parseFuelLimit` (`src/lib/fuel-variance.ts`) |
| `qr_code_id` | text | The clean identifier the fuel importer matches against. NOT unique in schema — see [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) |
| `cost_per_hour` | number | Drives plant utilisation value |
| `unit_type` | select | `fuel` or `plant` (default `fuel`). Splits the bucket between Fleet and Plant pages |
| `attention_dismissed_at` | date | When the user dismissed the vehicle's "over fuel limit" alert from the fuel page |

The legacy `plant` collection is a deprecated alias kept around for old
importers. New plant items go in `vehicles` with `unit_type="plant"`.

### 4.4 `fixed_costs`

Monthly or one-off costs. `is_recurring` is the partition key.

| Field | Type | Notes |
| --- | --- | --- |
| `project_code` | text | Denormalised |
| `site` | relation → `sites` | |
| `date_logged` | date | Drives the monthly tile and trend charts |
| `description`, `notes` | text | |
| `cost` | number | Rand value |
| `is_recurring` | bool | `true` (or missing) → monthly bucket; `false` → one-off bucket. Partitioned by `partitionFixedCosts` (`src/lib/partition-fixed-costs.ts`) |

### 4.5 `income_claims`

| Field | Type | Notes |
| --- | --- | --- |
| `claim_number` | text | The contractor's IPN / claim ref |
| `project_code` | text | Denormalised |
| `site` | relation → `sites` | |
| `period_start`, `period_end` | date | The period the claim covers |
| `gross_amount` | number | Rand |
| `certified_by` | text | Signature line on the cert |
| `paid_on` | date | Empty until the claim is actually paid — drives the income series |
| `status` | select | `draft` / `certified` / `paid` |
| `is_verified` | bool | Verified by a human |
| `notes` | text | |

### 4.6 `purchase_orders`

| Field | Type | Notes |
| --- | --- | --- |
| `purchase_order_number` | text | Unique (`idx_po_number`). Normalised to uppercase with "PO" prefix stripped before insert (`src/lib/purchase-order-schema.ts`) |
| `project_code` | text | Denormalised |
| `site` | relation → `sites` | Resolved by `matchSite` at ingest time |
| `supplier` | text | |
| `required_by_date` | date | When the materials are needed |
| `ordered_by` | text | Who placed it |
| `items_count`, `total_value` | number | Denormalised roll-ups for table views |
| `purchase_order_file` | file | The original PDF/image; mime allowlist enforced by the upload route |
| `raw_ocr_data` | json | Full OCR payload + ingest metadata |
| `is_verified` | bool | Human-verified flag |

### 4.7 `purchase_order_items`

Joins POs to line items. `cascadeDelete: true` on the parent relation
(`scripts/pb-bootstrap-schema.py:132`).

| Field | Type | Notes |
| --- | --- | --- |
| `purchase_order` | relation → `purchase_orders` | Required parent |
| `material_code`, `cost_code` | text | |
| `description` | text | The free-text the OCR matches against in the DN line |
| `qty` | number | Ordered quantity |
| `qty_outstanding` | number | Decreased as received_items roll up — but the canonical fulfilment % is computed from `received_items` directly (see `/dashboard/purchasing` page and the orphan-id note in `src/app/dashboard/purchasing/page.tsx:52-72`) |
| `unit`, `rate`, `disc_percent`, `vat_percent`, `amount` | | The amount field is the **computed** total, not stored separately |

### 4.8 `timesheets`

The collection with **no site** and **no project_code**. Site is resolved
through `employeeRates` at every read site.

| Field | Type | Notes |
| --- | --- | --- |
| `employee` | relation → `employees` | |
| `batch_reference` | relation → `data_import_batches` | |
| `date_logged` | date | |
| `hours_worked` | number | Decimal |

### 4.9 `fuel_logs`

| Field | Type | Notes |
| --- | --- | --- |
| `project_code` | text | Denormalised, falls back to `vehicle.project_code` when the source row has no site |
| `site` | relation → `sites` | Optional — empty when the source row had no site column |
| `vehicle` | relation → `vehicles` | Required by the schema but the importer tolerates a missing relation and falls back to QR |
| `batch_reference` | relation → `data_import_batches` | |
| `date_logged` | date | ISO datetime; importer anchors at UTC midnight |
| `litres` | number | Required |
| `odo_reading` | number | Required — used for variance calculation |
| `latest_consumption` | number | Optional. Empty cells preserved as `undefined` rather than 0 (intentional, see comment at `src/lib/pb-types.ts:173-181`) |

### 4.10 `plant_hours_log`

The most disciplined collection. Every site-attribution field is required.

| Field | Type | Notes |
| --- | --- | --- |
| `date_logged`, `start_date`, `end_date` | date | |
| `vehicle_qr` | text | The QR text — part of the **composite unique index** below |
| `vehicle` | relation → `vehicles` | Required (minSelect=1) |
| `site` | relation → `sites` | Required (minSelect=1) |
| `project_code` | text | Required |
| `start_hours`, `end_hours`, `hours` | number | |
| `hour_meter_budget`, `cost_per_hour`, `value` | number | `value = hours × cost_per_hour` |
| `over_budget`, `hours_mismatch` | bool | Computed at import time |
| `vehicle_unit_type_at_import` | select | Snapshot of the vehicle's `unit_type` at the moment of import — see the note in [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites) about why this discipline should extend to `site` |
| `batch_reference` | relation | Optional |

**Unique index: `(vehicle_qr, start_date)`.** Prevents accidental
double-imports of the same week for the same vehicle
(`scripts/pb-bootstrap-schema.py:257`). Surfaced as a friendly
`"duplicate: vehicle_qr=X start_date=Y"` error in the import response.

### 4.11 `delivery_notes`

| Field | Type | Notes |
| --- | --- | --- |
| `delivery_note_number` | text | Optional; OCR may fill it in |
| `project_code` | text | Denormalised |
| `site` | relation → `sites` | Resolved at upload time |
| `purchase_order` | relation → `purchase_orders` | Filled by OCR (`exact → normalised → Levenshtein`) or by manual `assign-po` route |
| `date_received` | date | |
| `received_by` | text | The uploader's identity — currently always `"web-upload"` |
| `slip_image` | file | The photo/PDF, served via `/api/delivery-notes/:id/slip` to n8n |
| `raw_ocr_data` | json | Full OCR payload + match metadata |
| `processing_status` | select | `pending` → `ocr-complete` → `auto-verified` / `needs-review` / `manual-verified` / `failed` |
| `ocr_confidence` | number | 0..100 integer |
| `ocr_completed_at` | date | |
| `received_by_name` | text | Signature OCR'd from the slip |
| `is_verified` | bool | Final flip from `auto-verified` or `manual-verified` |

### 4.12 `received_items`

Joins DNs to PO line items. `cascadeDelete: true` on the parent DN.

| Field | Type | Notes |
| --- | --- | --- |
| `delivery_note` | relation → `delivery_notes` | Required parent |
| `purchase_order_item` | relation → `purchase_order_items` | Optional — can be empty for items the OCR couldn't match, or for orphans where the PO item was deleted between fetch and create |
| `qty_received` | number | |
| `condition_notes` | text | Free-text audit trail (e.g. `"OCR 92% · desc: Cement 50kg · unit: bag"`) |

### 4.13 `data_import_batches`

The audit-trail spine. Every uploaded file — Excel, CSV, PDF, image — has a
batch row that tracks its lifecycle.

| Field | Type | Notes |
| --- | --- | --- |
| `batch_type` | select | `timesheets` / `fuel_logs` / `purchase_orders` / `delivery_notes` / `plant_hours` |
| `content_type` | text | MIME of the original file |
| `original_file` | file | The uploaded bytes; mime-restricted per `batch_type` |
| `date_uploaded` | date | |
| `uploaded_by` | text | Currently always `"web-import"` or `"web-upload"` |
| `status` | select | `pending` / `processing` / `processed` / `failed` / `dead` |
| `attempts` | number | Incremented on each retry |
| `last_error` | text | Last failure reason |
| `last_attempt_at` | date | |
| `next_retry_at` | date | |
| `processed_at` | date | When the batch reached `processed` |
| `processed_record_id` | text | The id of the record this batch created (PO id, DN id, etc.) |
| `file_hash` | text | SHA-256 of the original file; the dedup window for PO re-uploads uses this index |

---

## 5. Indexes and constraints summary

| Index | Where | Purpose |
| --- | --- | --- |
| `idx_sites_project_code` | `sites(project_code)` UNIQUE | Prevents duplicate site codes; enforced on the POST /api/sites route via the `idx_sites_project_code` name match (`src/app/api/sites/route.ts:64`) |
| `idx_po_number` | `purchase_orders(purchase_order_number)` UNIQUE | Prevents duplicate PO numbers — works alongside the AI extraction dedup by `purchase_order_number` |
| `idx_plant_hours_log_qr_start` | `plant_hours_log(vehicle_qr, start_date)` UNIQUE | Prevents accidental double-imports of the same week for the same vehicle; surfaced as `duplicate:` reason in the importer |
| `idx_batches_file_hash` | `data_import_batches(file_hash, status)` | Used by the PO upload dedup window (`src/app/api/purchase-orders/upload/route.ts:80-84`) |

No collection outside the above has a server-enforced unique constraint. In
particular, `vehicles.qr_code_id` and `vehicles.project_code` are
**not** unique — see [multi-site-assessment §5](./multi-site-assessment.md#5-vehicles-plant-and-employees-moving-between-sites).

---

## 6. PocketBase API rules

`scripts/pb-relax-rules.py` writes the ruleset used in production. For every
collection:

- `listRule`, `viewRule`, `createRule` → `@request.auth.id != ''` (any
  authenticated user)
- `updateRule`, `deleteRule` → `null` (superuser only)

The collection list is the full set of 13 — there is no collection where
clients can update or delete records through the API. The
`/api/records/[collection]/[id]/verify` route hand-picks two
collections (`purchase_orders`, `delivery_notes`) for the `is_verified`
toggle, but it runs server-side as `pbAdmin()`, so it is not constrained by
these rules.

The only routes that respect the API rules at all are the (currently
unused) browser-side `pb` client and the n8n file-proxy route
`/api/delivery-notes/:id/slip`, which uses the admin token in its
`Authorization` header. The PB layer never blocks the Next.js server.

---

## 7. Schema drift

The `pb-bootstrap-schema.py` script is the canonical schema definition. The
following migrations exist on disk and are not represented in the script:

| Migration | Purpose |
| --- | --- |
| `migrate-delivery-notes-ocr.py` | Adds the `processing_status` / `ocr_*` fields and the OCR webhook hooks to existing DNs |
| `migrate-plant-hours-backfill-vehicle.py` | Backfills `plant_hours_log.vehicle` relations from `vehicle_qr` for pre-relation rows |
| `scripts/migrations/*` (other) | One-off patches for specific schema or data issues |

Any time the schema changes, the new field must be added to **both**
`pb-bootstrap-schema.py` (so a fresh PB starts in the right shape) **and**
`src/lib/pb-types.ts` (so the TypeScript code agrees). Forgetting either
side is a known foot-gun.