MEGAINFRA-WEBAPP/docs/operations.md

# Operations

> Environment variables, scripts, tests, feature flags, and the
> PocketBase API rules in effect in production.

This is the "how do I run it, how do I ship it, where do things live"
document. It deliberately does not document the application logic — see
[`architecture.md`](./architecture.md), [`data-model.md`](./data-model.md),
[`data-flows.md`](./data-flows.md), and [`modules.md`](./modules.md).

---

## 1. Environment variables

The full set lives in `.env.local.example`. The Coolify-friendly subset
(plus PocketBase host env) lives in `docs/cooler-env.template`. They
should agree at all times.

| Variable | Client-exposed? | Required? | Default | Purpose |
| --- | --- | --- | --- | --- |
| `AUTH_MODE` | no | no | `pin` | `users` enables PocketBase logins; `pin` retains the staged rollout fallback. |
| `NEXT_PUBLIC_APP_PIN` | yes | yes in PIN mode | `2468` | PIN-mode login. Not used when `AUTH_MODE=users`. |
| `APP_PIN` | no | yes in PIN mode | unset | Server-side PIN for clock corrections in PIN mode. Named-login mode uses the signed-in admin instead. |
| `RECEIPT_CORRECTION_SECRET` | no | yes (receipt edits) | unset | Server-only code required by `POST /api/dn-batches/[id]/notes/[noteId]/correct-receipt` to edit a posted receipt. At least 12 chars; must not reuse the PIN. `openssl rand -hex 16`. |
| `MEGA_CUSTOMER_VAT_NO` | no | yes (DN pipeline) | unset | MEGA's VAT number. The DN pipeline checks the customer identity block printed on each slip against it; a mismatch marks the note `needs_attention`. |
| `NEXT_PUBLIC_APP_NAME` | yes | no | `MEGA INFRASTRUCTURE` | Brand on the lock screen. |
| `NEXT_PUBLIC_DEFAULT_THEME` | yes | no | `light` | `light` or `dark` — pre-hydration fallback. |
| `NEXT_PUBLIC_POCKETBASE_URL` | yes | yes (PB-dependent) | `http://127.0.0.1:8090` | Browser-reachable PB URL. **Also used by `pbAdmin()` server-side.** The runtime joins file URLs off this same base. |
| `NEXT_PUBLIC_API_BASE_URL` | yes | no | unset | Legacy. Documented but unused in current code. |
| `POCKETBASE_ADMIN_EMAIL` | no | yes (server) | unset | Superuser email used by `pbAdmin()`. |
| `POCKETBASE_ADMIN_PASSWORD` | no | yes (server) | unset | Superuser password. |
| `POCKETBASE_ADMIN_TOKEN` | no | optional | unset | If set, used in place of email + password — saves a re-auth on every cold start. Generate via PocketBase admin UI → "Generate token". |
| `N8N_PO_INGEST_URL` | no | yes (PO upload) | unset | Webhook URL the Next.js server forwards PO uploads to. Server-only. |
| `N8N_INGEST_SECRET` | no | yes (n8n callbacks) | unset | Shared secret n8n sends in `x-ingest-secret` on the ingest/process-batch/ocr-result/slip routes. Generate with `openssl rand -hex 32`. |
| `N8N_MCP_URL` | no | optional | unset | Public URL of the n8n MCP server (for the local Codex MCP config). |
| `ENABLE_CHAT` | no | **NO** (production) | unset | Master switch for `/api/chat` and `/dashboard/chat`. The `.env.local.example` warning is explicit: must stay unset in production until real authentication exists. |
| `CHAT_PROVIDER` | no | no | `anthropic` | `anthropic` or `minimax` (MiniMax OpenAI-compatible endpoint). |
| `ANTHROPIC_API_KEY` | no | conditional | unset | Anthropic API key. Leave unset to use an `ant auth login` profile instead. |
| `ANTHROPIC_MODEL` | no | no | `claude-opus-5` | Overrides the Claude model id `provider-anthropic.ts` requests, without a code change or redeploy. |
| `MINIMAX_API_KEY` | no | conditional | unset | MiniMax Open Platform API key (NOT a Token Plan subscription key). |
| `MINIMAX_BASE_URL` | no | no | `https://api.minimax.io/v1` | MiniMax OpenAI-compatible base URL. |
| `MINIMAX_MODEL` | no | no | `MiniMax-M3` | The only model in the M-series that supports BOTH vision AND tool calling (verified 2026-08-30). M2 / M2.7 accept tools but reject images. |
| `NODE_ENV` | no | yes (prod) | `production` | Standard. |
| `PORT` | no | no | `3000` | Next.js standalone server port. |
| `HOSTNAME` | no | no | `0.0.0.0` | Next.js standalone server bind. |
| `NEXT_TELEMETRY_DISABLED` | no | yes (build) | unset | Set to `1` in the Dockerfile. |

**Anything prefixed `NEXT_PUBLIC_` is inlined into the client bundle at
build time.** Changing `NEXT_PUBLIC_APP_PIN` requires a rebuild —
Coolify will not pick up the change otherwise. (Source:
`COOLIFY.md:55-58`.)

### Named logins and site access

`AUTH_MODE=users` uses the PocketBase `users` collection. An app admin sees all
sites and can add users at `/dashboard/settings/users`. New accounts default to
the editable initial password `mega2026` and must choose a new password on
first sign-in. Site managers use the operational dashboard for assigned sites
and can create clock accounts for those sites. Clock users work only in the
clock PWA: worker and fleet scans, document uploads, and upload status/retry for
assigned sites. Dashboard pages, timesheet imports, financial edits, and manual
attendance corrections are unavailable to clock users.

Office users have selected whole dashboard sections: Overview, Sites,
Workforce, Plant, and Financial. They have full use of the existing tools in
each selected section. Office site scope is either assigned sites or an
explicit All sites option, which includes future sites. Ask and user
administration are unavailable to office users. See [user access](setup/user-access.md)
for account setup and the additive Office user schema migration.
An admin can reset a user's password
in the app or PocketBase and set `must_change_password` for the next login.

The migration is dry-run by default:

```bash
python3 scripts/migrations/2026-09-28-users-login.py --env-file .env.local
```

Before applying, back up PocketBase and inspect the target host and existing
`dashboard@mega.co.za` account. The apply command requires `--apply`, the exact
`--confirm-host` shown by the dry run, `--bootstrap-user` with the selected
existing user's email, and either `--backup-confirmed` pointing to the backup
file or `--backup-key` naming a nonempty backup in PocketBase's backup service.
It adds the four user fields, makes that account the first app admin, closes
direct collection API rules, and protects data file fields. It does **not**
change `AUTH_MODE` or the bootstrap account's password.

Deploy the updated app code before applying the migration, because the old
code loads some files by public URL. Keep `AUTH_MODE=pin` during migration.
Once the bootstrap account can sign in, switch to `AUTH_MODE=users` and restart
the app. Check an admin, a one-site viewer, and a clock-only account on site.
To return to the PIN flow, set `AUTH_MODE=pin` and restart this app version.
Older app versions require the saved PocketBase backup because files are now
protected. Rotate the PocketBase superuser credential during rollout; it was
previously recorded in tracked deployment logs.

### 1.1 Comparing chat providers/models

`CHAT_PROVIDER=anthropic` + `ANTHROPIC_API_KEY` switches Ask to Claude,
which is more reliable than MiniMax at long multi-step tool use (reading
a sheet → matching names → proposing 40 creates in one action). To
compare providers or Claude model ids, run the same 5 test prompts
against each and compare tool-call count, correctness, and latency —
a suggested set:

1. A single-tool factual question ("what's the burn % on The Hills?").
2. A free-text report ("show me litres per vehicle per week on The
   Hills since 1 August as a chart").
3. A multi-step data-change ("add a fixed cost of R500 on Cedar Gate
   dated today", through to Confirm).
4. A file-driven task ("here are 5 names for The Hills" + a small
   spreadsheet, through matching and proposing).
5. An ambiguous request that should trigger a clarifying question
   rather than a guess ("what's the fuel spend?" with more than one
   site named similarly).

Switch models without a redeploy via `ANTHROPIC_MODEL` (e.g.
`ANTHROPIC_MODEL=claude-sonnet-5`) — see the environment table above.

---

## 2. Scripts in `scripts/`

### 2.1 PocketBase schema and rules

| Script | Purpose | Idempotent? |
| --- | --- | --- |
| `pb-bootstrap-schema.py` | Canonical schema definition for the 13 collections. Run after any schema change. | No — running twice may fail because the collections already exist. |
| `pb-relax-rules.py` | Writes the production API ruleset to every collection (`list/view/create` = any authed, `update/delete` = superuser). | Yes — re-running just rewrites the same rules. |
| `migrate-delivery-notes-ocr.py` | One-off migration that adds the OCR `processing_status` / `ocr_*` fields and seeds status on existing DNs. | No — running twice is a no-op on most steps. |
| `migrate-plant-hours-backfill-vehicle.py` | Backfills `plant_hours_log.vehicle` relations from `vehicle_qr` for pre-relation rows. | Idempotent. |

### 2.2 Build-time

| Script | Trigger | Output |
| --- | --- | --- |
| `build-icons.mjs` | `npm run build` (pre-step) | Regenerates the PWA icon set under `public/icons/` |
| `build-og-image.mjs` | `npm run build` (pre-step) | Regenerates `public/og-image.png` for OG previews |
| `patch-next-dev-blocker.cjs` | `npm install` (`postinstall` hook) | Patches Next 16's dev-only cross-site guard so `next dev` accepts any host |
| `generate-plant-hours-template.ts` | Manual | Used by `scripts/build-*.mjs` for the plant-hours template |

The `postinstall` patch is **only loaded by `next dev`** — never by
production. The Dockerfile runs `npm ci --ignore-scripts` in the `deps`
stage to skip the hook entirely (`Dockerfile:32`).

### 2.3 Migrations directory

`scripts/migrations/` holds one-off patches that the schema script
doesn't cover. Examples (without full enumeration — there are several):

- Aug 2026: `latest_consumption` field added to `fuel_logs`.
- Sep 2026: `file_hash` column added to `data_import_batches` for PO
  dedup.

Always add a new entry here when running a one-off schema change. The
canonical pattern is one file per concern, named
`migrate-<collection>-<concern>.py`.

---

## 3. PocketBase API rules

`scripts/pb-relax-rules.py` writes:

- `listRule`, `viewRule`, `createRule` = `@request.auth.id != ''` (any
  authenticated user)
- `updateRule`, `deleteRule` = `null` (superuser only)

Applied to all 13 collections:

```
DATA_IMPORT_BATCHES, SITES, EMPLOYEES, PURCHASE_ORDERS,
PURCHASE_ORDER_ITEMS, FIXED_COSTS, PLANT, VEHICLES, TIMESHEETS,
FUEL_LOGS, DELIVERY_NOTES, RECEIVED_ITEMS
```

What this means in practice:

- The **only client that talks to PB in production is `pbAdmin()`** —
  and it's the superuser, so the rules don't constrain it.
- The browser-side `pb` client (currently unused) would be constrained by
  the rules if it were active.
- The `/api/delivery-notes/[id]/slip` proxy uses the admin token in its
  `Authorization` header — also not constrained by these rules.

**Verifying in production.** Run the script again and compare the output.
If the rules match, every line should print `✓ ... updated`. If anything
prints `FAILED`, investigate before trusting the API.

---

## 4. Running the test suite

```bash
npm run typecheck   # tsc --noEmit
npm run test:run    # vitest run
npm run lint        # eslint . (flat config in eslint.config.mjs)
```

### 4.1 Current state (as of 2026-09-25)

```
Typecheck:  passes cleanly
Tests:      1397 passed (122 files)
Build:      next build --webpack passes
Lint:       131 problems (53 errors, 78 warnings) — not yet a gate
```

`next lint` was removed in Next 16, so `npm run lint` now calls the ESLint
CLI directly. The remaining errors are mostly the React Compiler rules
that `eslint-config-next` 16 turned on (`react-hooks/set-state-in-effect`,
`react-hooks/refs`, `react-hooks/purity`) plus `no-explicit-any`. They
are not crashes, but should be worked down before lint is added to CI.

### 4.2 Test inventory

The `src/lib/metrics/*.test.ts` files are the most important — they pin
every computation the dashboard exposes. Anything that touches a number
should be cross-checked against these.

The `src/lib/chat/*.test.ts` files pin the tool-schemas, the
think-filter, and the chat loop's tool-call limits.

The `src/app/api/**/*.test.ts` files mock `pbAdmin` and exercise the
business logic of each route (file dedup, batch state machine, OCR
tier decision, etc.).

---

## 5. Running the app

### 5.1 Local dev

```bash
cp .env.local.example .env.local
# edit NEXT_PUBLIC_POCKETBASE_URL, POCKETBASE_ADMIN_*, N8N_*
npm install
npm run dev    # next dev -H 0.0.0.0 -p 6004 --webpack
```

The dev server runs on **port 6004** (not 3000 — the user runs other
apps on 3000; do not change this back to 3000). The `--webpack` flag
is required because `@ducanh2912/next-pwa` is webpack-only.

Enter the PIN (`NEXT_PUBLIC_APP_PIN`, default `2468`) on `/login`. The
old `?demo=1` bypass was removed on 2026-09-26.

### 5.2 Production build

```bash
npm run build   # icons + og-image + next build --webpack
npm run start   # next start -H 0.0.0.0 -p 3000
```

### 5.3 Docker / Coolify

The `Dockerfile` and `docker-compose.yml` are Coolify v4-ready. See
`COOLIFY.md` for the full setup walkthrough. Key points:

- The standalone server in the final image is non-root, listens on
  `${PORT:-3000}` on `0.0.0.0`.
- The Docker healthcheck is `GET /api/health` every 30s with a 40s
  start period (`Dockerfile:86-87`).
- Traefik routes traffic to the container on port 3000 — there is no
  internal port mapping in the docker-compose.
- The `.dockerignore` excludes pre-built service-worker files so the
  build always ships the right hashes for the new release.

### 5.4 MCP server

```bash
npm run mcp     # tsx --env-file=.env.local src/mcp/server.ts
```

Runs over stdio using the PocketBase credentials in `.env.local`. Opens
no network listener. Exposes the same `tools` array the chat uses, so
an external agent (Codex, etc.) gets identical answers to the in-app
chat.

### 5.5 Deploy logs

The repo tracks one historical deploy log
(`deployment-vptid1uo0jnz2q4t1mcbqjj9-all-logs-2026-08-19-00-23-06.txt`)
as evidence of how the Coolify deploy surface looks when something
breaks. This is left in-tree for context, not as a reference.

---

## 6. Feature flags

| Flag | Default | Effect when on | Effect when off |
| --- | --- | --- | --- |
| `ENABLE_CHAT` | unset (= `false`) | `/api/chat` and `/dashboard/chat` work | Both return 404 / `notFound()` |
| `NEXT_PUBLIC_DEFAULT_THEME` | `light` | Initial theme before user preference loads | " |
| `CHAT_PROVIDER` | `anthropic` | `anthropic` if `ANTHROPIC_API_KEY` is set (or `ant auth login` profile); `minimax` / `openai` if `MINIMAX_API_KEY` is set |

There is no other feature-flag mechanism. Anything else that needed to
be toggleable in the past has been either hardcoded (variance tolerances,
`RATE_PER_LITRE`) or set via env (`RATE_PER_LITRE` is in
`src/lib/metrics/constants.ts` — **not** env-driven; change there).

---

## 7. Smoke tests after a deploy

The two routes that should always return 200 in production:

| Probe | URL | Expects |
| --- | --- | --- |
| Liveness | `GET /api/health` | `200` with `{ status: "ok", uptime, timestamp }` |
| Auth gate | `GET /login` | `200` HTML (the lock screen) |

`curl -sS -o /dev/null -w "%{http_code}\n" https://<host>/api/health`
should print `200`. If it doesn't, the container isn't ready (start
period is 40s).

## 8. Staged delivery-note ingestion

The staged pipeline is deliberately off until a representative site has been
observed. It is additive: original source files and the legacy runner remain
available, and financial rows are created only by the posting action.

### 8.1 Migration

Preview the schema first. This command never writes and does not require admin
credentials:

```bash
set -a; source .env.local; set +a
python3 scripts/pb-add-dn-staged-ingestion.py --dry-run
```

Apply only after reviewing the preview and confirming the exact PocketBase host:

```bash
set -a; source .env.local; set +a
python3 scripts/pb-add-dn-staged-ingestion.py --apply \
  --confirm-host "$NEXT_PUBLIC_POCKETBASE_URL"
```

The script adds the staged collections, fields, and indexes. Do not remove
collections or edit financial rows directly.

### 8.2 Required staged environment

The server needs the existing PocketBase admin values plus:

| Variable | Required value | Purpose |
| --- | --- | --- |
| `NEXT_PUBLIC_POCKETBASE_URL` | exact browser/server PB host | File reads and migration host guard. |
| `POCKETBASE_ADMIN_EMAIL` + `POCKETBASE_ADMIN_PASSWORD` or `POCKETBASE_ADMIN_TOKEN` | valid superuser credentials | Server reads and fenced writes. |
| `N8N_INGEST_SECRET` | high-entropy shared secret | Authenticates the tick callback. |
| `MINIMAX_API_KEY` | vision-capable API key | Reads decoded page images. |
| `MINIMAX_BASE_URL` | usually `https://api.minimax.io/v1` | Vision endpoint. |
| `MINIMAX_MODEL` | `MiniMax-M3` or approved vision model | Extraction model. |
| `DN_STAGED_INGESTION_ENABLED` | exactly `false` until rollout, then exactly `true` | Selects staged vs legacy tick runner. |
| `DN_PAGE_PROXY_SECRET` | high-entropy secret (or documented n8n fallback) | Signs review page URLs. |

Keep these values server-only except for `NEXT_PUBLIC_POCKETBASE_URL`.

### 8.3 Tick and lock recovery

`POST /api/dn-batches/tick` is the n8n scheduler entry point and requires
`x-ingest-secret`. The staged runner acquires a PocketBase lease per batch and
fences business mutations in the same transaction. A worker that loses its
lease must not retry writes. Expired `dn_pipeline_locks` rows are reclaimed by
the next tick; do not delete a live lock by hand. If a tick is interrupted,
inspect the batch stage, `last_error`, page statuses, and candidate stages, then
let the next tick resume idempotently. Repeated expiry indicates a timeout or
network/vision problem; fix that before increasing `DN_TICK_BATCH_SIZE`.

### 8.4 Shadow rollout and observation

1. Apply the migration and leave `DN_STAGED_INGESTION_ENABLED=false`.
2. Run the read-only comparison:

   ```bash
   npx tsx --env-file=.env.local scripts/compare-dn-pipelines.mts <batch-id>
   npx tsx --env-file=.env.local scripts/compare-dn-pipelines.mts --all
   ```

   It reports page coverage, candidate count, PO/item links, unit
   compatibility, quantities, computed value, and `writes: false`; it never
   promotes or changes PocketBase.
3. Observe one site's page-read failures, open pages, needs-review rate,
   exact-PO and item-link rates, duplicates, lock reclaims, promotion failures,
   and incorrect automatic financial matches. The last metric must remain zero.
4. Enable staged processing for that site by setting the flag to `true` and
   restarting the server. Expand site by site only after the metrics remain
   acceptable.

### 8.5 Rollback

Set `DN_STAGED_INGESTION_ENABLED=false` and restart to return to the legacy
runner. Do not delete staged rows: they are the audit trail and can be retried
or compared later. Do not edit `delivery_notes`, `delivery_note_items`, or
supplier ledger rows directly. Use the correction/audit workflow and investigate
the candidate, page, lock, and promotion event first. Posted candidates are
immutable unless an explicit correction process is approved.

---

## 9. What this document does NOT cover

- The application logic — see [`architecture.md`](./architecture.md).
- The collections — see [`data-model.md`](./data-model.md).
- The end-to-end pipelines — see [`data-flows.md`](./data-flows.md).
- The dashboard modules — see [`modules.md`](./modules.md).
- Whether the app will hold up at 50 sites — see
  [`multi-site-assessment.md`](./multi-site-assessment.md).