MEGAINFRA-WEBAPP/docs/operations.md

# Operations

> Environment variables, scripts, tests, feature flags, and the
> PocketBase API rules in effect in production.

This is the "how do I run it, how do I ship it, where do things live"
document. It deliberately does not document the application logic — see
[`architecture.md`](./architecture.md), [`data-model.md`](./data-model.md),
[`data-flows.md`](./data-flows.md), and [`modules.md`](./modules.md).

---

## 1. Environment variables

The full set lives in `.env.local.example`. The Coolify-friendly subset
(plus PocketBase host env) lives in `docs/cooler-env.template`. They
should agree at all times.

| Variable | Client-exposed? | Required? | Default | Purpose |
| --- | --- | --- | --- | --- |
| `NEXT_PUBLIC_APP_PIN` | yes | yes | `2468` | Login PIN. The lock screen validates it locally. **Must be changed in production.** |
| `NEXT_PUBLIC_APP_NAME` | yes | no | `MEGA INFRASTRUCTURE` | Brand on the lock screen. |
| `NEXT_PUBLIC_DEFAULT_THEME` | yes | no | `light` | `light` or `dark` — pre-hydration fallback. |
| `NEXT_PUBLIC_POCKETBASE_URL` | yes | yes (PB-dependent) | `http://127.0.0.1:8090` | Browser-reachable PB URL. **Also used by `pbAdmin()` server-side.** The runtime joins file URLs off this same base. |
| `NEXT_PUBLIC_API_BASE_URL` | yes | no | unset | Legacy. Documented but unused in current code. |
| `POCKETBASE_ADMIN_EMAIL` | no | yes (server) | unset | Superuser email used by `pbAdmin()`. |
| `POCKETBASE_ADMIN_PASSWORD` | no | yes (server) | unset | Superuser password. |
| `POCKETBASE_ADMIN_TOKEN` | no | optional | unset | If set, used in place of email + password — saves a re-auth on every cold start. Generate via PocketBase admin UI → "Generate token". |
| `N8N_PO_INGEST_URL` | no | yes (PO upload) | unset | Webhook URL the Next.js server forwards PO uploads to. Server-only. |
| `N8N_INGEST_SECRET` | no | yes (n8n callbacks) | unset | Shared secret n8n sends in `x-ingest-secret` on the ingest/process-batch/ocr-result/slip routes. Generate with `openssl rand -hex 32`. |
| `N8N_MCP_URL` | no | optional | unset | Public URL of the n8n MCP server (for the local Codex MCP config). |
| `ENABLE_CHAT` | no | **NO** (production) | unset | Master switch for `/api/chat` and `/dashboard/chat`. The `.env.local.example` warning is explicit: must stay unset in production until real authentication exists. |
| `CHAT_PROVIDER` | no | no | `anthropic` | `anthropic` or `minimax` (MiniMax OpenAI-compatible endpoint). |
| `ANTHROPIC_API_KEY` | no | conditional | unset | Anthropic API key. Leave unset to use an `ant auth login` profile instead. |
| `MINIMAX_API_KEY` | no | conditional | unset | MiniMax Open Platform API key (NOT a Token Plan subscription key). |
| `MINIMAX_BASE_URL` | no | no | `https://api.minimax.io/v1` | MiniMax OpenAI-compatible base URL. |
| `MINIMAX_MODEL` | no | no | `MiniMax-M3` | The only model in the M-series that supports BOTH vision AND tool calling (verified 2026-08-30). M2 / M2.7 accept tools but reject images. |
| `NODE_ENV` | no | yes (prod) | `production` | Standard. |
| `PORT` | no | no | `3000` | Next.js standalone server port. |
| `HOSTNAME` | no | no | `0.0.0.0` | Next.js standalone server bind. |
| `NEXT_TELEMETRY_DISABLED` | no | yes (build) | unset | Set to `1` in the Dockerfile. |

**Anything prefixed `NEXT_PUBLIC_` is inlined into the client bundle at
build time.** Changing `NEXT_PUBLIC_APP_PIN` requires a rebuild —
Coolify will not pick up the change otherwise. (Source:
`COOLIFY.md:55-58`.)

---

## 2. Scripts in `scripts/`

### 2.1 PocketBase schema and rules

| Script | Purpose | Idempotent? |
| --- | --- | --- |
| `pb-bootstrap-schema.py` | Canonical schema definition for the 13 collections. Run after any schema change. | No — running twice may fail because the collections already exist. |
| `pb-relax-rules.py` | Writes the production API ruleset to every collection (`list/view/create` = any authed, `update/delete` = superuser). | Yes — re-running just rewrites the same rules. |
| `migrate-delivery-notes-ocr.py` | One-off migration that adds the OCR `processing_status` / `ocr_*` fields and seeds status on existing DNs. | No — running twice is a no-op on most steps. |
| `migrate-plant-hours-backfill-vehicle.py` | Backfills `plant_hours_log.vehicle` relations from `vehicle_qr` for pre-relation rows. | Idempotent. |

### 2.2 Build-time

| Script | Trigger | Output |
| --- | --- | --- |
| `build-icons.mjs` | `npm run build` (pre-step) | Regenerates the PWA icon set under `public/icons/` |
| `build-og-image.mjs` | `npm run build` (pre-step) | Regenerates `public/og-image.png` for OG previews |
| `patch-next-dev-blocker.cjs` | `npm install` (`postinstall` hook) | Patches Next 16's dev-only cross-site guard so `next dev` accepts any host |
| `generate-plant-hours-template.ts` | Manual | Used by `scripts/build-*.mjs` for the plant-hours template |

The `postinstall` patch is **only loaded by `next dev`** — never by
production. The Dockerfile runs `npm ci --ignore-scripts` in the `deps`
stage to skip the hook entirely (`Dockerfile:32`).

### 2.3 Migrations directory

`scripts/migrations/` holds one-off patches that the schema script
doesn't cover. Examples (without full enumeration — there are several):

- Aug 2026: `latest_consumption` field added to `fuel_logs`.
- Sep 2026: `file_hash` column added to `data_import_batches` for PO
  dedup.

Always add a new entry here when running a one-off schema change. The
canonical pattern is one file per concern, named
`migrate-<collection>-<concern>.py`.

---

## 3. PocketBase API rules

`scripts/pb-relax-rules.py` writes:

- `listRule`, `viewRule`, `createRule` = `@request.auth.id != ''` (any
  authenticated user)
- `updateRule`, `deleteRule` = `null` (superuser only)

Applied to all 13 collections:

```
DATA_IMPORT_BATCHES, SITES, EMPLOYEES, PURCHASE_ORDERS,
PURCHASE_ORDER_ITEMS, FIXED_COSTS, PLANT, VEHICLES, TIMESHEETS,
FUEL_LOGS, DELIVERY_NOTES, RECEIVED_ITEMS
```

What this means in practice:

- The **only client that talks to PB in production is `pbAdmin()`** —
  and it's the superuser, so the rules don't constrain it.
- The browser-side `pb` client (currently unused) would be constrained by
  the rules if it were active.
- The `/api/delivery-notes/[id]/slip` proxy uses the admin token in its
  `Authorization` header — also not constrained by these rules.

**Verifying in production.** Run the script again and compare the output.
If the rules match, every line should print `✓ ... updated`. If anything
prints `FAILED`, investigate before trusting the API.

---

## 4. Running the test suite

```bash
npm run typecheck   # tsc --noEmit
npm run test:run    # vitest run
```

### 4.1 Current state (as of 2026-09-06)

```
Typecheck:  passes cleanly
Tests:      453 passed, 1 failed (1 of 40 files)
```

The single failing test is in
`src/app/api/purchase-orders/process-batch/route.test.ts`:

```
returns 404 when the batch does not exist
  expected 500 to be 404
```

The test mocks `pb.collection(...).getOne` to throw a plain `Error`
without a `status` field. The production code at
`src/app/api/purchase-orders/process-batch/route.ts:137` checks
`(err as { status?: number }).status === 404`. The mock's plain `Error`
has no `status` field, so the production code falls through to the 500
branch. The real PB SDK error does have `status: 404` for missing
records, so this only affects the unit test, not the live behaviour —
but the test is wrong (mocked the SDK shape badly) and the production
code's reliance on `status` is fragile to that.

**Recommended fix.** Either change the test to throw
`Object.assign(new Error("..."), { status: 404 })` to match the SDK
shape, or add `err.status ?? parseInt(err.message.match(/404/)?.[0] ?? "0")`
to the production code. **Tier 3** in
[multi-site-assessment.md](./multi-site-assessment.md).

### 4.2 Test inventory

The `src/lib/metrics/*.test.ts` files are the most important — they pin
every computation the dashboard exposes. Anything that touches a number
should be cross-checked against these.

The `src/lib/chat/*.test.ts` files pin the tool-schemas, the
think-filter, and the chat loop's tool-call limits.

The `src/app/api/**/*.test.ts` files mock `pbAdmin` and exercise the
business logic of each route (file dedup, batch state machine, OCR
tier decision, etc.).

---

## 5. Running the app

### 5.1 Local dev

```bash
cp .env.local.example .env.local
# edit NEXT_PUBLIC_POCKETBASE_URL, POCKETBASE_ADMIN_*, N8N_*
npm install
npm run dev    # next dev -H 0.0.0.0 -p 6004 --webpack
```

The dev server runs on **port 6004** (not 3000 — the user runs other
apps on 3000; do not change this back to 3000). The `--webpack` flag
is required because `@ducanh2912/next-pwa` is webpack-only.

Append `?demo=1` to any URL to skip the PIN gate.

### 5.2 Production build

```bash
npm run build   # icons + og-image + next build --webpack
npm run start   # next start -H 0.0.0.0 -p 3000
```

### 5.3 Docker / Coolify

The `Dockerfile` and `docker-compose.yml` are Coolify v4-ready. See
`COOLIFY.md` for the full setup walkthrough. Key points:

- The standalone server in the final image is non-root, listens on
  `${PORT:-3000}` on `0.0.0.0`.
- The Docker healthcheck is `GET /api/health` every 30s with a 40s
  start period (`Dockerfile:86-87`).
- Traefik routes traffic to the container on port 3000 — there is no
  internal port mapping in the docker-compose.
- The `.dockerignore` excludes pre-built service-worker files so the
  build always ships the right hashes for the new release.

### 5.4 MCP server

```bash
npm run mcp     # tsx --env-file=.env.local src/mcp/server.ts
```

Runs over stdio using the PocketBase credentials in `.env.local`. Opens
no network listener. Exposes the same `tools` array the chat uses, so
an external agent (Codex, etc.) gets identical answers to the in-app
chat.

### 5.5 Deploy logs

The repo tracks one historical deploy log
(`deployment-vptid1uo0jnz2q4t1mcbqjj9-all-logs-2026-08-19-00-23-06.txt`)
as evidence of how the Coolify deploy surface looks when something
breaks. This is left in-tree for context, not as a reference.

---

## 6. Feature flags

| Flag | Default | Effect when on | Effect when off |
| --- | --- | --- | --- |
| `ENABLE_CHAT` | unset (= `false`) | `/api/chat` and `/dashboard/chat` work | Both return 404 / `notFound()` |
| `?demo=1` query string | n/a | Skips the PIN gate on the current URL | Normal auth flow |
| `NEXT_PUBLIC_DEFAULT_THEME` | `light` | Initial theme before user preference loads | " |
| `CHAT_PROVIDER` | `anthropic` | `anthropic` if `ANTHROPIC_API_KEY` is set (or `ant auth login` profile); `minimax` / `openai` if `MINIMAX_API_KEY` is set |

There is no other feature-flag mechanism. Anything else that needed to
be toggleable in the past has been either hardcoded (variance tolerances,
`RATE_PER_LITRE`) or set via env (`RATE_PER_LITRE` is in
`src/lib/metrics/constants.ts` — **not** env-driven; change there).

---

## 7. Smoke tests after a deploy

The two routes that should always return 200 in production:

| Probe | URL | Expects |
| --- | --- | --- |
| Liveness | `GET /api/health` | `200` with `{ status: "ok", uptime, timestamp }` |
| Auth gate | `GET /login` | `200` HTML (the lock screen) |

`curl -sS -o /dev/null -w "%{http_code}\n" https://<host>/api/health`
should print `200`. If it doesn't, the container isn't ready (start
period is 40s).

## 8. Staged delivery-note ingestion

The staged pipeline is deliberately off until a representative site has been
observed. It is additive: original source files and the legacy runner remain
available, and financial rows are created only by the posting action.

### 8.1 Migration

Preview the schema first. This command never writes and does not require admin
credentials:

```bash
set -a; source .env.local; set +a
python3 scripts/pb-add-dn-staged-ingestion.py --dry-run
```

Apply only after reviewing the preview and confirming the exact PocketBase host:

```bash
set -a; source .env.local; set +a
python3 scripts/pb-add-dn-staged-ingestion.py --apply \
  --confirm-host "$NEXT_PUBLIC_POCKETBASE_URL"
```

The script adds the staged collections, fields, and indexes. Do not remove
collections or edit financial rows directly.

### 8.2 Required staged environment

The server needs the existing PocketBase admin values plus:

| Variable | Required value | Purpose |
| --- | --- | --- |
| `NEXT_PUBLIC_POCKETBASE_URL` | exact browser/server PB host | File reads and migration host guard. |
| `POCKETBASE_ADMIN_EMAIL` + `POCKETBASE_ADMIN_PASSWORD` or `POCKETBASE_ADMIN_TOKEN` | valid superuser credentials | Server reads and fenced writes. |
| `N8N_INGEST_SECRET` | high-entropy shared secret | Authenticates the tick callback. |
| `MINIMAX_API_KEY` | vision-capable API key | Reads decoded page images. |
| `MINIMAX_BASE_URL` | usually `https://api.minimax.io/v1` | Vision endpoint. |
| `MINIMAX_MODEL` | `MiniMax-M3` or approved vision model | Extraction model. |
| `DN_STAGED_INGESTION_ENABLED` | exactly `false` until rollout, then exactly `true` | Selects staged vs legacy tick runner. |
| `DN_PAGE_PROXY_SECRET` | high-entropy secret (or documented n8n fallback) | Signs review page URLs. |

Keep these values server-only except for `NEXT_PUBLIC_POCKETBASE_URL`.

### 8.3 Tick and lock recovery

`POST /api/dn-batches/tick` is the n8n scheduler entry point and requires
`x-ingest-secret`. The staged runner acquires a PocketBase lease per batch and
fences business mutations in the same transaction. A worker that loses its
lease must not retry writes. Expired `dn_pipeline_locks` rows are reclaimed by
the next tick; do not delete a live lock by hand. If a tick is interrupted,
inspect the batch stage, `last_error`, page statuses, and candidate stages, then
let the next tick resume idempotently. Repeated expiry indicates a timeout or
network/vision problem; fix that before increasing `DN_TICK_BATCH_SIZE`.

### 8.4 Shadow rollout and observation

1. Apply the migration and leave `DN_STAGED_INGESTION_ENABLED=false`.
2. Run the read-only comparison:

   ```bash
   npx tsx --env-file=.env.local scripts/compare-dn-pipelines.mts <batch-id>
   npx tsx --env-file=.env.local scripts/compare-dn-pipelines.mts --all
   ```

   It reports page coverage, candidate count, PO/item links, unit
   compatibility, quantities, computed value, and `writes: false`; it never
   promotes or changes PocketBase.
3. Observe one site's page-read failures, open pages, needs-review rate,
   exact-PO and item-link rates, duplicates, lock reclaims, promotion failures,
   and incorrect automatic financial matches. The last metric must remain zero.
4. Enable staged processing for that site by setting the flag to `true` and
   restarting the server. Expand site by site only after the metrics remain
   acceptable.

### 8.5 Rollback

Set `DN_STAGED_INGESTION_ENABLED=false` and restart to return to the legacy
runner. Do not delete staged rows: they are the audit trail and can be retried
or compared later. Do not edit `delivery_notes`, `delivery_note_items`, or
supplier ledger rows directly. Use the correction/audit workflow and investigate
the candidate, page, lock, and promotion event first. Posted candidates are
immutable unless an explicit correction process is approved.

---

## 9. What this document does NOT cover

- The application logic — see [`architecture.md`](./architecture.md).
- The collections — see [`data-model.md`](./data-model.md).
- The end-to-end pipelines — see [`data-flows.md`](./data-flows.md).
- The dashboard modules — see [`modules.md`](./modules.md).
- Whether the app will hold up at 50 sites — see
  [`multi-site-assessment.md`](./multi-site-assessment.md).