From 9a8d637ff1d38459dbb4a37584e6afcf17e47d7f Mon Sep 17 00:00:00 2001 From: DEls Date: Fri, 11 Sep 2026 10:21:29 +0200 Subject: [PATCH] docs(plan): ingestion restructure implementation plan 20 tasks across four phases, each phase ending at a shippable state. Phase 1 additive only, Phase 2 renames and document collections, Phase 3 route cutover, Phase 4 drops. Nothing is deleted before Phase 4, so every earlier phase is reversible. Co-Authored-By: Claude Opus 5 --- .../plans/2026-09-11-ingestion-restructure.md | 3282 +++++++++++++++++ 1 file changed, 3282 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-11-ingestion-restructure.md diff --git a/docs/superpowers/plans/2026-09-11-ingestion-restructure.md b/docs/superpowers/plans/2026-09-11-ingestion-restructure.md new file mode 100644 index 0000000..f6e1855 --- /dev/null +++ b/docs/superpowers/plans/2026-09-11-ingestion-restructure.md @@ -0,0 +1,3282 @@ +# Ingestion Restructure Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the `data_import_batches` upload pool with five per-entity document collections that each own their file, make upload status reflect what actually happened, put a real `site` on every ingested row, and remove the `project_code` / `qty_outstanding` duplication. + +**Architecture:** Four phases, each ending at a shippable state. Phase 1 is additive only — new columns and backfills, old code untouched and still working. Phase 2 renames collections and creates the three new document collections, migrating existing files across. Phase 3 cuts the routes over one path at a time. Phase 4 drops what is now unused. Nothing is deleted until Phase 4, so every earlier phase is reversible. + +**Tech Stack:** Next.js 16 (App Router, server routes), PocketBase (SQLite), vitest, Python 3 migration scripts under `scripts/migrations/`, n8n for the two AI-backed pipelines. + +**Spec:** `docs/superpowers/specs/2026-09-11-ingestion-restructure-design.md` + +--- + +## Before you start + +Read these first. They are short and they will save you from the two known foot-guns. + +- `docs/superpowers/specs/2026-09-11-ingestion-restructure-design.md` — the design this implements. +- `docs/data-model.md` §7 — **every schema change must land in BOTH `scripts/pb-bootstrap-schema.py` and `src/lib/pb-types.ts`.** Forgetting either side is the documented foot-gun in this repo. +- `docs/data-flows.md` — the current pipelines you are replacing. + +**Things that will bite you if you don't know them:** + +1. **PocketBase collection names are case-insensitive.** The live collections are named `DELIVERY_NOTES`, `TIMESHEETS` etc. in caps, but every call site in `src/` uses lowercase (`pb.collection("delivery_notes")`) and it works. Verified against production. So renames only need to change the *word*, and existing lowercase call sites keep working. +2. **PocketBase has no transactions.** Multi-write operations can partially fail. Every loop that creates child rows must tolerate a mid-loop failure without corrupting what already landed. +3. **You cannot add a `required` field to a collection that has rows violating it.** Backfill first, then tighten. Phase 1 is ordered to respect this. +4. **This runs against production data.** There is one site, 1367 timesheet rows, 320 fuel logs, 24 POs, 32 delivery notes. Every migration script in this plan takes `--apply` and is dry-run by default. Never add `--apply` to a script you have not run dry first. + +**Commands you will use constantly:** + +```bash +npm run test:run # vitest, single pass +npm run typecheck # tsc --noEmit +``` + +Migration scripts follow the existing convention in `scripts/migrations/`: + +```bash +PB_URL=https://pb.mega.infra.apphub.co.za \ +PB_ADMIN_EMAIL=admin@mega.co.za \ +PB_ADMIN_PASSWORD= \ +python3 scripts/migrations/