diff --git a/.superpowers/sdd/2026-09-18-hybrid-delivery-note-ingestion/progress.md b/.superpowers/sdd/2026-09-18-hybrid-delivery-note-ingestion/progress.md new file mode 100644 index 0000000..6049e45 --- /dev/null +++ b/.superpowers/sdd/2026-09-18-hybrid-delivery-note-ingestion/progress.md @@ -0,0 +1,161 @@ +# SDD ledger — plan: docs/superpowers/plans/2026-09-18-hybrid-delivery-note-ingestion.md + +Workspace: /Volumes/EXT/DEVELOPMENT/MEGAIFRASTRUCTURE/.worktrees/hybrid-dn-ingestion +Branch: codex/hybrid-dn-ingestion +Merge base: 7ab5570 +Baseline: npm run test:run — 85 files, 1023 tests passed + +## Pre-flight scan + +| Task(s) | Produce / consume or self-check | Finding | +| --- | --- | --- | +| 1 | Status helpers, schema and PB types consumed by 3–12 | Internally consistent; implementation must remain additive and dry-run only during development. | +| 2 | Canonical PNG pages consumed by 4, 7 and 9 | Internally consistent; original bytes remain in sources, so canonicalisation cannot destroy evidence. | +| 3 | Multi-source upload consumes Task 1 schema | Internally consistent; repeated capture-session lookup may query source rows because capture_session_id is source metadata. | +| 4 | Rich PageRead consumed by 5–7 | Internally consistent; new fields are additive and current commercial fields remain required/defaulted. | +| 5 | Assembly consumes Task 4 and supplies Task 7/9 | Internally consistent; legacy segmentPages remains an adapter so current tests and runner can coexist. | +| 6 | Resolver consumes Task 1 aliases and Task 4 reads; both legacy run.ts and staged runner need it | Ruling: put matching policy in the pure resolver and keep database loading thin; the feature flag preserves legacy production behavior. If wrong, duplicated DB adapters may need consolidation. | +| 7 | Lock/staged runner consumes Tasks 1–6 and switches tick behind a flag | Internally consistent; no production switch without DN_STAGED_INGESTION_ENABLED=true. | +| 8 | Promotion consumes candidate identity from Tasks 1/7 | Ruling: an unposted partial promotion may reconcile its own candidate rows, but a posted candidate is immutable without explicit correction. If wrong, a retry could either leave partial rows or overwrite audited data. | +| 9 | Split UI consumes Task 5 assembly and Task 1 status | Internally consistent; persisted-page route must retain legacy fallback. | +| 10 | Queue consumes Task 1 stages and Task 6 relations | Internally consistent; it must not alter supplier ledger totals. | +| 11 | Review actions consume Tasks 1, 6, 8 and existing dn_item_aliases | Internally consistent after plan correction distinguishing supplier aliases from item aliases. | +| 12 | Manual posting consumes Task 8 and staged state from Task 7 | Internally consistent; auto-post and manual post share the same idempotent promotion function. | +| 1 + 3 | Schema fields against upload payload | Compatible; source_count is updated only after successful source writes. | +| 2 + 4 | Canonical MIME against vision data URL | Compatible; extract must always use image/png after normalisation. | +| 4 + 5 | Printed pagination and continuation fields | Compatible; deterministic assembly uses values actually present on each page. | +| 5 + 8 | Assembly revisions against posted candidates | Compatible; revision endpoint rejects posted candidates. | +| 6 + 8 | Supplier-scoped identity | Compatible; exact PO may derive supplier before duplicate identity is evaluated. | +| 7 + 12 | Automatic and manual posting paths | Compatible if both call the same gate/promotion functions; reviewers must reject duplicated posting logic. | +| 9 + 11 | UI review actions against API validation | Compatible; browser suggestions are never authoritative. | + +## Rulings + +Ruling: Matching policy belongs in a pure resolver shared by legacy and staged adapters — keeps site scoping and auto-match safety in one unit — cost if wrong: a later refactor may be needed to untangle DB loading. + +Ruling: Only unposted partial promotions may be reconciled automatically; posted candidates require explicit correction — preserves auditability — cost if wrong: recovery from a mid-write failure may need a manual action. + +Task 1: dispatched to /root/luna_task1 at base 7ab55703b4041c41bcda378b74bd56306498b0d4 + +Task 1: Ruling: retain deployable apply capability but require explicit --apply plus an exact host confirmation; development and automated verification remain dry-run only — the spec requires an additive migration that can eventually be applied — cost if wrong: production schema application has extra operator ceremony. + +Task 1: fix round 1/5 (3 addressed, 2 open — replace same-name legacy index instead of appending; align review-event type optionality; commits 34dbdeb..ad23b65) + +Task 1: fix round 2/5 (2 addressed, 0 open; commits ad23b65..498b258) +Task 1: minor (deferred): index-name parser handles the migration's current conventional SQLite syntax, not every possible quoted/IF-NOT-EXISTS variant. +Task 1: complete (commits 7ab5570..498b258, review clean) + +Task 2: dispatched to /root/luna_task2 at base 498b25858aed52df2bfb1360c9fff4386d7bc997 + +Task 2: review found extractPageRange still trusted declared MIME, breaking mismatched-PDF page evidence; splitPages comment is stale. + +Task 2: fix round 1/5 (2 addressed, 0 open; commits 81c8f78..80fbe15) +Task 2: complete (commits 498b258..80fbe15, review clean) + +Task 3: dispatched at base 80fbe15 + +Task 3: review found duplicate-only uploads create empty batches, capture session filter interpolation is unsafe, legacy single-file validation regressed, and persisted source_count remains false. + +Task 3: fix round 1/5 (2 addressed, 4 open — oversize got detail; source_count update failure; in-request hash dedup; report accuracy; commits e7e92fd..798b804) + +Task 3: fix round 2/5 (2 addressed, 2 open — containment if final update and cleanup both fail; same-hash outcome after primary write failure; commits 798b804..a8ed5a1) + +Task 3: fix round 3/5 (3 addressed, 0 open; commits a8ed5a1..ba52884) +Task 3: complete (commits 80fbe15..ba52884, review clean) + +Task 4: dispatched at base ba5288465e0ccd82c76a9fcc1af858b399b8549b + +Task 4: Ruling: production runner integration of readOrReusePage is deferred to Task 7 run-staged.ts, as the plan explicitly assigns orchestration there; Task 4 owns the tested primitive — cost if wrong: cache code remains unused until Task 7 and could drift before integration. +Task 4: review found failure-state write errors are swallowed, rich types are weaker than the specified parser contract, and cached extraction bypasses validation. + +Task 4: fix round 1/5 (3 addressed, 1 open — report fix SHA mismatch; commits 0bd344a..ea791d6) + +Task 4: fix round 2/5 (1 addressed, 0 open; commits ea791d6..a1882ac) +Task 4: complete (commits ba52884..a1882ac, review clean) + +Task 5: dispatched at base a1882ac + +Task 5: Ruling: assembly_revision belongs on dn_batches because one revision covers the complete page grouping; retain note-level assembly_revision only as candidate provenance — cost if wrong: future per-candidate revisions may need a second token. +Task 5: review found missing batch schema field, unstable note ordering, incomplete posted-state guard, malformed-group gaps, and mobile hints overriding conflicting printed pagination. + +Task 5: fix round 1/5 (6 addressed, 2 open — stale candidates survived group merges; revision check was not serialized; commits ab57baf..ab57baf) + +Task 5: fix round 2/5 (2 addressed, 2 open — dismissed candidates could be deleted; report retained a dangling final-commit claim; commits ab57baf..297853f) + +Task 5: fix round 3/5 (2 addressed, 0 open — dismissed candidates are immutable during regrouping; report provenance corrected; commits 297853f..59c158c) +Task 5: complete (commits a1882ac..59c158c, review clean) + +Task 6: dispatched to /root/luna_task6 at base 59c158ce37e2d0b758f7e3b16cf54268f2ce4850 + +Task 6: review found supplier evidence was dropped at segmentation, real PocketBase PO projection requested a nonexistent field, supplier identifier schema/loading was absent, and low-confidence fuzzy evidence was discarded. + +Task 6: fix round 1/5 (5 addressed, 0 open — supplier evidence preserved; schema/loading aligned; fuzzy evidence retained; commits e598613..927c09b) +Task 6: complete (commits 59c158c..927c09b, review clean) + +Task 7: dispatched to /root/tara_task7 at base 927c09bbb27891b34604a077ed27915adac544af + +Task 7: Ruling: lease record ids are acquisition-specific and derived from batch plus a fresh owner token; the unique batch index, not a reused record id, is the atomic gate — PocketBase only supports id-based delete, so distinct ids fence old releases from replacement locks — cost if wrong: callers or tests that assumed one fixed id per batch must be adapted. + +Task 7: review found unsafe replacement-lock deletion, missing claimed_at on staged work, transient note-index candidate identity, and reused pages not reading their persisted canonical image. + +Task 7: fix round 1/5 (2 addressed, 3 open — candidate identity still changed on recovered continuations and ignored keyless rows; PocketBase file URLs lacked collection metadata; commits 1f988a9..02d2cbf) + +Task 7: fix round 2/5 (4 addressed, 1 open — surplus reconciliation did not recognize legacy promoted rows as terminal; commits 02d2cbf..66bf2a0) + +Task 7: fix round 3/5 (1 addressed, 0 open — legacy promoted and delivery-note-linked rows preserved; commits 66bf2a0..6ff5b59) +Task 7: complete (commits 927c09b..6ff5b59, review clean) + +Task 8: dispatched to /root/luna_task8 at base 6ff5b59 + +Task 8: review found legacy reruns could mutate posted candidates, create races did not converge, runner MIME/name provenance was lost, and promotion-failure visibility writes were swallowed. + +Task 8: fix round 1/5 (4 addressed, 2 open — candidate read failed open; failure-state writes retained a posted-state TOCTOU race; commits 4820706..43391a8) + +Task 8: Ruling: the additive staged-ingestion migration is a deployment prerequisite; missing dn_pipeline_locks fails closed with an actionable schema-required error rather than falling back to unlocked promotion — preserves posting correctness — cost if wrong: deployment order is stricter and unmigrated installations cannot run the new promotion code. + +Task 8: fix round 2/5 (2 addressed, 3 open — legacy upsert remained outside lease; lease lacked renewal; missing lock schema was reported as contention; commits 43391a8..e3e7135) + +Task 8: fix round 3/5 (3 addressed, 2 open — page-store writes were not lease-aware; ownership assertions were not transactionally fenced with mutations; commits e3e7135..16f42a7) + +Task 8: Ruling: protected PocketBase mutations must use the server transactional batch API to update the current acquisition record and the business row together; heartbeats remain liveness only — this closes the expiry/reclaim race without weakening reclaim — cost if wrong: all protected write adapters must support PocketBase batch semantics and older PocketBase servers cannot run this release. + +Task 8: fix round 4/5 (2 addressed, 0 open — page/candidate/manual/financial writes transactionally fenced; SDK batch support required; commits 16f42a7..919124f) +Task 8: complete (commits 6ff5b59..919124f, review clean) + +Task 9: dispatched to /root/luna_task9 at base 919124f + +Task 9: review found preparing rows hidden, empty groups after join, open pages not actionable, persisted page groups/dismissals not rehydrated, direct PocketBase image URLs bypassed protection, and missing upper page bound. + +Task 9: fix round 1/5 (7 addressed, 2 open — orphan pages absent from both groups/open_pages remained invisible; server page imported a runtime helper from a client module; commits 61c0e60..d22d7b1) + +Task 9: fix round 2/5 (2 addressed, 0 open — server-safe model helpers and actionable orphan pages; commits d22d7b1..26143e5) +Task 9: complete (commits 919124f..26143e5, review clean) + +Task 10: dispatched to /root/luna_task10 at base 26143e5 + +Task 10: review found staged doc_number/doc_type were not authoritative, raw supplier text incorrectly escaped Unknown supplier, supplier counts were quadratic, and tests did not render/match server mapping. + +Task 10: fix round 1/5 (4 addressed, 1 open — empty-string date prevented posted/uploaded fallback; commits 2e5e504..b2582d3) + +Task 10: fix round 2/5 (1 addressed, 0 open — date fallback restored; commits b2582d3..bf00cc0) +Task 10: complete (commits 26143e5..bf00cc0, review clean) + +Task 11: dispatched to /root/luna_task11 at base bf00cc0 + +Task 11: review found the existing correction UI still used the obsolete request body, readiness failed open when gates were missing, item linking allowed unresolved site/supplier ownership, alias approval metadata was incomplete, and null JSON could raise a server error. + +Task 11: fix round 1/5 dispatched (5 findings plus focused regression coverage requested; commits 25f8112..70117628 under review) + +Task 11: fix round 1/5 (5 addressed, 3 open — persisted financial checks could be ignored; missing assembly revisions passed; normalized alias keys could be empty; commits 70117628..9ce4250) + +Task 11: fix round 2/5 dispatched (3 findings plus focused regression coverage requested) + +Task 11: fix round 2/5 (3 production findings addressed, 2 proof gaps open — unknown failed-check coverage and route-level transactional rollback evidence; commits 9ce4250..0846ce5) + +Task 11: fix round 3/5 dispatched (tests only) + +Task 11: fix round 3/5 (2 proof gaps addressed, 0 open; commits 0846ce5..45b3c66) +Task 11: complete (commits bf00cc0..45b3c66, review clean) + +Task 12: dispatched at base 45b3c66 +Task 12: complete (code commit 5d09418, report commit e6fef00; focused and complete verification passed; production migration and feature flag unchanged) diff --git a/.superpowers/sdd/2026-09-18-hybrid-delivery-note-ingestion/task-12-report.md b/.superpowers/sdd/2026-09-18-hybrid-delivery-note-ingestion/task-12-report.md new file mode 100644 index 0000000..3456da1 --- /dev/null +++ b/.superpowers/sdd/2026-09-18-hybrid-delivery-note-ingestion/task-12-report.md @@ -0,0 +1,32 @@ +# Task 12 report + +## Implementation + +- Code commit: `5d094183e66663dca69e7d95325cabebc38e5d82` +- Added `POST /api/dn-batches/[id]/notes/[noteId]/post`. + - Reloads the batch, candidate, pages, supplier, PO, every PO item, and checks server-side. + - Reuses the fail-closed readiness evaluator and returns `409 needs_review` with reasons. + - Calls `promoteNote` directly so it owns the batch lease. + - Validates posted-candidate associations and returns the existing delivery note on retry. + - Adds an idempotent `post` audit event after promotion. +- Added read-only `scripts/compare-dn-pipelines.ts` with `` and `--all` modes. +- Documented migration, environment, tick/lock recovery, shadow rollout, metrics, feature flag, rollback, and direct-financial-edit warnings in `docs/operations.md`. +- Added route tests for review blocking, one-time posting, and idempotent retry. + +## Verification + +- `npx vitest run 'src/app/api/dn-batches/[id]/notes/[noteId]/post/route.test.ts' src/lib/dn-pipeline/readiness.test.ts src/lib/dn-pipeline/promote.test.ts --maxWorkers=1 --minWorkers=1` + - 36 tests passed. +- `npm run typecheck` + - Passed. +- `npm run test:run` + - 104 test files, 1,176 tests passed. + - No Poppler concurrency timeout occurred; serial rerun was not needed. +- `npm run build` + - Passed; Next.js listed the new posting route. +- `git diff --check` + - Passed. +- `python3 scripts/pb-add-dn-staged-ingestion.py --dry-run` with the configured PocketBase environment + - Passed; reported intended additive collections/fields/indexes and no writes. + +Production migration was not applied and `DN_STAGED_INGESTION_ENABLED` was not changed.