MEGAINFRA-WEBAPP/scripts/migrations
DEls 539ec93d2a feat(migration): move batch files into document collections
Migrates the 27 batches of type timesheets/fuel_logs/plant_hours out of
data_import_batches into the three document collections created in Task
6, preserving every source file and relinking every child row.

Status is derived from actual child count, not copied from the batch
row. Production proved the stored status cannot be trusted: every one of
these batches was written status=processed at creation time, before any
insert ran.

The bug is worse than the plan's own audit found. It predicted 18
phantom-processed batches, all plant_hours. Production has 21 -- three
timesheets batches are also phantom, including two byte-identical
re-uploads of the same wage sheet that both reported success and created
nothing. This hit payroll, not just plant-hour smoke tests.

Also: computed file_hash on migrated documents while the bytes were
already in hand (closes the same dedup gap Task 1 closed on delivery
notes and purchase orders), and split the schema-additive step
(--apply-fields) from the row migration (--apply) so the reversible part
can run independently of the irreversible one. The plan claimed
idempotency via a migrated_from_batch field that doesn't exist; replaced
with an explicit refuse-if-already-populated guard rather than pretend
resumability that isn't there.

27/27 migrated. 6/6 post-migration invariants verified: 3 fuel_sheets,
19 plant_hours_sheets, 5 timesheets, 1367/247/1 children relinked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 13:06:40 +02:00
..
2026-08-19-extend-data-import-batches.py
2026-08-24-data-import-batches-allow-xlsx.py
2026-08-24-fuel-logs-latest-consumption.py
2026-08-28-vehicles-attention-dismissed.py feat(fuel): PB schema migration for attention_dismissed_at 2026-08-28 12:26:23 +02:00
2026-08-28-vehicles-unit-type-optional.py fix(pb): make vehicles.unit_type optional 2026-08-28 15:26:33 +02:00
2026-08-29-fixed-costs-recurring-backfill.py fix(schema): ensure is_recurring=false writes are preserved 2026-08-28 13:43:38 +02:00
2026-08-31-data-import-batches-allow-pdf-images.py fix(pb): allow PDF and image MIME on data_import_batches.original_file 2026-08-31 12:20:41 +02:00
2026-09-11-add-document-fields.py feat(schema): add file_hash and queue fields to DNs and POs 2026-09-11 10:29:10 +02:00
2026-09-11-backfill-entity-file-hashes.py fix(migration): separate download failures from missing files 2026-09-11 10:39:54 +02:00
2026-09-11-backfill-fuel-log-sites.py fix(fuel): backfill site on 147 orphaned fuel logs 2026-09-11 10:44:19 +02:00
2026-09-11-backfill-timesheet-sites.py fix(timesheets): stamp site on log rows, ending read-time drift 2026-09-11 10:48:51 +02:00
2026-09-12-create-document-collections.py feat(schema): add timesheets, fuel_sheets, plant_hours_sheets documents 2026-09-11 11:09:14 +02:00
2026-09-12-migrate-batches-to-documents.py feat(migration): move batch files into document collections 2026-09-11 13:06:40 +02:00
2026-09-12-rename-collections.py refactor(schema): rename received_items and timesheets to their real names 2026-09-11 11:01:40 +02:00
2026-09-12-widen-sheet-mimetypes.py feat(schema): add timesheets, fuel_sheets, plant_hours_sheets documents 2026-09-11 11:09:14 +02:00