MEGAINFRA-WEBAPP/scripts/site-model/README.md

# Site model pipeline (Phase 1, run by hand)

Turns a consultant's drawings into the five JSON files the Site map page loads
(`bushveld.json`, `services.json`, `earthworks.json`, `schedules.json`, `surface.json`).
Specific to Civil Concepts' layer names on Bushveld Joy Road (C3092); other drawing sets
need the layer maps at the top of each script adjusted.

## Munyaka Village / Munyaka 2 (C3175, added 2026-10-28)

The first model covers the access-road construction issue (12 October 2026).
The separate village SDP Rev C (22 October) remains information-stage reference.
See `reports/site-models/munyaka/2026-10-28/README.md` for coverage, source gaps and holds.

Unpack the OneDrive archive and all ten nested ZIPs to a source folder. Convert
the unique DWG filenames with LibreDWG, keeping a copy of each input beside its
DXF. The DXF folder's `files.json` maps original DWG names to converted DXF
filenames (e.g. `"ACCESS NEW.dwg": "00.dxf"`). Keep numeric filenames short for
LibreDWG; run conversion from the DXF folder because it writes to the current
directory. Extract all PDF text with `pdftotext -layout` into `SOURCE/text/`.
Include the standalone village PDF in SOURCE for its source hash.

With `ezdxf numpy scipy shapely>=2.1` installed:

```sh
python3 scripts/site-model/extract_munyaka.py DXF_FOLDER SOURCE_FOLDER OUTPUT_FOLDER
python3 -m unittest discover -s scripts/site-model -p 'test_munyaka.py'
python3 scripts/site-model/upload.py --site 1d4hqov69qzajcp \
  --label 'Munyaka Village — access road and services · Rev 0 (2026-10-12)' OUTPUT_FOLDER
```

The extractor checks source coordinates, builds the stormwater network from the C3175-920-002 structure table (every invert used once), preserves sewer invert drops and the
SR1 existing-connection hold, excludes T.B.C./conflicting setting-out points,
and conserves new-work areas in disjoint provisional planning strips. It never
substitutes finished-design contours for natural ground.

## Tools
- LibreDWG (`brew install libredwg`) → `dwg2dxf`
- poppler (`brew install poppler`) → `pdftotext`, `pdftocairo`
- Python 3 with numpy

## Build (Bushveld Joy example)
1. Convert DWGs: `dwg2dxf -y -o out.dxf in.dwg` (road layout M910-001, signs & markings M910-002,
   studs M910-005, stormwater LS M921-001).
2. Run in this order, all writing into one output folder `OUT`:
   ```
   python3 -I extract_bushveld.py <road-layout.dxf> "<SO DATA folder>" OUT/bushveld.json
   python3 -I extract_extras.py <drawings folder> <converted DXF folder> OUT
   python3 -I extract_surface.py <road-layout.dxf> OUT
   python3 -I extract_xs.py <cross-section.pdf> <road-layout.dxf> OUT
   ```
   (`extract_xs.py` validates against `surface.json`, so run it last.)
3. Upload: `python3 upload.py --site <site id> --label "<text>" [--dry-run] OUT`.

## Checks printed by the scripts
- extract_bushveld: areas per work type and the 20 m chainage sections.
- extract_surface: contour-only surface vs peg levels (median / 90th percentile).
- extract_xs: rebuilt alignment length vs the drawing (674.412 m for Bushveld Joy) and
  cross-section vs design-surface agreement.
- upload: progress keys recorded on the site that the new model no longer has.

## Tests
`python3 -m unittest discover -s scripts/site-model -p "test_*.py"`

## Cedar Gate (C3277 / Equestria Ext. 289)

The 2026-10-20 delivery contains two projects. Use the C3277 archive and
`Cedar Gate Contours Chris.dwg`; exclude the Nellmapius drawings.

1. Convert the bound C3277-910-001 Rev A road layout, C3277-920to940 Rev B
   services layout, and Chris ground-contour DWG to DXF with `dwg2dxf`.
2. With Python packages `numpy scipy shapely>=2.1 ezdxf` available, run:
   ```sh
   python3 scripts/site-model/extract_cedar_gate.py DXF_FOLDER OUTPUT_FOLDER
   ```
3. Check `validation.json`, the source hashes in `sources.json`, surface coverage,
   and the rendered model. The initial reviewed output is in
   `reports/site-models/cedar-gate/2026-10-20`.
4. Cut/fill for Cedar Gate Road: convert C3277-912-001 Rev A (cross-sections) and
   C3277-911-001 Rev A (road long section) to DXF, then run
   ```sh
   python3 scripts/site-model/extract_cedar_gate_xs.py "C3277-912-001 ... - bound.dxf" raw.json OUTPUT_FOLDER \
     --long-section "C3277-911-001 ... - bound.dxf"
   ```
   `raw.json` is the CAD geometry cache from step 2 (`--raw-cache`). The 40 sections are read from
   their named CAD layers (XS_GROUND, XS_ROAD_SURFACE, layerworks, kerbs, shoulders and the revised
   side slopes to be shaped); scales come from the grid labels. Every section's centreline ground
   and design levels must agree with the long section (2 cm median, 10 cm max) or the build stops.
   The plotted 912 PDF is still accepted in place of the DXF (needs poppler).
5. Upload with the normal uploader using Cedar Gate site ID `u6n5fye4czinw6o`.

The extractor checks all four alignment lengths against drawing labels, conserves
hatch areas in disjoint 20 m planning sections, measures single kerb lines once,
and keeps surveyed ground separate from the finished-level surface. Service
routes are plan overlays without inferred inverts, pipe sizes or trench volumes.
External lower pavement layers and setting-out pegs are deliberately unquantified
pending the relevant source data. Earthworks cover Cedar Gate Road only: the long
section also covers Furrow Road, Libertas Avenue and the access road, but without
their cross-sections and pavement layers their formation volumes are not calculated.

### Cedar Gate pipes and manholes (added 2026-10-21)

`extract_cedar_gate.py` only has plan routes for services. The C3277-921-001 long sections hold the
manholes, kerb inlets, pipe sizes, invert and ground levels, and a coordinate list for every structure
(the sewer sheets 931-001/002 are bound into the same DWG). To add them to a built model folder:

```sh
dwg2dxf -y "C3277-921-001 REVA  -  INT SW LS - 2026-08-11 - bound.dwg"      # rename to a short .dwg first
python3 -I scripts/site-model/extract_cedar_gate_services.py LONGSECTION.dxf reports/site-models/cedar-gate/2026-10-20 \
  reports/site-models/cedar-gate/2026-10-21 "<921 SW LS>.pdf" "<931-001 SR LS>.pdf" "<931-002 SR LS>.pdf"
python3 -m unittest scripts/site-model/test_cedar_gate_services.py
```

The DWG also holds superseded copies of three sewer profiles; the issued PDFs decide which copy is kept
(`services-extraction.json` lists what was kept and dropped). The external SW-M1 line is on the DWG only.
Water is still plan routes only. Upload with the normal uploader.

### Nellmapius Ext. 25 (added 2026-10-21)

`extract_nellmapius.py` reads the converted `100_ROADS` and `400_KERBING` DWGs (not the Cedar Gate files) and writes
a model with road areas in 20 m sections, kerbs, stand/context lines, the four SVR survey benchmarks and an
indicative ground plane through them. Nothing else in the supplied set can be placed: the detail contour plan
has its contour labels drawn as outlines (no readable levels), the sewer long section is sheet 3 of 3, and the
services layout carries no levels. Do not use the model for earthworks or levels.

```sh
dwg2dxf -y 100_ROADS.dwg; dwg2dxf -y 400_KERBING.dwg      # LibreDWG
python3 -I scripts/site-model/extract_nellmapius.py 100_ROADS.dxf 400_KERBING.dxf reports/site-models/nellmapius-ext-25/2026-10-21
python3 -m unittest scripts/site-model/test_nellmapius.py
python3 scripts/site-model/upload.py --site dmqybquea190z22 --label "Nellmapius Ext. 25 — roads and kerbs" reports/site-models/nellmapius-ext-25/2026-10-21
```