MEGAINFRA-WEBAPP/scripts/site-model

Site model pipeline (Phase 1, run by hand)

Turns a consultant's drawings into the five JSON files the Site map page loads (bushveld.json, services.json, earthworks.json, schedules.json, surface.json). Specific to Civil Concepts' layer names on Bushveld Joy Road (C3092); other drawing sets need the layer maps at the top of each script adjusted.

Munyaka Village / Munyaka 2 (C3175, added 2026-10-28)

The first model covers the access-road construction issue (12 October 2026). The separate village SDP Rev C (22 October) remains information-stage reference. See reports/site-models/munyaka/2026-10-28/README.md for coverage, source gaps and holds.

Unpack the OneDrive archive and all ten nested ZIPs to a source folder. Convert the unique DWG filenames with LibreDWG, keeping a copy of each input beside its DXF. The DXF folder's files.json maps original DWG names to converted DXF filenames (e.g. "ACCESS NEW.dwg": "00.dxf"). Keep numeric filenames short for LibreDWG; run conversion from the DXF folder because it writes to the current directory. Extract all PDF text with pdftotext -layout into SOURCE/text/. Include the standalone village PDF in SOURCE for its source hash.

With ezdxf numpy scipy shapely>=2.1 installed:

python3 scripts/site-model/extract_munyaka.py DXF_FOLDER SOURCE_FOLDER OUTPUT_FOLDER
python3 -m unittest discover -s scripts/site-model -p 'test_munyaka.py'
python3 scripts/site-model/upload.py --site 1d4hqov69qzajcp \
  --label 'Munyaka Village — access road and services · Rev 0 (2026-10-12)' OUTPUT_FOLDER

The extractor checks source coordinates, builds the stormwater network from the C3175-920-002 structure table (every invert used once), preserves sewer invert drops and the SR1 existing-connection hold, excludes T.B.C./conflicting setting-out points, and conserves new-work areas in disjoint provisional planning strips. It never substitutes finished-design contours for natural ground.

Tools

• LibreDWG (brew install libredwg) → dwg2dxf
• poppler (brew install poppler) → pdftotext, pdftocairo
• Python 3 with numpy

Build (Bushveld Joy example)

1. Convert DWGs: dwg2dxf -y -o out.dxf in.dwg (road layout M910-001, signs & markings M910-002, studs M910-005, stormwater LS M921-001).
2. Run in this order, all writing into one output folder OUT:

   python3 -I extract_bushveld.py <road-layout.dxf> "<SO DATA folder>" OUT/bushveld.json
   python3 -I extract_extras.py <drawings folder> <converted DXF folder> OUT
   python3 -I extract_surface.py <road-layout.dxf> OUT
   python3 -I extract_xs.py <cross-section.pdf> <road-layout.dxf> OUT
   

   (extract_xs.py validates against surface.json, so run it last.)
3. Upload: python3 upload.py --site <site id> --label "<text>" [--dry-run] OUT.

Checks printed by the scripts

• extract_bushveld: areas per work type and the 20 m chainage sections.
• extract_surface: contour-only surface vs peg levels (median / 90th percentile).
• extract_xs: rebuilt alignment length vs the drawing (674.412 m for Bushveld Joy) and cross-section vs design-surface agreement.
• upload: progress keys recorded on the site that the new model no longer has.

Tests

python3 -m unittest discover -s scripts/site-model -p "test_*.py"

Cedar Gate (C3277 / Equestria Ext. 289)

The 2026-10-20 delivery contains two projects. Use the C3277 archive and Cedar Gate Contours Chris.dwg; exclude the Nellmapius drawings.

1. Convert the bound C3277-910-001 Rev A road layout, C3277-920to940 Rev B services layout, and Chris ground-contour DWG to DXF with dwg2dxf.
2. With Python packages numpy scipy shapely>=2.1 ezdxf available, run:

   python3 scripts/site-model/extract_cedar_gate.py DXF_FOLDER OUTPUT_FOLDER
   

3. Check validation.json, the source hashes in sources.json, surface coverage, and the rendered model. The initial reviewed output is in reports/site-models/cedar-gate/2026-10-20.
4. Cut/fill for Cedar Gate Road: convert C3277-912-001 Rev A (cross-sections) and C3277-911-001 Rev A (road long section) to DXF, then run

   python3 scripts/site-model/extract_cedar_gate_xs.py "C3277-912-001 ... - bound.dxf" raw.json OUTPUT_FOLDER \
     --long-section "C3277-911-001 ... - bound.dxf"
   

   raw.json is the CAD geometry cache from step 2 (--raw-cache). The 40 sections are read from their named CAD layers (XS_GROUND, XS_ROAD_SURFACE, layerworks, kerbs, shoulders and the revised side slopes to be shaped); scales come from the grid labels. Every section's centreline ground and design levels must agree with the long section (2 cm median, 10 cm max) or the build stops. The plotted 912 PDF is still accepted in place of the DXF (needs poppler).
5. Upload with the normal uploader using Cedar Gate site ID u6n5fye4czinw6o.

The extractor checks all four alignment lengths against drawing labels, conserves hatch areas in disjoint 20 m planning sections, measures single kerb lines once, and keeps surveyed ground separate from the finished-level surface. Service routes are plan overlays without inferred inverts, pipe sizes or trench volumes. External lower pavement layers and setting-out pegs are deliberately unquantified pending the relevant source data. Earthworks cover Cedar Gate Road only: the long section also covers Furrow Road, Libertas Avenue and the access road, but without their cross-sections and pavement layers their formation volumes are not calculated.

Cedar Gate pipes and manholes (added 2026-10-21)

extract_cedar_gate.py only has plan routes for services. The C3277-921-001 long sections hold the manholes, kerb inlets, pipe sizes, invert and ground levels, and a coordinate list for every structure (the sewer sheets 931-001/002 are bound into the same DWG). To add them to a built model folder:

dwg2dxf -y "C3277-921-001 REVA  -  INT SW LS - 2026-08-11 - bound.dwg"      # rename to a short .dwg first
python3 -I scripts/site-model/extract_cedar_gate_services.py LONGSECTION.dxf reports/site-models/cedar-gate/2026-10-20 \
  reports/site-models/cedar-gate/2026-10-21 "<921 SW LS>.pdf" "<931-001 SR LS>.pdf" "<931-002 SR LS>.pdf"
python3 -m unittest scripts/site-model/test_cedar_gate_services.py

The DWG also holds superseded copies of three sewer profiles; the issued PDFs decide which copy is kept (services-extraction.json lists what was kept and dropped). The external SW-M1 line is on the DWG only. Water is still plan routes only. Upload with the normal uploader.

Nellmapius Ext. 25 (added 2026-10-21)

extract_nellmapius.py reads the converted 100_ROADS and 400_KERBING DWGs (not the Cedar Gate files) and writes a model with road areas in 20 m sections, kerbs, stand/context lines, the four SVR survey benchmarks and an indicative ground plane through them. Nothing else in the supplied set can be placed: the detail contour plan has its contour labels drawn as outlines (no readable levels), the sewer long section is sheet 3 of 3, and the services layout carries no levels. Do not use the model for earthworks or levels.

dwg2dxf -y 100_ROADS.dwg; dwg2dxf -y 400_KERBING.dwg      # LibreDWG
python3 -I scripts/site-model/extract_nellmapius.py 100_ROADS.dxf 400_KERBING.dxf reports/site-models/nellmapius-ext-25/2026-10-21
python3 -m unittest scripts/site-model/test_nellmapius.py
python3 scripts/site-model/upload.py --site dmqybquea190z22 --label "Nellmapius Ext. 25 — roads and kerbs" reports/site-models/nellmapius-ext-25/2026-10-21