# Site model pipeline (Phase 1, run by hand)

Turns a consultant's drawings into the five JSON files the Site map page loads
(`bushveld.json`, `services.json`, `earthworks.json`, `schedules.json`, `surface.json`).
Specific to Civil Concepts' layer names on Bushveld Joy Road (C3092); other drawing sets
need the layer maps at the top of each script adjusted.

## Tools
- LibreDWG (`brew install libredwg`) → `dwg2dxf`
- poppler (`brew install poppler`) → `pdftotext`, `pdftocairo`
- Python 3 with numpy

## Build (Bushveld Joy example)
1. Convert DWGs: `dwg2dxf -y -o out.dxf in.dwg` (road layout M910-001, signs & markings M910-002,
   studs M910-005, stormwater LS M921-001).
2. Run in this order, all writing into one output folder `OUT`:
   ```
   python3 -I extract_bushveld.py <road-layout.dxf> "<SO DATA folder>" OUT/bushveld.json
   python3 -I extract_extras.py <drawings folder> <converted DXF folder> OUT
   python3 -I extract_surface.py <road-layout.dxf> OUT
   python3 -I extract_xs.py <cross-section.pdf> <road-layout.dxf> OUT
   ```
   (`extract_xs.py` validates against `surface.json`, so run it last.)
3. Upload: `python3 upload.py --site <site id> --label "<text>" [--dry-run] OUT`.

## Checks printed by the scripts
- extract_bushveld: areas per work type and the 20 m chainage sections.
- extract_surface: contour-only surface vs peg levels (median / 90th percentile).
- extract_xs: rebuilt alignment length vs the drawing (674.412 m for Bushveld Joy) and
  cross-section vs design-surface agreement.
- upload: progress keys recorded on the site that the new model no longer has.

## Tests
`python3 -m unittest discover -s scripts/site-model -p "test_*.py"`

## Cedar Gate (C3277 / Equestria Ext. 289)

The 2026-10-20 delivery contains two projects. Use the C3277 archive and
`Cedar Gate Contours Chris.dwg`; exclude the Nellmapius drawings.

1. Convert the bound C3277-910-001 Rev A road layout, C3277-920to940 Rev B
   services layout, and Chris ground-contour DWG to DXF with `dwg2dxf`.
2. With Python packages `numpy scipy shapely>=2.1 ezdxf` available, run:
   ```sh
   python3 scripts/site-model/extract_cedar_gate.py DXF_FOLDER OUTPUT_FOLDER
   ```
3. Check `validation.json`, the source hashes in `sources.json`, surface coverage,
   and the rendered model. The initial reviewed output is in
   `reports/site-models/cedar-gate/2026-10-20`.
4. Upload with the normal uploader using Cedar Gate site ID `u6n5fye4czinw6o`.

The extractor checks all four alignment lengths against drawing labels, conserves
hatch areas in disjoint 20 m planning sections, measures single kerb lines once,
and keeps surveyed ground separate from the finished-level surface. Service
routes are plan overlays without inferred inverts, pipe sizes or trench volumes.
External lower pavement layers, earthwork volumes and setting-out pegs are
deliberately unquantified pending reconciliation with the relevant source data.

### Cedar Gate pipes and manholes (added 2026-10-21)

`extract_cedar_gate.py` only has plan routes for services. The C3277-921-001 long sections hold the
manholes, kerb inlets, pipe sizes, invert and ground levels, and a coordinate list for every structure
(the sewer sheets 931-001/002 are bound into the same DWG). To add them to a built model folder:

```sh
dwg2dxf -y "C3277-921-001 REVA  -  INT SW LS - 2026-08-11 - bound.dwg"      # rename to a short .dwg first
python3 -I scripts/site-model/extract_cedar_gate_services.py LONGSECTION.dxf reports/site-models/cedar-gate/2026-10-20 \
  reports/site-models/cedar-gate/2026-10-21 "<921 SW LS>.pdf" "<931-001 SR LS>.pdf" "<931-002 SR LS>.pdf"
python3 -m unittest scripts/site-model/test_cedar_gate_services.py
```

The DWG also holds superseded copies of three sewer profiles; the issued PDFs decide which copy is kept
(`services-extraction.json` lists what was kept and dropped). The external SW-M1 line is on the DWG only.
Water is still plan routes only. Upload with the normal uploader.