Caught by the engineer: '.04 Appointment of Community Liaison Officer' showed
RATE ONLY in the spreadsheet while the PDF has Month, 8, R20 000.00,
R160 000.00. The values had landed on the row ABOVE it.
Two causes, both the same underlying mistake.
1. A BOQ row here is often TWO rows - the one with the item number, unit and
quantity, and a following row carrying the wrapped tail of the description:
r41 C=.04 Appointment of Community Liaison Officer Month 8 20 000
r42 for the duration of the contract period
Those tails were separate candidates, and the aligner matched the line to
the PREVIOUS line's tail. A row with no item number of its own and nothing
in the priced columns is now joined onto the line above.
2. The row classifier still tested the single column the header claimed for
DESCRIPTION (E) while the text is in D - the same header-lies problem I had
fixed for matching but not for classification. Row 41 was never counted as
a line, so it was never even a candidate. Both now use the description RUN.
Also worth being precise about: the 'RATE ONLY' in that cell was the
consultant's own pre-existing text in his template, not something this tool
wrote. The bug was the values going to the wrong row.
Central Towers 192 written / 23 unplaced -> 196 / 22, and four more genuine
disagreements are now visible because values land on the right rows.
Blueberry unchanged at 103 / 1. Zero formulas damaged on both, gate not
blocked.
The engineer reported that some of his tariffs on Blueberry Square were saying
'included' and did not pull through. He was right, and it was not a big issue
for the wrong reason - it was a real gap.
Candy prints a WORD in the rate column where a tariff is carried inside a sum
above rather than priced separately - INCL, incl VAT, PROV. Those lines have a
quantity but no figure, so there was nothing numeric to write and the cells
arrived BLANK. To the consultant that is indistinguishable from a failed
conversion.
The source's own wording is now written into the rate cell and the line is
listed as a note:
F15 = 'INCL' a) Office buildings
F17 = 'INCL' b) Survey Equipment
20 such lines on the Blueberry pair. The amount column is left alone because
there genuinely is no figure for that line - the money is in the sum above it.
Caught a placement bug doing it: the write sat after the early return for lines
whose quantity already matched, which is most of them, so it fired zero times.
Moved it before the quantity block.
Both pairs re-verified after: formulas intact, 0 damaged, gate not blocked.
Asked what the 160 items under Disagreements were. Almost none were
disagreements, and a panel that cries wolf at 160 teaches people to scroll
past it - worse than having no panel at all.
147 of the 160 were the formula guard firing once per row. The consultant's
amount column is a formula on nearly every line, so 'I protected your model'
was being reported as 'a person must choose'. It is now its own quiet note and
cannot crowd out findings that genuinely need a decision.
12 more were unit formatting - 'PC Sum' vs 'PCSum' vs 'PC.Sum' is one unit
written three ways. Unit comparison now collapses case, spacing and dots, so
only genuinely different units are flagged.
160 -> 19, and all 19 are real. The ones worth a human's eye: seven quantity
disagreements (template 47.5 vs source 95.0) and eleven unit mismatches, five
of which are 'm' vs 'm3' - pricing a cubic metre as a linear metre is a real
error, and only comparing the two documents finds it.
Also: a conversion takes 30-90 seconds and a frozen page reads as a broken
one, so the run now narrates its stages in a st.status panel.
The marker read 'dev' on the live site, so it was decoration rather than
evidence. A commit SHA is the wrong thing here twice over:
- Coolify does not pass a commit SHA into a Docker Compose build, so the
ARG fell back to 'dev' and would have done so forever.
- A file naming the commit has to live inside that commit, so it can only
ever name the parent. Amending the commit to fix that just moved the
discrepancy.
Replaced with a hash of the .py/.css/.toml actually copied into the image,
computed during the Docker build. It is true by construction, cannot go
stale, and answers the real question: is this the code that was built.
Two build-order bugs found and fixed while proving it (both caught by
building, which is the point):
- Docker's ENV does not evaluate $(cat file)
- the chown referenced a user created later in the file
Verified: build ok, fingerprint d7d427a2e770 in the image, sidebar renders
it, Central Towers converts inside the image (192 written, 23 unplaced, gate
not blocked).
It read 'dev' on the live site, so it was decoration rather than evidence.
Cause: Coolify does not pass a commit SHA into a Docker Compose build, so my
${BUILD_SHA:-dev} fallback always won.
Read from build.txt in the repo instead, which cannot drift from what is
deployed. Two things that took a try each:
- Docker's ENV does not evaluate $(cat file), so the file is copied into the
image and read at startup.
- Streamlit executes app.py in an exec context where __file__ is not
guaranteed; a bare reference to it inside build_sha() would have taken the
whole app down. It now walks a list of candidate paths and never raises.
Verified: docker build ok; build.txt inside the image reads e1f4127;
build_sha() resolves to e1f4127 in the container; the sidebar renders it.
The profile store was write-only: the app saved each consultant's detected
column map and displayed it, but every run re-detected from scratch. So the
original promise - drop their file in and it knows the format - wasn't kept,
and worse, the sidebar could show a stale map that didn't match the file in
front of you.
Now the saved map is applied, with two guards:
- it is only used when the columns it names still exist and still make sense
for THIS file, so a template that changed shape falls back to detection
- 'Re-detect from scratch' in the sidebar overrides it deliberately
The hint path runs the same data verification and description-run derivation as
the fresh path. Without that a warm run placed 194 lines and a cold one placed
192 - both defensible, but a converter that gives two answers to the same pair
of files cannot be trusted on a tender. Cold and warm are now identical on both
sample pairs.
Also: the build SHA is baked in at image build time and shown in the sidebar,
because 'is my deploy current?' is otherwise unanswerable from outside -
Streamlit serves a client-rendered shell, so curl cannot see which commit is
serving.
Automatic coverage is now 203 of 226 on Central Towers (90%), up from 65. The
Unplaced tab lists every remaining line with its five best candidate rows,
ranked and scored, because where a description repeats in the document only a
person can say which occurrence is which. Pick one or leave it out; chosen
lines are written on the next run through the same verification gate.
Two engine fixes found while building it:
- only a real reference may be carried down a blank item cell. The wrapped
header 'ITEM'/'NO' left a literal NO in that column and every row beneath
inherited it, so all of 1200LE rows 73-120 read as a mismatch.
- the '.NN' is not part of a line's identity. The source reuses one number
space for a parent heading and its first child, and prints '.01' twice in a
row; the consultant's template copies that duplication while its group
column stays sequential.
Also fixed: two tab registrations that had silently never applied, and a
NameError in the Verification tab.
Tried and reverted: per-section alignment. Blueberry 1 unmatched -> 14,
Central 23 -> 34. One sheet can span several sections, so splitting is worse.
A converter that writes hundreds of cells into someone else's spreadsheet has
to prove it afterwards, reading the OUTPUT back off disk rather than trusting
what the writer believes it did.
boq_engine/verify.py
- every formula the consultant had is still there (fatal)
- no sheet went missing (fatal)
- every write actually landed in the cell it claims (fatal)
- every source line accounted for (warn)
- nothing replaced without a human decision (warn)
- the workbook's own arithmetic recomputes (warn, optional)
populate() now records every cell it wrote, so the fidelity check reads back
real coordinates instead of sampling a guess.
A failed fatal check removes the download button outright. Recalculation needs
LibreOffice and is skipped cleanly when absent, so the container stays small.
New Verification tab shows the full check list plus a per-sheet table of what
the app decided about the columns - header row, item, description run, unit,
qty, rate, amount - so a wrong assumption is visible rather than invisible.
Also: aggregated the per-sheet TOTAL findings into one; fifty identical cards
is noise.
Second real pair (Central Towers: 20pp Candy PDF into a 53-sheet SANS 1200
workbook) exposed four failures. All fixed and verified against both pairs.
- item numbering: accept 1.1.1, 110...., .01 and rebuild the running counter
Candy restarts under each repeated heading. 0% -> 86% of lines numbered.
- wrapped headers: join 'QUAN-' + 'TITY'. qty was missing on 49 of 51 sheets.
- description is a RUN between ITEM and UNIT, not one column: the header
claims E, the data sits in D, and it moves within a single sheet.
- carry item numbers down blank cells, and score a missing number as neutral
rather than as disagreement.
Mapping: 1/108 unmatched (Blueberry), 33/226 (Central Towers).
Safety: the consultant's workbook is never damaged. Per-write guard plus a
final pass that restores any formula disturbed - 0 of 3633 formulas clobbered.
Writing a literal over a derived quantity would have destroyed their model
while looking like it worked.
Reads our Candy PDF (deterministically, no OCR or model needed) and populates
the consultant's own workbook using their formulas, column layout and PAYMENT
column. Nothing is ever silently overwritten: every disagreement is reported,
can be settled inside the app, and regenerated.
- extract Candy text PDF -> tree (x-position column detection)
- template layout detect, sequence align, write, totals, overrides
- validate qty x rate and children-vs-subtotal reconciliation
- profiles per-consultant layout profiles, stored on a volume
Shipped with a Dockerfile + docker-compose for Coolify.
