The MEGA CHECK sheet was meant to answer "which section do I look at". It
was quietly failing to, in four separate ways, each of which made the file
look more finished than it was.
A line with no row to go to was dropped on the floor. reconcile() collected
it, tagged it with sheet "-", and then only attached rows to a section whose
SHEET matched - so "-" matched nothing and the line vanished. It was the one
thing most worth telling a person, and it was the only thing not shown. Every
unplaced line now appears with its qty, unit and rate.
Sections were paired on a bare number, and Candy restarts its numbering inside
every schedule. Blueberry has a "1" in Schedule A, another "1" under Schedule
B and a "4" under a third. Keying on that merged unrelated sections: 6 of
Blueberry's 8 matched nothing, and the sheet showed 2 clean rows that read as
"this book is checked". Now keyed on the section title - 8 of 8 and 18 of 18.
"WORKBOOK SAYS" summed the whole sheet. Blueberry's SCHEDULE B holds sections
1 to 6, so "1 SITE CLEARANCE" was reported as R4,422,018 against a printed
R160,057.99 - the right section with a figure that is not that section's.
Each section is now summed from its own rows only, and a row that cannot be
attributed counts for nothing rather than for every section at once. That
last one had 141, 142 and 143 all reporting the same R100,722.77.
Untested sections are now stated as untested. A table of clean sections implies
the rest balance, so the sheet says plainly which ones could not be matched.
A section the PDF prints as 0.00 is a fill-in for the tenderer, not a
disagreement. Blueberry's "4 CABLE DUCTS" is twelve lines that all carry
qty 0.0, and the consultant then fills in their own quantities; reporting that
as a R348,746.25 difference points at our work when the source simply has
nothing to compare against. It now reads "no figure in PDF".
"every write landed" only checked that a cell was not empty, then stopped
after 600. A rate overwritten with a wrong number passed silently - proved by
planting one and watching the gate stay green. It now compares every value,
and the 600-cell cap is gone: a check that covers part of the file and reports
as if it covered all of it is worse than no check.
Damage still blocks the download. Incomplete or misplaced work is handed over
with the punch list, and needs_review now fires on unplaced lines too - it
used to stay quiet on Blueberry, which had one.
Verified by fault injection, not by reading the code: corrupting a written
cell is now caught, two rates swapped between rows are flagged without
blocking, and no unplaced line is ever dropped. All four behave.
Asked what the 160 items under Disagreements were. Almost none were
disagreements, and a panel that cries wolf at 160 teaches people to scroll
past it - worse than having no panel at all.
147 of the 160 were the formula guard firing once per row. The consultant's
amount column is a formula on nearly every line, so 'I protected your model'
was being reported as 'a person must choose'. It is now its own quiet note and
cannot crowd out findings that genuinely need a decision.
12 more were unit formatting - 'PC Sum' vs 'PCSum' vs 'PC.Sum' is one unit
written three ways. Unit comparison now collapses case, spacing and dots, so
only genuinely different units are flagged.
160 -> 19, and all 19 are real. The ones worth a human's eye: seven quantity
disagreements (template 47.5 vs source 95.0) and eleven unit mismatches, five
of which are 'm' vs 'm3' - pricing a cubic metre as a linear metre is a real
error, and only comparing the two documents finds it.
Also: a conversion takes 30-90 seconds and a frozen page reads as a broken
one, so the run now narrates its stages in a st.status panel.
The marker read 'dev' on the live site, so it was decoration rather than
evidence. A commit SHA is the wrong thing here twice over:
- Coolify does not pass a commit SHA into a Docker Compose build, so the
ARG fell back to 'dev' and would have done so forever.
- A file naming the commit has to live inside that commit, so it can only
ever name the parent. Amending the commit to fix that just moved the
discrepancy.
Replaced with a hash of the .py/.css/.toml actually copied into the image,
computed during the Docker build. It is true by construction, cannot go
stale, and answers the real question: is this the code that was built.
Two build-order bugs found and fixed while proving it (both caught by
building, which is the point):
- Docker's ENV does not evaluate $(cat file)
- the chown referenced a user created later in the file
Verified: build ok, fingerprint d7d427a2e770 in the image, sidebar renders
it, Central Towers converts inside the image (192 written, 23 unplaced, gate
not blocked).
It read 'dev' on the live site, so it was decoration rather than evidence.
Cause: Coolify does not pass a commit SHA into a Docker Compose build, so my
${BUILD_SHA:-dev} fallback always won.
Read from build.txt in the repo instead, which cannot drift from what is
deployed. Two things that took a try each:
- Docker's ENV does not evaluate $(cat file), so the file is copied into the
image and read at startup.
- Streamlit executes app.py in an exec context where __file__ is not
guaranteed; a bare reference to it inside build_sha() would have taken the
whole app down. It now walks a list of candidate paths and never raises.
Verified: docker build ok; build.txt inside the image reads e1f4127;
build_sha() resolves to e1f4127 in the container; the sidebar renders it.
The profile store was write-only: the app saved each consultant's detected
column map and displayed it, but every run re-detected from scratch. So the
original promise - drop their file in and it knows the format - wasn't kept,
and worse, the sidebar could show a stale map that didn't match the file in
front of you.
Now the saved map is applied, with two guards:
- it is only used when the columns it names still exist and still make sense
for THIS file, so a template that changed shape falls back to detection
- 'Re-detect from scratch' in the sidebar overrides it deliberately
The hint path runs the same data verification and description-run derivation as
the fresh path. Without that a warm run placed 194 lines and a cold one placed
192 - both defensible, but a converter that gives two answers to the same pair
of files cannot be trusted on a tender. Cold and warm are now identical on both
sample pairs.
Also: the build SHA is baked in at image build time and shown in the sidebar,
because 'is my deploy current?' is otherwise unanswerable from outside -
Streamlit serves a client-rendered shell, so curl cannot see which commit is
serving.
Automatic coverage is now 203 of 226 on Central Towers (90%), up from 65. The
Unplaced tab lists every remaining line with its five best candidate rows,
ranked and scored, because where a description repeats in the document only a
person can say which occurrence is which. Pick one or leave it out; chosen
lines are written on the next run through the same verification gate.
Two engine fixes found while building it:
- only a real reference may be carried down a blank item cell. The wrapped
header 'ITEM'/'NO' left a literal NO in that column and every row beneath
inherited it, so all of 1200LE rows 73-120 read as a mismatch.
- the '.NN' is not part of a line's identity. The source reuses one number
space for a parent heading and its first child, and prints '.01' twice in a
row; the consultant's template copies that duplication while its group
column stays sequential.
Also fixed: two tab registrations that had silently never applied, and a
NameError in the Verification tab.
Tried and reverted: per-section alignment. Blueberry 1 unmatched -> 14,
Central 23 -> 34. One sheet can span several sections, so splitting is worse.
A converter that writes hundreds of cells into someone else's spreadsheet has
to prove it afterwards, reading the OUTPUT back off disk rather than trusting
what the writer believes it did.
boq_engine/verify.py
- every formula the consultant had is still there (fatal)
- no sheet went missing (fatal)
- every write actually landed in the cell it claims (fatal)
- every source line accounted for (warn)
- nothing replaced without a human decision (warn)
- the workbook's own arithmetic recomputes (warn, optional)
populate() now records every cell it wrote, so the fidelity check reads back
real coordinates instead of sampling a guess.
A failed fatal check removes the download button outright. Recalculation needs
LibreOffice and is skipped cleanly when absent, so the container stays small.
New Verification tab shows the full check list plus a per-sheet table of what
the app decided about the columns - header row, item, description run, unit,
qty, rate, amount - so a wrong assumption is visible rather than invisible.
Also: aggregated the per-sheet TOTAL findings into one; fifty identical cards
is noise.
Reads our Candy PDF (deterministically, no OCR or model needed) and populates
the consultant's own workbook using their formulas, column layout and PAYMENT
column. Nothing is ever silently overwritten: every disagreement is reported,
can be settled inside the app, and regenerated.
- extract Candy text PDF -> tree (x-position column detection)
- template layout detect, sequence align, write, totals, overrides
- validate qty x rate and children-vs-subtotal reconciliation
- profiles per-consultant layout profiles, stored on a volume
Shipped with a Dockerfile + docker-compose for Coolify.
