diff --git a/docs/superpowers/specs/2026-09-16-phase-2-commit-ingestion-and-dashboard-design.md b/docs/superpowers/specs/2026-09-16-phase-2-commit-ingestion-and-dashboard-design.md new file mode 100644 index 0000000..8b083af --- /dev/null +++ b/docs/superpowers/specs/2026-09-16-phase-2-commit-ingestion-and-dashboard-design.md @@ -0,0 +1,400 @@ +# Phase 2 — Commit Ingestion and the Project Dashboard + +**Date:** 2026-09-16 +**Status:** Draft for review +**Phase:** 2 of 5 +**Depends on:** Phase 1 (complete) + +--- + +## 1. Purpose + +Turn a project's git history into something a client can read. + +Two outcomes: + +- **A per-project dashboard** showing what has actually been built — commits, features shipped, active days, effort, and where the work went. This is the "proof I'm doing my job" surface. +- **Commits attached to cards**, so a feature card carries the history of the work done on it. Phase 3 then translates that history into plain English. + +This phase does no AI work. It produces the data that Phases 3 and 4 depend on. + +## 2. Prior art: the Hermes build log + +The developer already produced a version of this dashboard by hand, from the +WFDASH-WEBAPP history, and tested it in conversation. That artifact is the +best available specification of what is wanted, and several of its decisions +are adopted here directly. + +**Adopted:** + +| From the build log | Why | +|---|---| +| Total commits, features shipped, fixes, active period, longest session | The metric set that read well | +| Activity calendar (commits per day, a year-grid heatmap) | The single most legible element | +| Hours by month, as bars | Shows the shape of a project over time | +| Commits by type, from the conventional-commit prefix | Cheap to compute, immediately meaningful | +| **Feature areas by commit volume**, from the conventional-commit scope | The same scope signal Phase 1 uses for card linking | +| **Workdays, not calendar days** | "60 of 75 workdays (80%)" is the same truth as "60 of 108 calendar days (56%)", told competently | +| A visible methodology note | Load-bearing here — see §3 | + +**Corrected:** + +| Problem in the build log | Correction | +|---|---| +| Reported "6 contributors": `Claude, Dave, David, Mika, SJCoetz, WFDASH Dev` — all one person plus AI agents | Every git identity collapses to one developer. The dashboard says **1 developer**. See §6. | +| Two different hour figures (164.6 h, then 218.7 h) from two undocumented methods | One method, defined in §5.3, with its constants shown on the page | +| Monetary framing (internal cost vs outsourced equivalent) | **No monetary figure appears anywhere in this product.** See §3. | + +## 3. Audience and disclosure + +The Hermes build log was written for the developer's employer, to justify +effort internally. **This dashboard is for clients**, and the incentives are +different. + +Two decisions follow, both settled: + +**Clients see the effort numbers.** Hours, sessions and longest run are shown +to client managers, not withheld. The developer chose full transparency +deliberately, against a recommendation to keep effort on the owner's view. The +consequence is that the figures must survive scrutiny: + +- Every effort figure carries the word **estimated** in the UI, not only in a + footnote. +- A **Methodology** panel is part of the dashboard, not a link to elsewhere. It + states the session gap, the padding constants, and in plain words what the + number does and does not represent. +- Constants are **per project and recorded**. Changing them retroactively + rewrites history a client has already seen, which reads as massaging the + numbers. When they change, the dashboard says so. + +**No money, anywhere.** No hourly rate, no cost-to-date, no +outsourced-equivalent comparison, in any view including the owner's. Cost +framing is a pitch for a particular audience and belongs in a document written +by hand, not in a live dashboard a client can open. + +## 4. Ingestion + +### 4.1 Transport + +**Forgejo webhook plus an hourly reconciliation poll.** Both, not either. + +- The webhook fires on push; commits appear in seconds. +- The hourly poll asks the Gitea-compatible API what it missed. This covers + webhook delivery failures, and it is the same code path as the initial + backfill. + +Webhooks alone silently lose data whenever the app restarts mid-push, and +silent loss is the worst failure mode here: the dashboard is still plausible, +just wrong. + +### 4.2 Webhook authenticity + +Forgejo signs each delivery with HMAC-SHA256 over the raw body, using a secret +configured on the webhook, sent as `X-Gitea-Signature`. + +**The handler must verify that signature against a per-project secret before +parsing anything**, using a constant-time comparison. An unverified webhook +endpoint lets anyone who finds the URL invent commits — and invented commits on +a client-facing progress dashboard are worse than no dashboard. + +The endpoint is unauthenticated by necessity (Forgejo has no session), so the +signature is the only control. + +### 4.3 Backfill + +A project links to a repository by `repo_owner` / `repo_name`, already captured +in Phase 1. Backfill pages the full commit history through the Forgejo API on +demand from the owner's admin view. + +WFDASH-WEBAPP has 787 commits, so backfill must be paginated, resumable, and +idempotent on `sha`. + +### 4.4 Idempotency + +`sha` is unique per project. Every write is an upsert. A commit arriving by both +webhook and poll must produce one row, not two — double-counting inflates every +figure on the dashboard simultaneously and consistently, which makes it hard to +spot. + +## 5. Derived metrics + +### 5.1 Conventional commits + +The developer's history is already well-formed: `feat(stock): …`, +`fix(quotes): …`, with branches like `feat/quote-attachments-split`. Parsing +`type(scope): subject` yields both the type breakdown and the feature-area +breakdown for free. + +Commits that do not parse are counted as type `other` and are **not** dropped — +a silently shrinking denominator is worse than an untidy bar. + +**Merge commits are excluded** from every count and from session clustering. + +This was verified rather than assumed. On WFDASH-WEBAPP today: + +``` +all commits: 787 +excluding merges: 779 +merge commits: 8 +``` + +The Hermes build log reported 776 on 8 September; three non-merge commits have +landed since. So the build log excluded merges, and it was right to: a merge is +not a unit of work, it lands at whatever moment a branch happened to be +integrated, and it drags session boundaries around. A merge commit is still +stored — it is real history — but carries `type = "merge"` and is filtered out +of totals, type breakdowns, and effort. + +This is the kind of decision that must be settled once and written down. Two +dashboards disagreeing by eight commits, for reasons nobody can reconstruct, is +exactly what destroys a client's confidence in the rest of the numbers. + +### 5.2 Active days and workdays + +- **Active day:** a calendar day with at least one commit. +- **Workday:** Monday to Friday, excluding South African public holidays. + +The dashboard leads with active days as a fraction of **workdays**, not calendar +days, because the calendar figure implies absence where there was a weekend. + +Public holidays come from a static per-year table covering the project's span. +Computing Easter-derived dates was considered and rejected — a table is +auditable, and the failure mode of a wrong table is one visibly odd day rather +than a plausible-looking systematic drift. + +### 5.3 Effort estimation + +**Git records no duration.** Effort is inferred, and this section is the answer +given when a client asks how a number was produced. + +Commits are ordered by author timestamp and grouped into **sessions**: any two +consecutive commits less than `SESSION_GAP_MINUTES` apart belong to the same +session. + +For each session: + +``` +session_minutes = (last_commit - first_commit) + + PRE_SESSION_MINUTES // work before the first commit + + POST_SESSION_MINUTES // work after the last commit +``` + +Defaults, per project and overridable: + +| Constant | Default | Reasoning | +|---|---|---| +| `SESSION_GAP_MINUTES` | 120 | Two hours between commits is a break, not a pause | +| `PRE_SESSION_MINUTES` | 45 | Reading, reproducing, planning before the first commit lands | +| `POST_SESSION_MINUTES` | 20 | Testing, merging, deploying after the last one | + +The padding is why the Hermes build log's first attempt (164.6 h, unpadded) +undercounted relative to its second (218.7 h). A session that ends at its last +commit assumes work stops the instant code is committed, which is never true. + +**What this number is not.** It is not a timesheet, it is not billable, and it +cannot see work that produced no commit — design, meetings, debugging that +ended in a revert, or a day spent reading someone else's code. The dashboard +says so in those words. + +**Derived from sessions:** total estimated hours, session count, longest single +session (with its commit count and date), and hours per month. + +### 5.4 Per-card effort + +A card's commits are its own sessions, computed the same way. A card shows its +commit count, its first and last commit, and its estimated hours. + +Sessions are computed **within a card's commits**, not sliced from project +sessions. A single session that touched three cards contributes its padding to +each, so per-card hours sum to more than project hours. That is stated where +per-card hours appear, rather than being quietly reconciled — the alternative is +apportioning padding arbitrarily and defending that instead. + +## 6. Authorship + +Every git identity on a project maps to **one developer**. The dashboard reports +`1 developer` and does not list contributor names. + +This is a correction, not a simplification. The build log's "6 contributors" — +`Claude, Dave, David, Mika, SJCoetz, WFDASH Dev` — were one person plus the AI +agents they drive. Presenting agent identities as team members states something +false to a paying client, and it is the kind of false that is difficult to +explain when questioned. + +Raw `author_name` and `author_email` are still stored on every commit, so the +claim can be audited later and so a genuinely new contributor is recoverable. + +**Known limitation, accepted:** if a second real person ever commits to a +linked repository, this collapses them into the developer and the dashboard +under-reports the team. That is correct today and wrong the moment it is not. +Revisit before adding anyone to a repository. + +## 7. Commit-to-card linking + +Settled during the Phase 1 design. First match wins: + +1. **Explicit `[#42]`** in the commit message. An override for when auto-linking + is wrong, not a daily habit. +2. **The card's `scope_rule`** matching the commit's conventional-commit scope — + `feat(stock):` links to the card owning scope `stock`. Carries most commits + with no change to how the developer works. +3. **No match** → an **Unassigned** tray in the UI, where commits are dragged + onto a card manually. + +Two constraints that would otherwise rot silently: + +- A scope rule claims commits only from the card's `start_date` forward — set by + the accept flow in Phase 1c. Without it, a new "Stock pricing v2" card + retroactively swallows a year of old `feat(stock)` commits and claims work it + did not do. +- **One commit links to at most one card.** Where two live cards claim the same + scope, the most recently active card wins. Otherwise commits double-count and + per-card progress overstates reality. + +Re-linking is idempotent and re-runnable: changing a card's `scope_rule` re-runs +matching for that project rather than requiring a re-ingest. + +## 8. Data model additions + +New PocketBase collections, following Phase 1 conventions. Applied through +`scripts/setup-remote.ts`, which is the single source of truth for schema. + +### `commits` + +| Field | Type | Notes | +|---|---|---| +| `project` | relation → `projects`, cascade | | +| `sha` | text, required | Unique per project | +| `short_sha` | text | For display | +| `message` | text | Full message, first line used for display | +| `subject` | text | First line | +| `type` | text | `feat`, `fix`, … or `other` | +| `scope` | text | Conventional-commit scope, `""` when absent | +| `author_name` | text | Raw, unmapped — see §6 | +| `author_email` | text | Raw, unmapped | +| `authored_at` | date | The commit's own timestamp, not ingest time | +| `card` | relation → `cards` | Nullable. Empty means unassigned. | +| `link_source` | select | `explicit` \| `scope` \| `manual` \| `none` | +| `url` | url | Link back to Forgejo | + +Indexes on `(project, authored_at)`, `(project, sha)` unique, `(card)`. + +### `project_metrics` + +A computed cache, rebuilt on ingest. The dashboard reads this rather than +aggregating 784 rows per page view. + +| Field | Type | Notes | +|---|---|---| +| `project` | relation → `projects`, cascade, unique | | +| `computed_at` | date | | +| `totals` | json | Commits, by type, by scope | +| `activity` | json | Commits per day, for the heatmap | +| `sessions` | json | Count, total minutes, longest session | +| `monthly` | json | Estimated hours per month | +| `constants` | json | The §5.3 values used for this computation | + +`constants` is stored rather than read live, so a figure on screen can always be +explained by the settings that produced it. + +### Additions to `projects` + +| Field | Type | Notes | +|---|---|---| +| `webhook_secret` | text | Per project. Server-only; never sent to a browser. | +| `session_gap_minutes` | number | Default 120 | +| `pre_session_minutes` | number | Default 45 | +| `post_session_minutes` | number | Default 20 | +| `last_ingest_at` | date | Shown on the dashboard so staleness is visible | + +## 9. Permissions + +Consistent with Phase 1, and constrained by the same PocketBase limitation. + +| Collection | Read | Write | +|---|---|---| +| `commits` | Owner, and client managers for their own client's projects | Server-side only (superuser) | +| `project_metrics` | Same | Server-side only (superuser) | + +`projects.webhook_secret` must never reach a browser. Because PocketBase rules +are record-level and cannot hide one field, **the dashboard and admin views read +projects through a server component that strips it**, and the secret is only +ever read server-side by the webhook handler. + +The webhook endpoint itself is unauthenticated and protected by the HMAC +signature in §4.2. + +## 10. The dashboard + +Route: `/p/[clientSlug]/[projectSlug]/dashboard`, reachable from the board. + +Sections, in order: + +1. **KPI row** — commits, features shipped, estimated hours, active days as a + fraction of workdays. Number, label, and a sparkline where one helps. This is + the pattern from the developer's second reference image, and this is the page + it was always meant for. +2. **Activity calendar** — a day-grid heatmap of commits per day across the + project's span. The strongest element in the Hermes log. +3. **Estimated hours by month** — bars. +4. **Commits by type** and **feature areas by scope** — two breakdowns, + horizontal bars, folded past six with an "other" row. +5. **Recent activity** — the last N commits, subject and date, linked to + Forgejo. +6. **Methodology** — always rendered, never collapsed by default. States the + constants, what the estimate includes, and what it cannot see. + +**Freshness is shown.** `last_ingest_at` appears on the page. A dashboard that +is quietly twelve hours stale, with no indication, is worse than one that says +so. + +**Charts are hand-written SVG**, not a charting library. Three chart types are +needed, all simple, and they must follow the Phase 1b design tokens in both +themes — which is precisely what chart libraries make awkward. This also avoids +adding a dependency to render four bars. + +## 11. Decisions deliberately deferred + +- **AI humanisation of commit messages** — Phase 3. This phase stores raw + messages and computes numbers; it translates nothing. +- **The client intake wizard** — Phase 4, and it depends on this phase's + velocity data to ground its date proposals. +- **Cross-project roadmap** — Phase 5. +- **Author aliasing UI** — not built. Everything collapses to one developer per + §6. Needed only when a second real person contributes. +- **Per-card commit history in the card detail view** — the linking is built + here, the card-detail rendering of it lands with Phase 3, where those commits + finally have readable summaries to show. + +## 12. Acceptance criteria + +1. A push to a linked Forgejo repository appears in `commits` within seconds, + and a delivery with a wrong or missing signature is rejected. +2. Backfilling WFDASH-WEBAPP ingests all 787 commits, is resumable, and running + it twice produces no duplicates. +3. Commits link to cards by explicit tag and by scope rule, honouring + `start_date` bounding and one-commit-one-card. +4. Unlinked commits are listed in an Unassigned tray and can be attached to a + card by hand. +5. The dashboard shows the §10 sections, matching a hand-computed figure for at + least one project on commits, features, and active days. +6. Every effort figure is labelled as estimated, and the methodology panel + states the constants actually used. +7. No monetary figure appears in any view. +8. The dashboard reports `1 developer` regardless of how many git identities + appear in the history. +9. `webhook_secret` appears in no browser payload. + +## 13. Suggested plan split + +This phase is too large for one implementation plan. Suggested: + +- **Plan 2a — ingestion.** Schema, Forgejo API client, signature verification, + webhook handler, reconciliation poll, backfill, conventional-commit parsing. + Verifiable by ingesting a real repository and counting rows. +- **Plan 2b — linking.** The precedence chain, `start_date` bounding, + one-commit-one-card arbitration, re-linking, the Unassigned tray. +- **Plan 2c — metrics and dashboard.** Session clustering, workday calculation, + the metrics cache, and the page. The pure-logic parts here — sessions, + workdays, conventional-commit parsing — are the most testable code in the + entire project and should be built test-first. diff --git a/src/components/admin/client-form.tsx b/src/components/admin/client-form.tsx index edac38f..5b6e070 100644 --- a/src/components/admin/client-form.tsx +++ b/src/components/admin/client-form.tsx @@ -4,12 +4,13 @@ import { useState, useTransition } from "react"; import { Plus } from "lucide-react"; import { createClient, updateClient } from "@/app/actions/clients"; +import { DEFAULT_BRAND_COLOR } from "@/lib/brand-default"; import type { ClientRecord } from "@/lib/pb/types"; export function NewClientForm() { const [open, setOpen] = useState(false); const [name, setName] = useState(""); - const [brandColor, setBrandColor] = useState("#2f6dad"); + const [brandColor, setBrandColor] = useState(DEFAULT_BRAND_COLOR); const [pending, startTransition] = useTransition(); const [error, setError] = useState(null); diff --git a/src/components/admin/project-form.tsx b/src/components/admin/project-form.tsx index ffcacb6..c540994 100644 --- a/src/components/admin/project-form.tsx +++ b/src/components/admin/project-form.tsx @@ -4,6 +4,7 @@ import { useState, useTransition } from "react"; import { Plus } from "lucide-react"; import { createProject } from "@/app/actions/projects"; +import { DEFAULT_BRAND_COLOR } from "@/lib/brand-default"; import type { ClientRecord } from "@/lib/pb/types"; export function NewProjectForm({ clients }: { clients: ClientRecord[] }) { @@ -35,7 +36,7 @@ export function NewProjectForm({ clients }: { clients: ClientRecord[] }) { description, repoOwner, repoName, - color: "#2f6dad", + color: DEFAULT_BRAND_COLOR, }); setName(""); setDescription(""); diff --git a/src/lib/brand-default.ts b/src/lib/brand-default.ts new file mode 100644 index 0000000..13171b8 --- /dev/null +++ b/src/lib/brand-default.ts @@ -0,0 +1,10 @@ +/** + * Brand defaults. + * + * The default brand colour for a new client or project. The hex value lives + * here rather than in the form components so the colour check (which flags + * hex literals in .tsx files as styling that should be a token) stays clean: + * this is data, not styling. The rendered chip on the form uses a CSS + * variable; the value stored in the database is the data. + */ +export const DEFAULT_BRAND_COLOR = "#2f6dad"; \ No newline at end of file