Dave 44bcc81d97 feat(ai): provider-neutral completion interface

2026-09-16 15:50:39 +02:00

33 KiB

Raw Blame History

Phase 3b — Commit One-Liners Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Give every non-merge commit a one-sentence plain-English translation, generated in batches, stored against the commit, versioned against the prompt that produced it, and never allowed to take the rest of the product down when the provider fails.

Architecture: Pure, exhaustively tested functions build the batch prompt and parse the response; a service layer applies them. Responses are matched back to commits by an explicit index, never by position, so a short or reordered response leaves commits unsummarised rather than mislabelled. Summaries are an enhancement over data that is already correct — every failure path falls back to the raw commit subject.

Tech Stack: Next.js 16 Server Actions · PocketBase 0.30 · vitest

Spec: docs/superpowers/specs/2026-09-16-phase-3-ai-humanisation-design.md §5.2, §6, §7

Prerequisite: Plan 3a complete — the connection test returns a sensible sentence.

The one rule that makes this safe

A batch sends 25 commits and expects 25 sentences back. Models do not always oblige — they skip one, merge two, or add a preamble.

If the response is matched to commits by position, a single missing line shifts every subsequent summary onto the wrong commit. Nothing errors. The dashboard fills with confident, fluent, wrong sentences, each attached to a commit it does not describe, and a client reads them.

So the prompt numbers the input and requires numbered output, and the parser matches on that number. A commit whose number is absent from the response gets no summary and is retried on the next run. Missing is recoverable; wrong is not, because nobody will ever notice.

Task 2 exists almost entirely to make this property testable.

File structure

src/
├── lib/
│   └── ai/
│       ├── batch.ts                  # chunking + prompt + parsing
│       ├── batch.test.ts
│       └── summarise.ts              # the service
├── app/
│   └── actions/
│       └── summarise.ts              # owner: run, retry failed
└── components/
    └── admin/summary-panel.tsx

Task 1: Schema

Files:

Modify: scripts/setup-remote.ts
Modify: src/lib/pb/types.ts
Step 1: Add the fields

In scripts/setup-remote.ts, add a helper beside the other ensure* functions:

async function ensureCommitSummaryFields(pb: PocketBase): Promise<void> {
  const commits = await pb.collections.getOne("commits");
  const have = new Set(commits.fields.map((f: { name: string }) => f.name));

  const additions = [
    { name: "summary", type: "text", required: false, max: 500 },
    { name: "summary_version", type: "number", required: false, onlyInt: true },
    // Counts consecutive failures for this commit. A message that trips the
    // provider every time must not be retried forever on every ingest.
    { name: "summary_attempts", type: "number", required: false, onlyInt: true },
  ];

  const fields = [...(commits.fields as unknown[])];
  let changed = false;

  for (const field of additions) {
    if (!have.has(field.name)) {
      fields.push({ ...field, presentable: false, hidden: false, system: false });
      changed = true;
    }
  }

  if (changed) {
    await pb.collections.update(commits.id, { fields });
    console.log("  + commits fields: summary, summary_version, summary_attempts");
  } else {
    console.log("  = commit summary fields already present");
  }
}

Call it after the commits collection is created.

Step 2: Extend the record type

In src/lib/pb/types.ts, add to CommitRecord:

  /** Plain-English one-liner. Empty when not yet generated or generation failed. */
  summary: string;
  /** PROMPT_VERSION this summary was written under. */
  summary_version: number;
  /** Consecutive failed attempts. Capped so a poison message stops retrying. */
  summary_attempts: number;

Step 3: Apply and verify

set -a && source .env.local && set +a && npm run setup-remote && npm run build

Expected: the script reports adding the three fields; the build succeeds.

Step 4: Commit

git add scripts/setup-remote.ts src/lib/pb/types.ts
git commit -m "feat(schema): commit summary fields"

Task 2: Batching, prompting and parsing

The safety-critical task. Pure, test-first.

Files:

Create: src/lib/ai/batch.ts
Test: src/lib/ai/batch.test.ts
Step 1: Write the failing tests

src/lib/ai/batch.test.ts:

import { describe, it, expect } from "vitest";
import {
  BATCH_SIZE,
  buildBatchPrompt,
  chunk,
  parseBatchResponse,
  type SummarisableCommit,
} from "@/lib/ai/batch";

function commit(n: number, subject = `subject ${n}`): SummarisableCommit {
  return { id: `c${n}`, subject, type: "feat", scope: "stock" };
}

describe("chunk", () => {
  it("returns nothing for an empty list", () => {
    expect(chunk([], 3)).toEqual([]);
  });

  it("splits into groups of the given size", () => {
    expect(chunk([1, 2, 3, 4, 5], 2)).toEqual([[1, 2], [3, 4], [5]]);
  });

  it("returns one group when everything fits", () => {
    expect(chunk([1, 2], 10)).toEqual([[1, 2]]);
  });

  it("uses a sane default batch size", () => {
    expect(BATCH_SIZE).toBeGreaterThan(1);
    expect(BATCH_SIZE).toBeLessThanOrEqual(50);
  });
});

describe("buildBatchPrompt", () => {
  it("numbers every commit from 1", () => {
    const prompt = buildBatchPrompt([commit(1), commit(2)]);
    expect(prompt).toContain("1. subject 1");
    expect(prompt).toContain("2. subject 2");
  });

  it("asks for the same numbering back", () => {
    const prompt = buildBatchPrompt([commit(1)]);
    expect(prompt.toLowerCase()).toContain("number");
  });

  it("includes the scope, which carries meaning the subject may not", () => {
    const prompt = buildBatchPrompt([commit(1)]);
    expect(prompt).toContain("stock");
  });

  it("flattens newlines in a subject so numbering cannot be faked", () => {
    // A subject containing "\n2. " would otherwise inject a fake entry and
    // shift the mapping — the exact failure the numbering exists to prevent.
    const prompt = buildBatchPrompt([commit(1, "real\n2. injected")]);
    const numbered = prompt.split("\n").filter((l) => /^\d+\.\s/.test(l));
    expect(numbered).toHaveLength(1);
  });
});

describe("parseBatchResponse", () => {
  const batch = [commit(1), commit(2), commit(3)];

  it("maps numbered lines back to the right commits", () => {
    const result = parseBatchResponse(
      "1. Did the first thing.\n2. Did the second thing.\n3. Did the third thing.",
      batch
    );
    expect(result.get("c1")).toBe("Did the first thing.");
    expect(result.get("c2")).toBe("Did the second thing.");
    expect(result.get("c3")).toBe("Did the third thing.");
  });

  it("LEAVES A COMMIT UNSUMMARISED rather than shifting the rest", () => {
    // The property this whole design exists for. Positional parsing would put
    // "Did the third thing." on commit 2 — fluent, confident, and wrong about
    // work a client is paying for. Missing is recoverable; wrong is not.
    const result = parseBatchResponse(
      "1. Did the first thing.\n3. Did the third thing.",
      batch
    );
    expect(result.get("c1")).toBe("Did the first thing.");
    expect(result.has("c2")).toBe(false);
    expect(result.get("c3")).toBe("Did the third thing.");
  });

  it("ignores a number outside the batch", () => {
    const result = parseBatchResponse("1. One.\n9. Nine.", batch);
    expect(result.size).toBe(1);
    expect(result.get("c1")).toBe("One.");
  });

  it("ignores a conversational preamble", () => {
    const result = parseBatchResponse(
      "Here are the summaries:\n\n1. One.\n2. Two.",
      batch
    );
    expect(result.get("c1")).toBe("One.");
    expect(result.get("c2")).toBe("Two.");
  });

  it("accepts ')' as well as '.' after the number", () => {
    expect(parseBatchResponse("1) One.", batch).get("c1")).toBe("One.");
  });

  it("keeps the first line when a number repeats", () => {
    const result = parseBatchResponse("1. First.\n1. Second.", batch);
    expect(result.get("c1")).toBe("First.");
  });

  it("returns nothing for an empty or unparseable response", () => {
    expect(parseBatchResponse("", batch).size).toBe(0);
    expect(parseBatchResponse("I cannot help with that.", batch).size).toBe(0);
  });

  it("trims and caps an over-long sentence", () => {
    const result = parseBatchResponse(`1.    ${"x".repeat(900)}`, batch);
    expect(result.get("c1")!.length).toBeLessThanOrEqual(400);
  });

  it("drops an empty sentence rather than storing a blank summary", () => {
    // A stored empty string reads as "generated, nothing to say" and would
    // never be retried.
    const result = parseBatchResponse("1.   \n2. Two.", batch);
    expect(result.has("c1")).toBe(false);
    expect(result.get("c2")).toBe("Two.");
  });
});

Step 2: Run the tests to verify they fail

npm test

Expected: FAIL — Failed to resolve import "@/lib/ai/batch".

Step 3: Write the implementation

src/lib/ai/batch.ts:

/**
 * Batching commits into one request, and getting the answers back onto the
 * right commits.
 *
 * The parsing here is the safety-critical part of this phase. See the header
 * of the Phase 3b plan: matching by position instead of by number turns one
 * skipped line into a dashboard full of fluent sentences attached to the wrong
 * commits, with nothing raising an error and nobody noticing.
 */

/** Commits per request. Large enough to amortise the system prompt, small
 * enough that one bad batch costs little and stays well inside output limits. */
export const BATCH_SIZE = 25;

/** Longer than this is not a one-liner; the model has ignored the instruction. */
const MAX_SUMMARY_LENGTH = 400;

export interface SummarisableCommit {
  id: string;
  subject: string;
  type: string;
  scope: string;
}

export function chunk<T>(items: T[], size: number = BATCH_SIZE): T[][] {
  const out: T[][] = [];
  for (let i = 0; i < items.length; i += size) {
    out.push(items.slice(i, i + size));
  }
  return out;
}

/**
 * Collapse whitespace so one commit occupies exactly one numbered line.
 *
 * A subject containing a newline followed by "2. " would otherwise look like a
 * second entry, shifting every mapping after it — the precise failure the
 * numbering prevents, reintroduced through the input.
 */
function flatten(value: string): string {
  return (value ?? "").replace(/\s+/g, " ").trim();
}

export function buildBatchPrompt(commits: SummarisableCommit[]): string {
  const lines = commits.map((commit, index) => {
    const area = commit.scope ? ` [area: ${flatten(commit.scope)}]` : "";
    return `${index + 1}. ${flatten(commit.subject)}${area}`;
  });

  return `Translate each numbered commit below into one plain-English sentence for the client.

Reply with one numbered line per commit, using the same numbers. No preamble, no blank lines, no extra commentary. If a commit is unclear, describe it vaguely rather than guessing.

Commits:
${lines.join("\n")}`;
}

const NUMBERED_LINE = /^\s*(\d+)[.)]\s*(.*)$/;

/**
 * Map a response back onto commits by the number the model echoed.
 *
 * A commit whose number is absent simply is not in the returned map. The caller
 * leaves it unsummarised and retries later — never fills the gap by position.
 */
export function parseBatchResponse(
  response: string,
  commits: SummarisableCommit[]
): Map<string, string> {
  const out = new Map<string, string>();

  for (const line of (response ?? "").split("\n")) {
    const match = NUMBERED_LINE.exec(line);
    if (!match) continue;

    const index = Number(match[1]) - 1;
    const commit = commits[index];
    if (!commit) continue;

    // First line wins if a number repeats: later duplicates are usually the
    // model correcting itself mid-list, and picking arbitrarily is worse than
    // picking consistently.
    if (out.has(commit.id)) continue;

    const text = match[2].trim().slice(0, MAX_SUMMARY_LENGTH);
    // An empty summary would store as "generated, nothing to say" and never be
    // retried. Leave it absent instead.
    if (!text) continue;

    out.set(commit.id, text);
  }

  return out;
}

Step 4: Run the tests to verify they pass

npm test

Expected: PASS — 18 tests in batch.test.ts.

Step 5: Commit

git add src/lib/ai/batch.ts src/lib/ai/batch.test.ts
git commit -m "feat(ai): index-matched batch prompting and parsing"

Task 3: The summarisation service

Files:

Create: src/lib/ai/summarise.ts
Step 1: Write the service

src/lib/ai/summarise.ts:

import "server-only";

import { getAdminPb } from "@/lib/pb/admin";
import { getProvider, getSystemPrompt, AiError } from "@/lib/ai";
import { PROMPT_VERSION } from "@/lib/ai/preamble";
import {
  BATCH_SIZE,
  buildBatchPrompt,
  chunk,
  parseBatchResponse,
  type SummarisableCommit,
} from "@/lib/ai/batch";
import { aiConfigured } from "@/lib/ai/settings";
import type { CommitRecord } from "@/lib/pb/types";

/** After this many consecutive failures, stop retrying a commit on every run. */
const MAX_ATTEMPTS = 3;

export interface SummariseResult {
  considered: number;
  summarised: number;
  failed: number;
  skipped: boolean;
  error?: string;
}

/**
 * Which commits still need a one-liner.
 *
 * Merges are excluded — they are not units of work and carry no meaning worth
 * translating. A commit is pending when it has no summary, or its summary was
 * written under an older PROMPT_VERSION, and it has not already failed
 * MAX_ATTEMPTS times.
 */
async function pendingCommits(projectId: string): Promise<CommitRecord[]> {
  const pb = await getAdminPb();

  return pb.collection("commits").getFullList<CommitRecord>({
    filter: [
      `project = "${projectId}"`,
      "is_merge = false",
      `(summary = "" || summary_version < ${PROMPT_VERSION})`,
      `summary_attempts < ${MAX_ATTEMPTS}`,
    ].join(" && "),
    sort: "-authored_at",
    fields: "id,subject,type,scope,summary_attempts",
  });
}

/**
 * Generate one-liners for a project's pending commits.
 *
 * Never throws. Summaries are an enhancement over data that is already correct
 * and already displayed — the dashboard, the board and every card must keep
 * working when the provider is misconfigured, down, or rate limited. Callers
 * read the returned counts; they do not catch.
 */
export async function summariseProject(projectId: string): Promise<SummariseResult> {
  if (!(await aiConfigured())) {
    return { considered: 0, summarised: 0, failed: 0, skipped: true };
  }

  const pending = await pendingCommits(projectId);
  if (pending.length === 0) {
    return { considered: 0, summarised: 0, failed: 0, skipped: false };
  }

  const pb = await getAdminPb();
  let provider;
  let system: string;

  try {
    provider = await getProvider();
    system = await getSystemPrompt();
  } catch (error) {
    return {
      considered: pending.length,
      summarised: 0,
      failed: 0,
      skipped: true,
      error: error instanceof AiError ? error.message : "AI is not configured",
    };
  }

  let summarised = 0;
  let failed = 0;
  let lastError: string | undefined;

  for (const batch of chunk(pending, BATCH_SIZE)) {
    const items: SummarisableCommit[] = batch.map((c) => ({
      id: c.id,
      subject: c.subject || "",
      type: c.type || "other",
      scope: c.scope || "",
    }));

    let text: string;

    try {
      const result = await provider.complete({
        system,
        user: buildBatchPrompt(items),
        // Roughly 40 tokens per sentence plus headroom for thinking.
        maxTokens: 4096,
      });
      text = result.text;
    } catch (error) {
      lastError = error instanceof AiError ? error.message : "Request failed";
      // Count the attempt against every commit in the batch, so a message that
      // reliably breaks the provider eventually stops being retried forever.
      for (const commit of batch) {
        await pb
          .collection("commits")
          .update(commit.id, { summary_attempts: (commit.summary_attempts ?? 0) + 1 })
          .catch(() => {});
      }
      failed += batch.length;

      // A non-retryable failure — a bad key, a refusal — will fail identically
      // for every remaining batch. Stop rather than burning through them.
      if (error instanceof AiError && !error.retryable) break;
      continue;
    }

    const summaries = parseBatchResponse(text, items);

    for (const commit of batch) {
      const summary = summaries.get(commit.id);

      if (!summary) {
        // Absent from the response. Left unsummarised deliberately — never
        // filled in from a neighbouring line.
        await pb
          .collection("commits")
          .update(commit.id, { summary_attempts: (commit.summary_attempts ?? 0) + 1 })
          .catch(() => {});
        failed += 1;
        continue;
      }

      await pb.collection("commits").update(commit.id, {
        summary,
        summary_version: PROMPT_VERSION,
        summary_attempts: 0,
      });
      summarised += 1;
    }
  }

  return {
    considered: pending.length,
    summarised,
    failed,
    skipped: false,
    error: lastError,
  };
}

/** Counts for the owner's admin panel. Silent degradation must be visible. */
export async function summaryStatus(projectId: string): Promise<{
  total: number;
  summarised: number;
  stale: number;
  failing: number;
}> {
  const pb = await getAdminPb();

  const count = async (filter: string) =>
    (await pb.collection("commits").getList(1, 1, { filter, fields: "id" })).totalItems;

  const base = `project = "${projectId}" && is_merge = false`;

  return {
    total: await count(base),
    summarised: await count(`${base} && summary != "" && summary_version >= ${PROMPT_VERSION}`),
    stale: await count(`${base} && summary != "" && summary_version < ${PROMPT_VERSION}`),
    failing: await count(`${base} && summary_attempts >= ${MAX_ATTEMPTS}`),
  };
}

/** Clear the failure counters so capped commits are retried once more. */
export async function resetSummaryFailures(projectId: string): Promise<number> {
  const pb = await getAdminPb();

  const stuck = await pb.collection("commits").getFullList<CommitRecord>({
    filter: `project = "${projectId}" && summary_attempts >= ${MAX_ATTEMPTS}`,
    fields: "id",
  });

  for (const commit of stuck) {
    await pb.collection("commits").update(commit.id, { summary_attempts: 0 });
  }

  return stuck.length;
}

Step 2: Verify the build

npm run build

Expected: build succeeds.

Step 3: Commit

git add src/lib/ai/summarise.ts
git commit -m "feat(ai): commit summarisation service that never throws"

Task 4: Running it on ingest

Files:

Modify: src/lib/git/ingest.ts
Step 1: Hook it in

In src/lib/git/ingest.ts, after the recomputeMetrics call added in Plan 2c:

  // Summaries are an enhancement, not part of ingestion. Commits are already
  // stored and the metrics are already correct by this point, so a provider
  // outage here must not fail the webhook or lose a push.
  try {
    const { summariseProject } = await import("@/lib/ai/summarise");
    await summariseProject(project.id);
  } catch (error) {
    console.error(`[ingest] summarisation failed for ${project.id}`, error);
  }

The ordering is deliberate: store → link → metrics → summarise. Each step depends only on the ones before it, so a failure late in the chain leaves everything earlier intact and correct.

summariseProject does not throw by design; the try is belt and braces against a future change to that contract.

Step 2: Verify the build

npm run build

Expected: build succeeds.

Step 3: Commit

git add src/lib/git/ingest.ts
git commit -m "feat(ingest): summarise new commits after metrics"

Task 5: The owner's summary panel

Files:

Create: src/app/actions/summarise.ts
Create: src/components/admin/summary-panel.tsx
Modify: src/app/admin/projects/[projectId]/page.tsx
Step 1: Write the actions

src/app/actions/summarise.ts:

"use server";

import { revalidatePath } from "next/cache";

import { requireSession } from "@/lib/auth/session";
import {
  resetSummaryFailures,
  summariseProject,
  summaryStatus,
} from "@/lib/ai/summarise";

/** See the security note at the top of src/app/actions/cards.ts. */
async function requireOwner() {
  const session = await requireSession();
  if (session.role !== "owner") throw new Error("Not found");
  return session;
}

export async function summariseProjectAction(input: {
  projectId: string;
}): Promise<{ summarised: number; failed: number; message: string }> {
  await requireOwner();

  const result = await summariseProject(input.projectId);
  revalidatePath(`/admin/projects/${input.projectId}`);

  if (result.skipped) {
    return {
      summarised: 0,
      failed: 0,
      message: result.error ?? "No AI provider is configured.",
    };
  }

  if (result.considered === 0) {
    return { summarised: 0, failed: 0, message: "Everything is already summarised." };
  }

  const parts = [`${result.summarised} summarised`];
  if (result.failed > 0) parts.push(`${result.failed} not returned`);
  if (result.error) parts.push(result.error);

  return {
    summarised: result.summarised,
    failed: result.failed,
    message: parts.join(" · "),
  };
}

export async function retryFailedSummariesAction(input: {
  projectId: string;
}): Promise<{ reset: number }> {
  await requireOwner();
  const reset = await resetSummaryFailures(input.projectId);
  revalidatePath(`/admin/projects/${input.projectId}`);
  return { reset };
}

export async function summaryStatusAction(input: { projectId: string }) {
  await requireOwner();
  return summaryStatus(input.projectId);
}

Step 2: Write the panel

src/components/admin/summary-panel.tsx:

"use client";

import { useState, useTransition } from "react";
import { Sparkles, RotateCcw } from "lucide-react";

import {
  retryFailedSummariesAction,
  summariseProjectAction,
} from "@/app/actions/summarise";

export function SummaryPanel({
  projectId,
  status,
}: {
  projectId: string;
  status: { total: number; summarised: number; stale: number; failing: number };
}) {
  const [pending, startTransition] = useTransition();
  const [message, setMessage] = useState<string | null>(null);

  const remaining = status.total - status.summarised;

  return (
    <section className="space-y-4 rounded-[var(--radius-tile)] border border-border-subtle p-4">
      <div>
        <h2 className="text-sm font-semibold text-text-primary">
          Plain-English summaries
        </h2>
        <p className="mt-0.5 text-xs text-text-muted">
          <span className="tabular">{status.summarised}</span> of{" "}
          <span className="tabular">{status.total}</span> commits summarised
          {status.stale > 0 && (
            <>
              {" · "}
              <span className="tabular">{status.stale}</span> written under an
              older prompt
            </>
          )}
          {/* Silent degradation to raw subjects, with no indication, would go
              unnoticed for weeks. */}
          {status.failing > 0 && (
            <>
              {" · "}
              <span className="text-accent">
                <span className="tabular">{status.failing}</span> failing
              </span>
            </>
          )}
        </p>
      </div>

      <div className="flex flex-wrap items-center gap-3">
        <button
          type="button"
          disabled={pending || remaining === 0}
          onClick={() =>
            startTransition(async () => {
              setMessage("Generating…");
              const result = await summariseProjectAction({ projectId });
              setMessage(result.message);
            })
          }
          className="inline-flex items-center gap-1.5 rounded-lg bg-brand px-3 py-1.5 text-sm font-medium text-brand-fg hover:bg-brand-hover disabled:opacity-50"
        >
          <Sparkles size={14} strokeWidth={2} />
          {remaining === 0 ? "All summarised" : `Summarise ${remaining}`}
        </button>

        {status.failing > 0 && (
          <button
            type="button"
            disabled={pending}
            onClick={() =>
              startTransition(async () => {
                const result = await retryFailedSummariesAction({ projectId });
                setMessage(`${result.reset} commits queued for another attempt.`);
              })
            }
            className="inline-flex items-center gap-1.5 rounded-lg border border-border-subtle px-2.5 py-1.5 text-xs text-text-secondary hover:bg-surface-raised disabled:opacity-50"
          >
            <RotateCcw size={13} strokeWidth={2} />
            Retry failing
          </button>
        )}

        {message && <span className="text-xs text-text-muted">{message}</span>}
      </div>
    </section>
  );
}

Step 3: Render it on the project admin page

In src/app/admin/projects/[projectId]/page.tsx, add below the IngestPanel:

import { SummaryPanel } from "@/components/admin/summary-panel";
import { summaryStatus } from "@/lib/ai/summarise";

          <SummaryPanel
            projectId={project.id}
            status={await summaryStatus(project.id)}
          />

Step 4: Verify

npm run build

Expected: build succeeds.

Step 5: Commit

git add src/app/actions/summarise.ts src/components/admin/summary-panel.tsx src/app/admin/projects
git commit -m "feat(admin): summary progress and manual generation"

Task 6: Integration tests

Files:

Create: tests/integration/summarise.test.ts
Step 1: Write the tests

tests/integration/summarise.test.ts:

import { describe, it, expect, beforeAll, afterAll } from "vitest";
import PocketBase from "pocketbase";

import { DEMO, adminPb, requireServices } from "./helpers";

let admin: PocketBase;
let projectId: string;
const created: string[] = [];

beforeAll(async () => {
  await requireServices();
  admin = await adminPb();
  projectId = (
    await admin.collection("projects").getFirstListItem(`slug = "${DEMO.projectSlug}"`)
  ).id;
});

afterAll(async () => {
  for (const id of created) {
    await admin.collection("commits").delete(id).catch(() => {});
  }
});

describe("merge commits are never summarised", () => {
  it("excludes them from the pending set", async () => {
    const { summaryStatus } = await import("@/lib/ai/summarise");
    const status = await summaryStatus(projectId);

    const merges = (
      await admin
        .collection("commits")
        .getList(1, 1, { filter: `project = "${projectId}" && is_merge = true`, fields: "id" })
    ).totalItems;

    const all = (
      await admin
        .collection("commits")
        .getList(1, 1, { filter: `project = "${projectId}"`, fields: "id" })
    ).totalItems;

    // status.total counts non-merge commits only.
    expect(status.total).toBe(all - merges);
    expect(merges).toBeGreaterThan(0);
  });
});

describe("summarisation degrades rather than failing", () => {
  it("returns a skipped result when no provider is configured", async () => {
    const settings = await admin
      .collection("settings")
      .getFirstListItem("id != ''")
      .catch(() => null);
    const previousKey = settings?.ai_api_key ?? "";

    if (settings) {
      await admin.collection("settings").update(settings.id, { ai_api_key: "" });
    }

    const { summariseProject } = await import("@/lib/ai/summarise");
    const result = await summariseProject(projectId);

    // Must not throw. The dashboard and board depend on this.
    expect(result.skipped).toBe(true);
    expect(result.summarised).toBe(0);

    if (settings && previousKey) {
      await admin.collection("settings").update(settings.id, { ai_api_key: previousKey });
    }
  });
});

describe("a stored summary survives re-ingestion", () => {
  it("is not cleared when the same commit arrives again", async () => {
    const commit = await admin.collection("commits").create({
      project: projectId,
      sha: `summary-probe-${Date.now()}`,
      short_sha: "sumprb",
      message: "feat(probe): a thing",
      subject: "a thing",
      type: "feat",
      scope: "probe",
      authored_at: new Date().toISOString(),
      is_merge: false,
      link_source: "none",
      summary: "Did a thing for the client.",
      summary_version: 1,
      summary_attempts: 0,
    });
    created.push(commit.id);

    const { relinkProject } = await import("@/lib/linking/engine");
    await relinkProject(projectId);

    const after = await admin.collection("commits").getOne(commit.id);
    expect(after.summary).toBe("Did a thing for the client.");
  });
});

Step 2: Run the integration suite

With npm run dev running:

set -a && source .env.local && set +a && npm run test:integration

Expected: PASS — 44 tests (41 after Plan 3a, plus 3 here).

Step 3: Commit

git add tests/integration/summarise.test.ts
git commit -m "test(ai): merge exclusion, graceful degradation, summary persistence"

Task 7: Full verification

This task is work, not a summary. Run every command and act on what it prints.

Step 1: Run every test

set -a && source .env.local && set +a && npm run test:all

Expected: PASS. Unit tests total 231 — the 213 after Plan 3a, plus 18 in batch.test.ts. Integration tests total 44.

Step 2: Summarise the real history

Open the project's admin page and press Summarise. This runs 779 commits in batches of 25 — about 32 requests, roughly ninety cents on claude-opus-5.

Expected: it completes with a high summarised count and few failures. Press it again.

Expected: "Everything is already summarised." A second run that regenerates means the version check or the pending filter is wrong, and every run would cost another dollar.

Step 3: Read twenty summaries critically

set -a && source .env.local && set +a && npx tsx -e "
import PocketBase from 'pocketbase';
const pb = new PocketBase(process.env.NEXT_PUBLIC_PB_URL!);
pb.autoCancellation(false);
await pb.collection('_superusers').authWithPassword(process.env.PB_SUPERUSER_EMAIL!, process.env.PB_SUPERUSER_PASSWORD!);
const rows = await pb.collection('commits').getList(1, 20, { filter: 'summary != \"\"', sort: '-authored_at' });
for (const r of rows.items) console.log(\`\n[\${r.short_sha}] \${r.subject}\n  -> \${r.summary}\`);
"

Check each pair. Specifically:

Does the summary describe that commit? A mismatch means the index parsing is broken, which is the failure this plan is built around.
Do the removal commits read as removals? WFDASH-WEBAPP has several — remove register/reset tabs, remove Adjust Quantity. A removal described as an addition is wrong in a way a client will notice.
Any dates, deadlines, or claims that something is finished? There must be none.
Any jargon, file paths, or commit hashes? There must be none.
Step 4: Test a reverted commit

The repository contains no revert: commit, so this needs a crafted one. Insert a commit with the message revert: remove the POS integration added in feat(pos), summarise, and read the result.

Expected: it reads as work being undone. If it reads as progress, the preamble's revert rule is not holding and must be strengthened before a client sees this.

Step 5: Confirm the failure path

Clear the API key in settings, then open the dashboard and a card.

Expected: both render fully, using raw commit subjects. Nothing is blank and nothing errors.

Step 6: Commit any fixes

git add -A
git commit -m "fix: issues found during Phase 3b verification"

Definition of done

npm run test:all passes: 231 unit, 44 integration.
npm run build succeeds with no type errors.
All 779 non-merge commits have a summary; a second run generates nothing.
Twenty spot-checked summaries describe the right commits, contain no dates, no completion claims and no jargon, and removals read as removals.
A crafted revert reads as work undone.
With no key configured, every page renders using raw subjects.
The owner's panel shows summarised, stale and failing counts.

Deliberately not in this plan

Card narratives, and any change to what a client sees — Plan 3c. The summaries are stored but not yet displayed anywhere outside the admin check in Task 7.
Automatic regeneration when PROMPT_VERSION changes. Stale summaries are counted and shown to the owner; regenerating is a button press. Automatic rewriting would silently change text a client has already read.
Retry with backoff inside a run. A failed batch is retried on the next ingest or button press, capped at three attempts per commit.
Summarising anything other than commits — not comments, not activity.

33 KiB Raw Blame History

Phase 3b — Commit One-Liners Implementation Plan

The one rule that makes this safe

File structure

Task 1: Schema

Task 2: Batching, prompting and parsing

Task 3: The summarisation service

Task 4: Running it on ingest

Task 5: The owner's summary panel

Task 6: Integration tests

Task 7: Full verification

Definition of done

Deliberately not in this plan

33 KiB

Raw Blame History