feat(pricelist): Catalogue search hits every field (incl. synonyms)

The Catalogue tab's search box used to OR across just sku,
description, and brand. Now it hits:

  - sku, description, brand, supplier, category, subcategory
    (PB's case-insensitive `~` operator, each indexed)
  - search_text (a denormalised text blob)

The denormalised column is the key part: PB can't substring-
match a JSON field, so without this, synonyms from
`specifications.synonyms` ("slugger cutter", "turbo tough",
etc.) would never match in the search box. The n8n workflow's
Code node now flattens every searchable bit — including the
synonyms array — into search_text on every POST. The 46 pre-
existing rows were backfilled once with the same formula.

The `price_list.supplier` field also stored the PB id on the
46 pre-existing rows (n8n was setting the resolved name, but
the original LLM tests that seeded the table didn't). Fixed
that in the same backfill — supplier now reads 'JEISA -
WESTSCOTT GROUP' instead of 'q6hpwbxsicyoiuu', which makes
the supplier filter dropdown work AND makes
'supplier ~ "jeisa"' match.

Migration `2026-06-30-add-price-list-search-text`:
  - Adds `search_text` (text) column
  - Adds `idx_pl_search_text` non-unique index
  - Idempotent — re-runs are safe and self-heal the index

Updates:
  - `getPriceLists` helper ORs across all 6 columns + search_text
  - `PriceList` TS type gains `search_text: string`
  - `pricelist-extraction.md` Code node snippet shows how the
    n8n workflow should populate `search_text` on every POST
  - Search input placeholder reads 'Search anything — SKU,
    description, brand, supplier, synonyms…'

Tests: 229/230 (same pre-existing failure, unrelated).
TypeScript clean.
This commit is contained in:
SJCoetz 2026-06-30 12:10:27 +02:00
parent b75873a51b
commit 1fb67efaee
6 changed files with 156 additions and 23 deletions

View file

@ -28,8 +28,10 @@ WFDASH has two related collections, both already in PB:
| `markup` | number | workflow default (25%) — operator override later |
| `selling_price` | number | workflow computes: `cost_price × (1 + markup/100)` |
| `stock_qty` | number | workflow default (0) — inventory updates later |
| `specifications` | json | AI: rich JSON (attributes + synonyms) |
| `created` / `updated` | autodate | — |
| `specifications` | json | AI: rich JSON (attributes + synonyms) |
| `source_file` | text | workflow: `raw_price_list.id` this row was extracted from |
| `search_text` | text | workflow: denormalised search blob (sku + description + brand + supplier + category + subcategory + specifications.synonyms) — PB can't substring-match JSON, so the Catalogue search box hits this column directly |
| `created` / `updated` | autodate | — |
Indexed on `sku`, `category`, `subcategory`, `brand`, `supplier` — so
the agent's "find me all HSS cutters 12mm from JEISA" becomes a
@ -342,20 +344,36 @@ const DEFAULT_MARKUP = 25;
const rows = parsed.items
.filter(it => it.cost_price != null) // skip POA / TBA
.map(it => ({
sku: it.sku,
description: it.description,
category: it.category,
subcategory: it.subcategory || null,
brand: it.brand || null,
supplier, // denormalized from raw
cost_price: it.cost_price,
markup: DEFAULT_MARKUP,
selling_price: Math.round(it.cost_price * (1 + DEFAULT_MARKUP / 100) * 100) / 100,
unit: it.unit || 'each',
stock_qty: 0,
specifications: it.specifications,
}));
.map(it => {
const synonyms = Array.isArray(it.specifications?.synonyms) ? it.specifications.synonyms : [];
// search_text: denormalised blob of every searchable bit. PB
// can't substring-match JSON, so the Catalogue tab's search box
// hits this column directly to support synonym-based fuzzy
// matches ("slugger cutter" finds the HSS cutters whose
// specifications.synonyms contains it). Kept lowercase for
// case-insensitive matching on the PB side.
const searchText = [
it.sku, it.description, it.brand, supplier,
it.category, it.subcategory, ...synonyms,
].filter(Boolean).join(' ').toLowerCase().trim();
return {
sku: it.sku,
description: it.description,
category: it.category,
subcategory: it.subcategory || null,
brand: it.brand || null,
supplier, // denormalized from raw
cost_price: it.cost_price,
markup: DEFAULT_MARKUP,
selling_price: Math.round(it.cost_price * (1 + DEFAULT_MARKUP / 100) * 100) / 100,
unit: it.unit || 'each',
stock_qty: 0,
specifications: it.specifications,
source_file: rawId, // trace-back + verify key
search_text: searchText, // fuzzy search index
};
});
return rows.map(r => ({ json: r }));
```

View file

@ -24,6 +24,7 @@
"pb:migrate:raw-price-list": "npx tsx scripts/migrations/2026-06-30-add-raw-price-list.ts",
"pb:migrate:pricelist-notes": "npx tsx scripts/migrations/2026-06-30-add-pricelist-notes.ts",
"pb:migrate:price-list-source-file": "npx tsx scripts/migrations/2026-06-30-add-price-list-source-file.ts",
"pb:migrate:price-list-search-text": "npx tsx scripts/migrations/2026-06-30-add-price-list-search-text.ts",
"smoke:counter": "npx tsx scripts/smoke/counter.ts",
"smoke:counter-concurrent": "npx tsx scripts/smoke/counter-concurrent.ts",
"smoke:create-from-email": "npx tsx scripts/smoke/create-from-email.ts",

View file

@ -0,0 +1,83 @@
/**
* Migration — add `search_text` (text) to `price_list`.
*
* PB can't substring-match a JSON field, so for the Catalogue tab's
* "search everything" feature, we denormalize the searchable bits of
* each row into a flat text field. The WFDASH Catalogue helper
* (`getPriceLists`) then ORs against `search_text` and the other
* string columns.
*
* What's in search_text (per row, joined with spaces):
* - sku
* - description
* - brand
* - supplier
* - category
* - subcategory
* - specifications.synonyms (the buyer-friendly aliases)
*
* Idempotent: re-runs are safe.
*
* Run with:
* npm run pb:migrate:price-list-search-text
*/
import PocketBase from "pocketbase";
const PB_URL = process.env.NEXT_PUBLIC_POCKETBASE_URL || "http://127.0.0.1:8090";
const ADMIN_EMAIL = process.env.PB_ADMIN_EMAIL;
const ADMIN_PASSWORD = process.env.PB_ADMIN_PASSWORD;
if (!ADMIN_EMAIL || !ADMIN_PASSWORD) {
throw new Error("PB_ADMIN_EMAIL and PB_ADMIN_PASSWORD are required.");
}
async function main() {
const pb = new PocketBase(PB_URL);
await pb.collection("_superusers").authWithPassword(ADMIN_EMAIL, ADMIN_PASSWORD);
const cols = await pb.collections.getFullList();
const priceList = cols.find((c) => c.name === "price_list");
if (!priceList) {
throw new Error("price_list collection not found.");
}
// Always re-check the index — running the migration on a
// collection that already has `search_text` should still ensure
// the index exists (handles the case where the field was added
// before the index step was introduced).
const INDEX_SQL = "CREATE INDEX `idx_pl_search_text` ON `price_list` (`search_text`)";
const hasIndex = (priceList.indexes || []).some(
(i) => typeof i === "string" && i.includes("idx_pl_search_text"),
);
if (priceList.fields.some((f) => f.name === "search_text")) {
if (hasIndex) {
console.log("price_list.search_text + index already present, skipping.");
return;
}
await pb.collections.update(priceList.id, { indexes: [...(priceList.indexes || []), INDEX_SQL] });
console.log("Added price_list index on search_text (field already existed).");
console.log("search_text index migration complete.");
return;
}
await pb.collections.update(priceList.id, {
fields: [
...priceList.fields,
{ name: "search_text", type: "text", required: false, options: { max: 0 } },
],
// Add a non-unique index so the search box doesn't full-scan
// the table once we have thousands of rows.
indexes: [
...(priceList.indexes || []),
"CREATE INDEX `idx_pl_search_text` ON `price_list` (`search_text`)",
],
});
console.log("Added price_list.search_text (+ index)");
console.log("search_text migration complete.");
}
main().catch((err) => {
console.error(err);
process.exit(1);
});

View file

@ -328,7 +328,7 @@ export function PricelistCatalogue() {
<div className="relative flex-1 sm:max-w-sm">
<Search className="absolute left-3 top-1/2 -translate-y-1/2 h-4 w-4 text-muted-foreground" />
<Input
placeholder="Search SKU, description, or brand…"
placeholder="Search anything — SKU, description, brand, supplier, synonyms…"
value={search}
onValueChange={setSearch}
className="pl-9"

View file

@ -1353,11 +1353,17 @@ export async function deleteRawPriceList(id: string) {
* Catalogue — fetch a page of rows from the `price_list` collection.
*
* Backs the "Catalogue" tab on the `/pricelist` page. Search hits
* `sku`, `description`, and `brand` (case-insensitive `~`). Optional
* `category` and `brand` filters narrow further. PB can't substring-
* match the `specifications` JSON (it's stored as a blob) so synonym
* search isn't supported here — that goes through the LLM agent at
* quote time.
* every searchable text field:
* - `sku`, `description`, `brand`, `supplier`, `category`,
* `subcategory` — each via PB's case-insensitive `~` operator
* - `search_text` — the denormalised blob that includes the
* `specifications.synonyms` array (PB can't substring-match a
* JSON field, so the n8n workflow and the backfill script
* flatten the searchable bits into this column)
*
* The search clauses are OR'd so a hit on any column matches the row.
* Optional `category` and `brand` filters AND the search (narrow
* further).
*
* Newest first — the table is a feed of recent extractions, not an
* alphabetised directory.
@ -1373,7 +1379,21 @@ export async function getPriceLists(
const clauses: string[] = [];
if (search) {
const q = search.replace(/"/g, '\\"');
clauses.push(`(sku ~ "${q}" || description ~ "${q}" || brand ~ "${q}")`);
// OR against every text column + the denormalised search_text.
// The explicit columns are still listed so PB's filter planner
// can use their indexes; search_text is the catch-all for the
// JSON bits (synonyms).
clauses.push(
`(
sku ~ "${q}"
|| description ~ "${q}"
|| brand ~ "${q}"
|| supplier ~ "${q}"
|| category ~ "${q}"
|| subcategory ~ "${q}"
|| search_text ~ "${q}"
)`,
);
}
if (filters.category) {
clauses.push(`category = "${filters.category.replace(/"/g, '\\"')}"`);

View file

@ -512,6 +512,17 @@ export type PriceListRow = {
};
/** `raw_price_list.id` this row was extracted from. Empty for manually-created rows. */
source_file: string;
/**
* Denormalised search blob. PB can't substring-match a JSON
* field, so the n8n workflow and the backfill script flatten
* every searchable bit of a row into this column: sku,
* description, brand, supplier, category, subcategory, and
* the `specifications.synonyms` array. The Catalogue tab's
* search box hits this column directly so fuzzy matches via
* synonyms work ("slugger cutter" finds the HSS cutters whose
* synonyms list contains it).
*/
search_text: string;
};
/**