The two-tier mtime cache from #4783 fixed the per-request SKILL.md rescan
but left two concurrency holes that only bite at container cold start,
when the frontend fires several profile-data requests at once and the
caches are empty:
1. `_get_profile_skills_stats()` had no lock, so concurrent misses on the
same profile each ran `os.walk(followlinks=True)` + parsed every
SKILL.md simultaneously.
2. `_build_profile_rows_fast()` ran outside `_LIST_PROFILES_CACHE_LOCK`
in `list_profiles_api()`, so every concurrent request rebuilt all rows
(each walking every profile's skill tree) at once.
With ThreadingHTTPServer (one OS thread per request) and Docker overlay2,
this stacked thousands of concurrent stat() calls and stalled workers
57-70s (per the report's thread dumps).
Fix:
- Add a per-profile compute lock (registry guarded by a meta-lock) and
use double-checked locking in `_get_profile_skills_stats()`: concurrent
misses on one profile collapse to a single compute, while independent
profiles still compute in parallel.
- Single-flight the row build in `list_profiles_api()` by holding
`_LIST_PROFILES_CACHE_LOCK` across the build + cache write. Lock order
is strictly list-lock -> per-profile skills-lock, so no deadlock.
The report's third suggestion (debounce the mtime probe) is deliberately
NOT taken: the every-call cheap probe is the #4783 out-of-band
change-detection contract (test_issue4783 asserts it MUST run on every
call). Serializing the misses removes the herd without weakening that
contract, since only the expensive compute is guarded, not the probe.
Adds tests/test_issue5364_skills_stats_thundering_herd.py proving the
herd collapses (single compute / single build under a concurrent burst),
independent profiles still parallelize, and the every-call probe contract
is preserved. All existing #4783 contract tests still pass.
Co-authored-by: claw-io <claw-io@users.noreply.github.com>
ab38ffe25f