Windows locks session JSON files briefly during antivirus scans or
browser polling, causing os.replace() to raise WinError 5
(ERROR_ACCESS_DENIED). Add _safe_replace() helper that retries with
exponential backoff (50ms to 800ms, max 5 attempts) on PermissionError.
No-op on non-Windows platforms.
Replaces all 5 os.replace() call sites in models.py:
- Session.save() (main file + backup)
- _write_session_index() (full rebuild + fast path)
- prune_session_from_index()
Codex deep-gate caught a CORE regression in the original top-N cap: capping
_apply_sidebar_state_db_overrides also capped the SOURCE classification that
/api/sessions filters on BEFORE the lazy lineage correction, so a stale-CLI JSON
row whose state.db source is webui could be silently filtered out of a WebUI-only
sidebar once it fell beyond the cap. Split the read: the indexed sessions-table
source/title lookup runs for ALL rows; only the expensive messages COUNT/MAX
aggregation (the actual 5-18s bottleneck) is capped to top-N. Added a 500-row
end-to-end regression with a real state.db proving the beyond-cap stale-CLI row
stays webui.
On power-user instances with thousands of sessions, GET /api/sessions
blocked 5-18s in the all_sessions.state_db_overrides stage:
_apply_sidebar_state_db_overrides read state.db for every row (2400+ ids)
on every concurrent poll. With the gateway-watcher reading and the agent
writing the same state.db, this became lock/IO contention that piled up
concurrent polls and flapped the UI to 'Connection lost'.
Cap the override lookup to the top-N most-recent (paint-priority) rows
that are actually visible, mirroring the existing lineage-enrichment cap
(#4638). The caller passes an already pinned-first/newest-first sorted
list, so the top-N covers the visible window; rows beyond the cap keep
their JSON source/title and are corrected lazily on history-panel open.
Cap defaults to 300, env-configurable via
HERMES_WEBUI_STATE_DB_OVERRIDE_TOP_N (0 disables). Fails open.
Reported by @latipun, reproduced by @b3nw.
Co-authored-by: latipun <latipun@users.noreply.github.com>
Sidebar rows whose backing state.db session has zero messages (a '+'-click
that opened a row but the first turn never committed, or a sidebar nav that
opened then closed before any message landed) now get pruned instead of
lingering forever. The existing #3238/#4591 orphan-prune path only handled
CLI/API-server sidecars; this adds a parallel pass for native-WebUI rows.
- api/models.py: add agent_session_zero_message_sids() batch helper that
mirrors agent_session_rows_existing()'s safe-degrade contract (returns
frozenset() on any error so a transient failure never causes data loss).
Add a small tombstone file at SESSION_DIR/_pruned_webui_orphans.json
(capped at the last 500 sids) so the next poll's
recover_missing_index_sidecars does not re-add a pruned orphan to the
sidebar index — the per-poll fsync'd index write + state.db probe loop
the original PR would have hit on every poll for every orphan. The
helper, the load, and the full-scan fallback all honor the tombstone.
new_session() and import_cli_session() clear any matching tombstone
entry so a fresh session with the same id isn't shadowed.
- api/routes.py: extract the webui zero-message prune into a private
_prune_orphaned_webui_zero_message_sessions() helper, then call it
from BOTH branches of _build_session_list_cache_payload. Established
installs pin show_cli_sessions=False (per api/config.py:7637-7648)
and are exactly the long-time users who accumulated the #4985 404
orphans — the original PR only ran the prune in the if branch and
silently skipped the else branch (maintainer review
IC_kwDOR1LuPM8AAAABHsyFGg). The helper also honors the tombstone
defensively so the prune can't thrash on a row the index still
carries from a prior poll.
- tests/test_issue4985_orphaned_webui_zero_message.py: 39 regression
tests — the original 25 (7 helper-direct + 12 monkeypatch predicate
tests + 6 real-pipeline end-to-end tests) plus 14 follow-up tests
driven by the MUST-FIX / tombstone / r6 signal-swap reviews:
* test_helper_prunes_orphan_keeps_safety_retained_rows — direct call
to the extracted helper with a row set covering every shape the
gate is responsible for.
* test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_false
— explicit regression for the established-install case the
original PR missed.
* test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_true
— companion positive control.
* test_tombstone_persists_across_polls — drives two pipeline builds,
asserts prune fires once total and the tombstoned sid is NOT
re-added to SESSION_INDEX_FILE.
* test_tombstone_does_not_block_new_session_with_same_id — defensive
cover for the new_session() / import_cli_session() tombstone clear.
* test_tombstone_trimmed_to_last_N_entries — verifies the cap.
* test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_true
— companion positive control (show_cli_sessions=True path).
* test_tombstone_self_heals_when_message_added — confirms
Session.save()'s belt-and-suspenders tombstone clear fires when
a sidecar commit lands with len(messages) > 0.
* test_tombstone_loader_is_fail_open_on_corrupt_file[non-json|version-mismatch|non-dict-root]
— 3-parameter regression for the fail-open corrupt-file contract
(#5023 lesson).
* test_sidecar_only_webui_session_is_retained — sidecar JSON carries
real messages but state.db.messages is empty (mirroring lag); the
r6 loaded-sidecar probe retains the row.
* test_sidecar_only_webui_session_self_heals_tombstone — the r4.5
self-heal path combined with the r6 sidecar-only retention rule.
* test_truly_empty_sidecar_with_title_still_pruned — companion
negative test that the r6 fix does NOT over-retain empty sidecars
with non-Untitled titles.
* test_sidecar_only_webui_session_retained_when_show_cli_sessions_false
— established-install path also respects the r6 retention rule.
- CHANGELOG.md: Unreleased entry under Fixed.
fix: target post-#1171 survivors + hoist + tombstone + r6 sidecar-load probe
The r5 sidecar-only retention signal keyed off the row's cached
``message_count`` — which is stale-positive on the very phantom rows
#4985 exists to prune (sidecar ``messages`` empty but cached count > 0).
The r6 signal swaps that cached-count probe for a ``Session.load(sid)
.messages`` probe, gated to ``state.db``-empty candidates only so the
common live-row path stays a no-op. Three retain tests now write real
populated ``messages`` arrays (not just ``[]``) so the loaded-sidecar
check has something to find; the stale-count phantom test is left as
the canonical r6 regression (its ``messages=[]`` + ``message_count=5``
sidecar is exactly the shape the r6 loaded-sidecar check correctly
classifies as orphan). Maintainer review 4584722701.
Opus SHIP-WITH-FIXES (no code change required): the comment/docstring overstated
that the session-list-change listener covers ALL structural mutations. In-app
mutations fire it, but externally-driven changes (scheduled cron completion,
external CLI writes) don't — for those the 30s streaming TTL is the load-bearing
backstop (≤30s bounded, self-healing lag). Comment-only accuracy fix; logic
unchanged (both gates: perf fix correct, freeze effective, staleness bounded).
/api/sessions re-ran the expensive CLI/cron session projection on every
poll while a turn was streaming — a major cause of the multi-second sidebar
latency + 100% CPU on cron-heavy installs (continuing #4672/#4808/#4889).
Root cause: the CLI/cron projection is gated by _CLI_SESSIONS_CACHE, whose
key folds in _sqlite_file_stat_cache_key -> _sqlite_content_fingerprint
(MAX(rowid) FROM messages). During a live turn the gateway writes a message
row per streamed delta, so that fingerprint advances on essentially every
~5s poll — busting the cache and re-running the full candidate-join +
projection (and the lineage-metadata pass) on every poll, contending for the
same SQLite/global lock the streaming worker holds.
The route-level session-list cache already froze its key during streaming
(#4808 _session_list_cache_streaming_freeze_marker), but that freeze never
reached this inner CLI-sessions cache, so the heavy CLI/cron query still
re-ran whenever the outer cache validated.
Fix: while any stream is active, the CLI-sessions cache key folds in the same
stable streaming-freeze marker (keyed only on the set of active stream ids)
instead of the volatile content fingerprint, and the cache TTL widens to a
streaming window. The projection is reused across polls mid-stream and
rebuilt at most once per window. The instant a stream starts/stops the marker
changes, so freshly-finished rows surface promptly. Structural mutations
(cron completion, new/renamed/archived sessions, attention) now also clear
the CLI cache via the existing session-list-changed listener — those signals
never fire per streamed token, which is what makes the freeze safe.
Idle behavior is byte-identical (freeze only engages with active streams).
Applies to both the active-profile and all-profiles cache paths.
Adds tests/test_issue4842_cli_sessions_streaming_freeze.py (6 tests): marker
idle/stable/changes semantics, the core 'key stable across message writes
while streaming' guarantee, idle key still advances with the fingerprint,
streaming TTL > idle TTL, and the structural-mutation listener clears the
CLI cache.
Opus gate finding (BLOCKING): the two #4385 archive-reappears regression tests
mocked Session.load_metadata_only to return an archived stub WITHOUT writing a
real sidecar file, and never set SESSION_DIR. The new stat-gate in
_state_projection_sidecar_metadata short-circuits on SESSION_DIR/{sid}.json
before consulting metadata, so those mocks were never reached -> archived=False.
This is NOT a production regression (in prod, load_metadata_only reads exactly
SESSION_DIR/{sid}.json and returns None when absent, so archived metadata only
exists when the file does — in which case the stat() succeeds and behavior is
identical). The divergence existed only at the tests' mock boundary.
Fix: both tests now write a real archived sidecar JSON under a monkeypatched
SESSION_DIR, exercising the actual production read path (strictly better
coverage than mocking load_metadata_only). Also documented the
SESSION_DIR-as-sole-source assumption on the cache function per Opus's
forward-looking note.
GET /api/sessions pinned CPU to 100% and took multiple seconds on profiles with
many cron sessions. The state.db -> sidebar projection did three pieces of
redundant per-row file I/O, in BOTH the visible pass and the higher-capped
(CRON_PROJECT_CHIP_LIMIT=200) cron-only second pass:
1. _state_projection_sidecar_metadata(sid): uncached open() + 64KB prefix read +
JSON-key scan, once per row.
2. get_last_workspace(): up to two file reads + an is_dir()/remote probe,
returning the SAME workspace for every row.
3. cron/jobs.json: full read + parse per untitled cron row.
On a cron-heavy install that is hundreds of file reads per build, and because
the enclosing _CLI_SESSIONS_CACHE is keyed on a state.db content fingerprint
that advances on every streamed message row, the whole scan was re-paid on
essentially every 5s poll during a live turn (the recurring 100% CPU / slow
/api/sessions from #4672 -> #4808 -> #4842).
Fix:
- Memoize _state_projection_sidecar_metadata by the sidecar's
(path, mtime_ns, size, ctime_ns) stat signature in a bounded module cache, so
a warm build re-stats instead of re-reads. A rename/archive/edit bumps the
signature and invalidates just that one entry (no staleness). This survives
the 5s CLI-cache busting during streaming, which is the actual trigger.
Mirrors the Claude Code parse cache (#4718). Folded into
clear_cli_sessions_cache() for explicit-reset/test parity.
- Build-scoped memos for cron jobs.json (parse once -> job_id->name map) and
get_last_workspace() (resolve once), matching the existing _cron_pid() pattern.
Output is byte-identical. Adds tests/test_issue4842_cron_projection_perf.py
(7 tests: warm-rebuild no re-read, stat-change invalidation, dict-copy isolation,
missing-sidecar default, jobs.json once, workspace once, lifecycle clear).
Dual-gate (Codex + Opus) findings on the Claude Code transcript parse cache:
1. Cache-key staleness hardening: add st_ctime_ns to the (path, mtime_ns, size,
max_messages) key so a rare same-size, same-mtime in-place edit can't serve a
stale parse. Size is already the strong discriminator for append-only JSONL;
ctime_ns is cheap defense-in-depth (a spurious bump costs one harmless re-parse,
never a stale read).
2. Timestamp-fallback bug-for-bug fix: the row-builder guard used identity
(first_ts is None and last_ts is None) but the field assignment uses truthiness
(first_ts or last_ts or _mtime). They disagreed on a falsy 0.0 timestamp
(epoch-0/1970 transcripts), leaving created_at/updated_at/last_message_at = None
instead of the file mtime the pre-cache inline code fell back to. Guard now uses
the same 'not first_ts and not last_ts' test.
Adds 3 regression tests: epoch-0 mtime fallback, the shared-dict read-only
contract (pins the load-bearing invariant), and same-size+mtime ctime invalidation.
The /api/sessions cold build (which a profile switch triggers) was
dominated by get_claude_code_sessions() re-parsing every Claude Code
JSONL transcript line-by-line on each call — 650-1000ms on a 200-file
/ ~130MB tree, repeated on every switch, every 5s CLI-cache expiry, and
every sidebar poll, because the directory is global but the higher
session cache is keyed per active profile.
Memoize the per-file parse by (path, mtime_ns, size) so a warm build
re-stats files instead of re-parsing them; any genuine edit/append
changes the stat signature and invalidates just that file. Also hoist
the loop-invariant get_last_workspace() out of the Claude Code row loop
(it stats config + probes terminal cwd, ~200 redundant calls per build).
Measured /api/sessions cold-rebuild on a 250-session profile: ~670-880ms
-> ~100-200ms (verified in-browser). Output byte-identical.
Credit @rodboev for the browser-level profiling that isolated the real
bottleneck (the prior pre-warm hypothesis could not move it).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex reproduced a SILENT regression: the PR's 4096-BYTE read could miss the
continuation marker the old read_text()[:4096] (4096 CHARS) found. Session.save()
writes compression_anchor_summary (ensure_ascii=False) BEFORE parent_session_id,
so a multi-byte emoji summary puts the marker at char offset ~1230 but byte
offset ~4900 — within the old 4096-char window, past a 4096-byte one → continuation
child no longer detected. Fix: read 16384 bytes (4096 chars worst-case for 4-byte
UTF-8) then slice [:4096] chars, keeping the same needle + bounded I/O. Multibyte
regression test added + proven non-vacuous; updated the read-size assertion to <=16384.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
Codex final-gate found the boundary<watermark advanced signal made
sidecar_advanced_past_watermark globally true, so a stale ts>watermark suffix
row appearing in state.db BEFORE the edited checkpoint bypassed the skip and
resurrected. Gate the bypass on state_replay_idx >= len(sidecar_visible_sequence)
(checkpoint consumed); apply the same gate to the same-second CORE-B exception.
Regression test added + proven non-vacuous. All legit cases (state-only reply
after checkpoint, CORE-B same-second) still pass.
Co-authored-by: AlexeyDsov <alexeydsov@gmail.com>
Final-gate Codex repro found a non-empty-sidecar drop: when the post-edit USER
turn is checkpointed into the sidecar (ts == advanced watermark) but its
ASSISTANT reply exists only in state.db (ts > watermark), the
sidecar_advanced_past_watermark guard was False (max_sidecar_timestamp ==
watermark, not >), so the 'timestamp > watermark' skip dropped the legitimate
reply. GET /api/session returned ['first','first reply','edited prompt'] and
omitted the assistant.
Fix: treat truncation_boundary < watermark (genuinely advanced) as advanced too,
so post-watermark state-only rows merge through even when the sidecar tail only
EQUALS the watermark. Conservative skip preserved for boundary None / == watermark.
Hoisted the loop-invariant boundary check out of the per-message loop. Regression
test added + proven non-vacuous.
Co-authored-by: AlexeyDsov <alexeydsov@gmail.com>
Opus deep-gate found a latent data-loss the Codex harnesses missed: the
empty-sidecar branch kept at_or_after (ts>=watermark) UNCONDITIONALLY, which is
only valid once the watermark was ADVANCED strictly past the original cutoff
(a new turn committed -> boundary < watermark). In the not-advanced states the
content above the watermark is the deleted suffix, so it got resurrected — the
same data-loss class this fix exists to kill. Reachable via crash/cold-load
metadata-vs-sidecar divergence.
- Keep at_or_after ONLY when boundary < watermark (genuinely advanced).
- boundary == watermark (just truncated) OR boundary is None (legacy: a positive
watermark always meant 'frozen at cutoff' pre-#4767) -> conservative ts<=watermark
filter; never resurrects.
- Replaced the backward-scan heuristic with the provable boundary guard.
- Removed the now-dead 'or (not sidecar_messages and watermark>0)' clause in the
non-empty-sidecar path (unreachable; empty sidecar returns earlier).
- Added 2 regression tests (boundary==watermark, legacy None-boundary), both proven
non-vacuous; updated 2 contributor tests that assumed advanced-watermark-without-boundary
(an unreachable state) to pass the boundary the real persistence path always writes.
Co-authored-by: AlexeyDsov <alexeydsov@gmail.com>
Converged after deep bounce cycles: Finding A (production regression — skipped the
state.db source correction before the CLI filter, dropping WebUI sessions with stale-CLI
sidecar flags) FIXED via _apply_sidebar_state_db_overrides(result) before the routes.py
filter on both index + full-scan paths; Finding B (flaky lineage test) FIXED via hermetic
_persisted_session_ids_snapshot fixture. 16 own tests deterministic. Code byte-faithful
from PR head e1a3ecf22eca.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: santastabber <santastabber@users.noreply.github.com>
