Root cause of the silent-crash-after-hours cluster (#4765/#2233/#4633): the
global in-memory SESSIONS LRU evicted with a blind popitem(last=False), which
could drop an actively streaming or not-yet-persisted session (data loss) and
was capped only via an env var. On long-running installs the effective result
was unbounded RAM growth until segfault.
- Add _session_is_evictable(): a session is evictable ONLY when it is not
streaming (no active_stream_id), has no in-flight turn (no pending_user_message
/ pending_started_at), and its full state is proven on disk (sidecar
message_count >= in-memory count; metadata-only stubs and zero-message shells
are trivially safe).
- Add _evict_sessions_over_cap(): replaces all 8 blind popitem loops across
models.py, routes.py, streaming.py. Walks the LRU oldest-first and removes only
provably-safe entries; never acquires LOCK/stream locks itself (caller holds
LOCK) so no lock-ordering deadlock. May briefly exceed the cap rather than ever
evict an active/unsaved session.
- Make the cap configurable via config.yaml webui.sessions_cache_max
(get_sessions_cache_max()); precedence config.yaml -> HERMES_WEBUI_SESSIONS_MAX
(legacy) -> DEFAULT_SESSIONS_CACHE_MAX=300. No new HERMES_* env var. Invalid or
<1 values fall back so a typo can never disable the bound.
- Evicted sessions lazily reload from their JSON sidecar via the existing
get_session() accessor; no call sites changed. _index.json sidebar behavior
unchanged (the sidebar reads the index, not SESSIONS).
- Add tests/test_issue4765_sessions_lru_eviction.py (8 tests): eviction past
cap, active/streaming never evicted, unsaved/stale-tail never evicted, lazy
reload with identical content, and no-data-loss under heavy churn.
- README: document the config.yaml key + safety semantics.
Fixes#4765.
The Node-executed gate in test_anchor_fallback_ownership.py
(test_render_messages_keeps_anchor_owned_turn_out_of_legacy_activity_rebuilds)
eval()s the real renderMessages(). Commit 7536f6f1 made renderMessages schedule
its post-render pass via _postProcessWithAnchorSuppression(), but the harness
only stubbed postProcessRenderedMessages() — so the eval threw
'ReferenceError: _postProcessWithAnchorSuppression is not defined' and the test
failed on Linux CI (shard 2). It passed locally only because Windows hit the
unrelated WinError 206 command-line-too-long first, masking the real error.
Add a no-op stub for _postProcessWithAnchorSuppression alongside the existing
postProcessRenderedMessages stub. Verified by dumping the generated node script
to a temp .js file and running 'node file.js' (bypassing the Windows -e length
limit): the eval no longer throws and the test's assertions pass.
Commit 7536f6f1 routed the deferred post-render dispatches through
_postProcessWithAnchorSuppression() (holds overflow-anchor suppression across
the async media/layout settle frame, then calls postProcessRenderedMessages).
Six pre-existing tests string-matched the old
'requestAnimationFrame(()=>postProcessRenderedMessages(inner))' literal and
failed on the rename — behavior is preserved (the wrapper still invokes
postProcessRenderedMessages), so this is a test-fix not a code-fix.
Per the gate-cert recommendation, the tests now assert the BEHAVIOR chain
(post-render is scheduled via _postProcessWithAnchorSuppression, and that
wrapper calls postProcessRenderedMessages) rather than the exact rAF literal, so
a future wrapper rename can't re-orphan them.
Files: test_csv_table_rendering, test_excalidraw_inline_embed,
test_issue483_inline_diff_viewer, test_issue484_json_tree_viewer, test_issue347,
test_pdf_html_preview. Verified: the 6 updated assertions pass locally (the only
local failures are the pre-existing Windows-only WinError 206 command-line-too-long
in Node-harness tests, unrelated, green on Linux CI).
The sync-frame guards (_fixMobileScrollJank / _suppressBrowserOverflowAnchor)
only cover the render frame itself. postProcessRenderedMessages() — syntax
highlight, inline diff/csv/pdf/html/excalidraw, katex/mermaid — is scheduled a
FRAME LATER via requestAnimationFrame(), after those guards have released. Each
of those can change the height of rows ABOVE the viewport; on mobile
(overflow-anchor:auto) the browser's native anchor engine then compensates
scrollTop a SECOND time in that unguarded frame, yanking an unpinned reader to
another turn (the residual mobile 往回大跳).
Wrap all three deferred post-process dispatches (fast-path cache branch, main
render tail, live-tool remount) in _postProcessWithAnchorSuppression(), which
routes through the shared _suppressBrowserOverflowAnchor() and holds suppression
one extra frame so late media/layout reflow is covered too. Desktop rests at
overflow-anchor:none so the wrapper is a verified no-op there.
Reproduced on an isolated debug instance with a cloned 1179-message session:
above-viewport +350px during the async settle window jumped scrollTop +350 on
mobile (auto) and 0 with the wrapper; desktop (none) 0 both ways. static/ui.js
only.
Codex gate on #5346 flagged two brittle static extractors that broke on the
cancelStream(reason) signature change + the new '[stream] cancel requested'
stdout log:
- test_cancel_stream_owner_guard.py: the Node harness parsed ALL of stdout as
JSON; the provenance console.info line (fires during runAll) polluted it.
Parse the LAST non-empty stdout line (the result JSON is always emitted last,
after runAll resolves).
- test_sprint36.py: two extractors did src.find('async function cancelStream()')
(exact, no params). Switched to a signature-tolerant regex and widened the
catch-block window (the provenance log/comments now precede the try/catch).
Both pre-existing tests, updated to match the intentional #5345 change (not the
code bent to fit the test).
/api/clarify/pending always returns HTTP 200 when present (returns
{"pending": null} for an unknown session — it never 404s). The front-end
clarify poller warned "Clarify endpoint unavailable. Please restart
server." on ANY caught error whose message merely contained "404" or
"not found", so an unrelated stale-session 404 ("Session not found", e.g.
an old-profile session polling briefly after a profile switch) or a
transient error produced a misleading missing-endpoint toast that pointed
operators at the wrong layer.
Clarify polling now branches on the structured HTTP status that api()
attaches to the thrown Error (err.status):
- 404 "Session not found" -> handled as a stale-session poll (stop + hide
card silently), no toast;
- restart-server warning fires only on a genuine route-not-found 404 whose
body is NOT session-scoped;
- poll failures are logged with path, status, polling session id, and
current session id for diagnosis.
Interrupt provenance: cancelStream()/cancelSessionStream() now log a
'[stream] cancel requested' line with the trigger reason
(composer-stop / slash-stop / slash-interrupt / busy-interrupt /
sidebar-stop). Passive UI lifecycle events (session switch, tab hide, page
unload) already tear down only the local SSE transport via
closeLiveStream() and never call /api/chat/cancel — only explicit
Stop/interrupt paths interrupt the backend agent/tool run. This is
confirmed by test_clarify_pending_never_404s locking the handler shape.
Supersedes #5343 (which handled only the profile-switch sub-case and kept
the broad message-scrape). Adds tests/test_issue5345_*.py (9 tests).
Co-authored-by: claw-io <claw-io@users.noreply.github.com>
Co-authored-by: ruizanthony <ruizanthony@users.noreply.github.com>
Phase 2 of cleanup now sweeps _index.json for stale entries with no
backing file and no in-memory session, removing them regardless of
title (catches multi-language 'Untitled' variants). Phase 3 only
deletes the index when Phase 1 removed files AND Phase 2 couldn't
run, fixing the cache-busting on every cleanup call.
Fix two phase-interaction bugs flagged by Greptile review:
- Track phase1_removed_ids to prevent Phase 2 double-counting
sessions already removed from disk by Phase 1.
- Track phase2_rewrote_index so Phase 3 skips deletion when
Phase 2 already cleaned the index in-place.
Adds test_issue5331_index_only_ghost_cleanup.py with 12 test cases.
The root non-CLI classification surfaced subagent rows in the sidebar without
read_only, exposing delete/truncate/pin — and /api/session/delete calls
delete_cli_session() which erases the child's state.db transcript (data loss).
- /api/sessions list: coerce subagent rows to read_only=True + is_cli_session=False
so the UI offers no mutation affordances.
- New _session_is_subagent_view_only() shared guard; applied to the direct
mutation routes that bypass _get_or_materialize_session(): delete / clear /
truncate / pin now 400 for subagent children. (rename/move already route
through the gated _get_or_materialize_session and 403.)
- tests: guard helper + static contract that all 4 routes + list coercion present.
Fixes#5307
- _get_or_materialize_session happy path: reject an already-persisted subagent
sidecar (source_tag/raw_source=='subagent' or state.db subagent) even when it
was stored read_only=False (pre-fix materialization), so chat-start can't write it.
- import_cli existing-session refresh: coerce read_only=True on the persisted
sidecar (and response) when it's a subagent child.
- test: persisted read_only=False subagent sidecar is still refused by the helper.
Fixes#5307
Codex found a 3rd writable path: POST /api/chat/start -> _get_or_materialize_session()
materialized source='subagent' as a writable sidecar before the not_claimable guard.
Fix: refuse subagent children (PermissionError) at that shared chokepoint, checked via
_is_subagent_child_session_id(sid) (state.db source, independent of cli_meta) BEFORE the
materialize path — so all three entry points (GET synth, import_cli, chat-start) now
consistently keep a delegated child view-only. CLI/TUI/Desktop materialization preserved.
Tests: materialize helper refuses subagent child + still allows tui.
Fixes#5307
Opt live assistant streaming nodes out of the global theme color/background transitions so token-by-token markdown updates do not flash or fade on light themes.
Adds a targeted regression test in test_smooth_text_fade.py while keeping the opt-in smooth text fade feature intact.
- api/routes.py GET /api/session synthesized response: stop hardcoding
is_cli_session=True; serialize bool(synth.is_cli_session) so a recovered
subagent child stays not-CLI-classified (was overriding the helper's False).
- api/routes.py POST /api/session/import_cli: gate source='subagent' into the
read-only view payload (is_cli_session=False, imported=False) BEFORE
import_cli_session(), so a subagent child can never be materialized as a
writable WebUI sidecar via this endpoint.
- tests: assert import_cli routes subagent children read-only (no materialize).
Both were paths that bypassed the _is_claimable_cli_source denylist. Fixes#5307
Reworked per the Codex gate finding: the first approach widened the client
import predicate, which (a) conflicted with #3603's intentional _isExternalSession
gate and (b) let import_cli persist the subagent child as a WRITABLE, CLI-classified
session that then passed the poll-skip/active-refresh gates.
Corrected to a server-side, view-only recovery:
- api/routes.py: mark source='subagent' as NON-claimable in _is_claimable_cli_source
(both cli_meta and state.db source denylists). A subagent child now resolves via
the not_claimable branch -> read-only Session with its state.db transcript, and
build_session takes is_cli_flag (False for subagent children) so the recovered
session is NOT CLI-classified and can't widen the frontend _isExternalSession gates.
- static/sessions.js: REVERTED to master (no client change needed; #3603 contract intact).
- tests: assert reason=not_claimable, read_only=True, is_cli_session!=True, transcript present;
#2782 deleted-webui 404 preserved; #3603 _isExternalSession contract preserved.
58 tests green (5307 + 3603 + claim-cli + core-data-loss). Fixes#5307
A delegated subagent child (source='subagent' in state.db) usually has no WebUI
sidecar but is registered in the WebUI index sharing the parent's lineage. That
made GET /api/session -> _claim_or_synthesize_cli_session return 'was_webui' ->
404, so the child pane opened empty despite state.db holding messages.
- api/routes.py: add _state_db_session_source() + _is_subagent_child_session_id();
exclude subagent children from the was_webui 404 gate so they recover their
state.db transcript (the #2782 self-heal 404 for deleted WebUI sessions is kept).
- static/sessions.js: add _isSubagentChildSession() + _sessionNeedsServerImportForLoad()
(kept separate from _isExternalSession to avoid widening refresh-gating); the
main session-tap, lineage-segment, and child-open handlers now trigger the
import/merge path for subagent children.
- tests: test_5307_subagent_child_transcript.py (7 tests).
Fixes#5307
#5306 (flicker): while a parent WebUI session is the active/streaming session,
a linked delegate subagent child that transiently reports message_count===0
between /api/sessions polls was dropped by _sidebarRowHasVisibleMessages BEFORE
_attachChildSessionsToSidebarRows could stack it under its parent. It never
entered sessionsRaw, so the row vanished, then reappeared on the next refresh
once its list metadata caught up — the flicker. Extend the visibility predicate
with an active-parent exception (mirroring the existing active-session
exception): a child_session whose parent_session_id is the active sidebar
session stays visible even at message_count 0. Scoped to the active parent so
truly-empty unrelated sessions are still hidden.
#5305 (orphan): a delegated subagent child whose WebUI parent is filtered out of
the current render (project/profile/source scope) was promoted to a contextless
top-level "Subagent Session" orphan. Suppress cross-surface child_session rows
whose parent row is absent from the render instead of orphaning them, mirroring
the existing archived-hidden-parent suppression (#4293). The genuinely-external
parent case (messaging/CLI) still orphans via the parentIsExternal branch.
Tests: tests/test_5306_subagent_sidebar_flicker.py (7 tests) locks both
invariants and the no-regression guards, executing the real sessions.js helper
regions under node like the existing lineage tests.
The agent source at /opt/hermes inside the container may contain a
.playwright/ directory with browser dependency files that have restricted
permissions. rsync fails with exit code 23 when attempting to read them,
which kills the container build ("Failed to stage hermes-agent source").
- Add --exclude=.playwright to rsync in docker_init.bash
- Add rm -rf .playwright to cp -a fallback path for symmetry
- Update test_docker_init_excludes_egg_info_during_staging to assert
both the rsync --exclude and a broad .playwright presence check
- Add a brief note in AGENTS.md Contribution style about mirroring
directory exclusions in both rsync and cp paths
Fixes#5315
Reapply footer control ordering on current upstream/master while preserving the required situational chip renderer. Persist composer_control_order with backend validation, keep the settings descriptions reorder-aware, and make primary/situational chip renderers participate in same-group drag ordering.
Verified with: node --check static/boot.js; node --check static/panels.js; git diff --check; ./scripts/test.sh tests/test_issue4598_composer_control_visibility.py
The embedded workspace terminal spawns a PTY shell that runs arbitrary
commands as the server-process user. check_auth() returns True
unconditionally when no password/passkey is configured (the default
out-of-the-box state), so without a network-scope gate the terminal
endpoints were reachable by any unauthenticated caller able to hit the
port — which on a passwordless public bind is remote code execution.
Apply the same local-origin gate the onboarding/bootstrap endpoints use
(_onboarding_gate_allows) to /api/terminal/{start,input,resize,close} and
/api/terminal/output: with auth disabled, accept only loopback/private
origins, ignore spoofable X-Forwarded-For/X-Real-IP unless
HERMES_WEBUI_TRUST_FORWARDED_FOR=1, and honor HERMES_WEBUI_ONBOARDING_OPEN=1
as the explicit opt-out for a deliberately-exposed server. Auth-enabled
servers (cookie already verified upstream) and genuine same-host clients
are unaffected.
Also fixes a latent test-isolation leak in
test_extension_route_remains_behind_webui_auth: it set HERMES_WEBUI_PASSWORD
but never invalidated the process-wide password-hash cache, so its result
depended on suite execution order (exposed when the new test file shifted
ordering). Invalidate before+after so it reads the env var deterministically.
12 new gate tests in tests/test_cvd3_terminal_local_origin_gate.py.
A public TTS host could 301/302/303-redirect POST /audio/speech to an internal
target (e.g. http://169.254.169.254), and urllib's default redirect handler
would follow it carrying the Authorization bearer — both an SSRF bounce past the
base-url guard and a credential leak. Now uses a no-redirect opener
(_NoRedirectTtsHandler raises on any redirect) via the _tts_open seam. +redirect
rejection regression test. Residual DNS-rebinding TOCTOU (re-resolve at connect)
is a narrower low-severity window noted for follow-up.
