Root cause of the silent-crash-after-hours cluster (#4765/#2233/#4633): the
global in-memory SESSIONS LRU evicted with a blind popitem(last=False), which
could drop an actively streaming or not-yet-persisted session (data loss) and
was capped only via an env var. On long-running installs the effective result
was unbounded RAM growth until segfault.
- Add _session_is_evictable(): a session is evictable ONLY when it is not
streaming (no active_stream_id), has no in-flight turn (no pending_user_message
/ pending_started_at), and its full state is proven on disk (sidecar
message_count >= in-memory count; metadata-only stubs and zero-message shells
are trivially safe).
- Add _evict_sessions_over_cap(): replaces all 8 blind popitem loops across
models.py, routes.py, streaming.py. Walks the LRU oldest-first and removes only
provably-safe entries; never acquires LOCK/stream locks itself (caller holds
LOCK) so no lock-ordering deadlock. May briefly exceed the cap rather than ever
evict an active/unsaved session.
- Make the cap configurable via config.yaml webui.sessions_cache_max
(get_sessions_cache_max()); precedence config.yaml -> HERMES_WEBUI_SESSIONS_MAX
(legacy) -> DEFAULT_SESSIONS_CACHE_MAX=300. No new HERMES_* env var. Invalid or
<1 values fall back so a typo can never disable the bound.
- Evicted sessions lazily reload from their JSON sidecar via the existing
get_session() accessor; no call sites changed. _index.json sidebar behavior
unchanged (the sidebar reads the index, not SESSIONS).
- Add tests/test_issue4765_sessions_lru_eviction.py (8 tests): eviction past
cap, active/streaming never evicted, unsaved/stale-tail never evicted, lazy
reload with identical content, and no-data-loss under heavy churn.
- README: document the config.yaml key + safety semantics.
Fixes#4765.
Phase 2 of cleanup now sweeps _index.json for stale entries with no
backing file and no in-memory session, removing them regardless of
title (catches multi-language 'Untitled' variants). Phase 3 only
deletes the index when Phase 1 removed files AND Phase 2 couldn't
run, fixing the cache-busting on every cleanup call.
Fix two phase-interaction bugs flagged by Greptile review:
- Track phase1_removed_ids to prevent Phase 2 double-counting
sessions already removed from disk by Phase 1.
- Track phase2_rewrote_index so Phase 3 skips deletion when
Phase 2 already cleaned the index in-place.
Adds test_issue5331_index_only_ghost_cleanup.py with 12 test cases.
_session_is_subagent_view_only guard added to the remaining paths that could
mutate a stale persisted subagent sidecar:
- _handle_handoff_summary (appends tool msg to state.db)
- _handle_chat_sync (fallback POST /api/chat)
- /api/session/draft POST (composer_draft save)
- /api/personality/set (per-session personality save)
Fixes#5307
Enumerated every /api/session/<mutation> route and applied _session_is_subagent_view_only:
duplicate, branch, retry, undo, toolsets, compress, archive (existing-sidecar path)
now 400 for subagent children; delete/clear/truncate/pin already gated; rename/move/update
route through the gated _get_or_materialize_session (403). This closes the data-loss
paths (delete_cli_session, retry/undo truncate, branch/duplicate fork).
Also: /api/sessions list coercion now also honors state.db source='subagent' for rows
whose stale index says webui/fork, so they can't surface as writable/CLI sidebar rows.
Fixes#5307
The root non-CLI classification surfaced subagent rows in the sidebar without
read_only, exposing delete/truncate/pin — and /api/session/delete calls
delete_cli_session() which erases the child's state.db transcript (data loss).
- /api/sessions list: coerce subagent rows to read_only=True + is_cli_session=False
so the UI offers no mutation affordances.
- New _session_is_subagent_view_only() shared guard; applied to the direct
mutation routes that bypass _get_or_materialize_session(): delete / clear /
truncate / pin now 400 for subagent children. (rename/move already route
through the gated _get_or_materialize_session and 403.)
- tests: guard helper + static contract that all 4 routes + list coercion present.
Fixes#5307
- api/agent_sessions.py is_cli_session_row(): add 'subagent' to non_cli_sources
so EVERY consumer (sidebar rows, /api/sessions, etc.) classifies a delegated
child as non-CLI (the root the per-site fixes were compensating for).
- api/routes.py GET /api/session happy path: coerce is_cli_session=False +
read_only=True in the serialized payload for a subagent child before redaction,
so a stale writable sidecar can't be exposed as writable to the browser.
Fixes#5307
- _get_or_materialize_session happy path: reject an already-persisted subagent
sidecar (source_tag/raw_source=='subagent' or state.db subagent) even when it
was stored read_only=False (pre-fix materialization), so chat-start can't write it.
- import_cli existing-session refresh: coerce read_only=True on the persisted
sidecar (and response) when it's a subagent child.
- test: persisted read_only=False subagent sidecar is still refused by the helper.
Fixes#5307
- import_cli _read_only_view now also treats resolved cli_meta source_tag/raw_source
=='subagent' as view-only (all_profiles=true resolves cli_meta from a non-active
profile, which the active-profile state.db _sa_child check could miss).
- import_cli existing-session refresh branch no longer hardcodes is_cli_session=True
for a subagent child (both the persisted update and the response payload).
Fixes#5307
Codex round 4 found POST /api/session/archive's missing-sidecar fallback also
calls import_cli_session(). grep confirms exactly 3 import_cli_session() call
sites in routes.py; all 3 now refuse source='subagent' children:
- 4792 _get_or_materialize_session (chat-start) -> PermissionError
- 22382 import_cli endpoint -> read-only view payload
- 13885 archive fallback -> 400 'Subagent sessions cannot be archived'
So no path can materialize a delegated child as a writable WebUI sidecar.
Fixes#5307
Codex found a 3rd writable path: POST /api/chat/start -> _get_or_materialize_session()
materialized source='subagent' as a writable sidecar before the not_claimable guard.
Fix: refuse subagent children (PermissionError) at that shared chokepoint, checked via
_is_subagent_child_session_id(sid) (state.db source, independent of cli_meta) BEFORE the
materialize path — so all three entry points (GET synth, import_cli, chat-start) now
consistently keep a delegated child view-only. CLI/TUI/Desktop materialization preserved.
Tests: materialize helper refuses subagent child + still allows tui.
Fixes#5307
- api/routes.py GET /api/session synthesized response: stop hardcoding
is_cli_session=True; serialize bool(synth.is_cli_session) so a recovered
subagent child stays not-CLI-classified (was overriding the helper's False).
- api/routes.py POST /api/session/import_cli: gate source='subagent' into the
read-only view payload (is_cli_session=False, imported=False) BEFORE
import_cli_session(), so a subagent child can never be materialized as a
writable WebUI sidecar via this endpoint.
- tests: assert import_cli routes subagent children read-only (no materialize).
Both were paths that bypassed the _is_claimable_cli_source denylist. Fixes#5307
Reworked per the Codex gate finding: the first approach widened the client
import predicate, which (a) conflicted with #3603's intentional _isExternalSession
gate and (b) let import_cli persist the subagent child as a WRITABLE, CLI-classified
session that then passed the poll-skip/active-refresh gates.
Corrected to a server-side, view-only recovery:
- api/routes.py: mark source='subagent' as NON-claimable in _is_claimable_cli_source
(both cli_meta and state.db source denylists). A subagent child now resolves via
the not_claimable branch -> read-only Session with its state.db transcript, and
build_session takes is_cli_flag (False for subagent children) so the recovered
session is NOT CLI-classified and can't widen the frontend _isExternalSession gates.
- static/sessions.js: REVERTED to master (no client change needed; #3603 contract intact).
- tests: assert reason=not_claimable, read_only=True, is_cli_session!=True, transcript present;
#2782 deleted-webui 404 preserved; #3603 _isExternalSession contract preserved.
58 tests green (5307 + 3603 + claim-cli + core-data-loss). Fixes#5307
A delegated subagent child (source='subagent' in state.db) usually has no WebUI
sidecar but is registered in the WebUI index sharing the parent's lineage. That
made GET /api/session -> _claim_or_synthesize_cli_session return 'was_webui' ->
404, so the child pane opened empty despite state.db holding messages.
- api/routes.py: add _state_db_session_source() + _is_subagent_child_session_id();
exclude subagent children from the was_webui 404 gate so they recover their
state.db transcript (the #2782 self-heal 404 for deleted WebUI sessions is kept).
- static/sessions.js: add _isSubagentChildSession() + _sessionNeedsServerImportForLoad()
(kept separate from _isExternalSession to avoid widening refresh-gating); the
main session-tap, lineage-segment, and child-open handlers now trigger the
import/merge path for subagent children.
- tests: test_5307_subagent_child_transcript.py (7 tests).
Fixes#5307
Reapply footer control ordering on current upstream/master while preserving the required situational chip renderer. Persist composer_control_order with backend validation, keep the settings descriptions reorder-aware, and make primary/situational chip renderers participate in same-group drag ordering.
Verified with: node --check static/boot.js; node --check static/panels.js; git diff --check; ./scripts/test.sh tests/test_issue4598_composer_control_visibility.py
The embedded workspace terminal spawns a PTY shell that runs arbitrary
commands as the server-process user. check_auth() returns True
unconditionally when no password/passkey is configured (the default
out-of-the-box state), so without a network-scope gate the terminal
endpoints were reachable by any unauthenticated caller able to hit the
port — which on a passwordless public bind is remote code execution.
Apply the same local-origin gate the onboarding/bootstrap endpoints use
(_onboarding_gate_allows) to /api/terminal/{start,input,resize,close} and
/api/terminal/output: with auth disabled, accept only loopback/private
origins, ignore spoofable X-Forwarded-For/X-Real-IP unless
HERMES_WEBUI_TRUST_FORWARDED_FOR=1, and honor HERMES_WEBUI_ONBOARDING_OPEN=1
as the explicit opt-out for a deliberately-exposed server. Auth-enabled
servers (cookie already verified upstream) and genuine same-host clients
are unaffected.
Also fixes a latent test-isolation leak in
test_extension_route_remains_behind_webui_auth: it set HERMES_WEBUI_PASSWORD
but never invalidated the process-wide password-hash cache, so its result
depended on suite execution order (exposed when the new test file shifted
ordering). Invalidate before+after so it reads the env var deterministically.
12 new gate tests in tests/test_cvd3_terminal_local_origin_gate.py.
A public TTS host could 301/302/303-redirect POST /audio/speech to an internal
target (e.g. http://169.254.169.254), and urllib's default redirect handler
would follow it carrying the Authorization bearer — both an SSRF bounce past the
base-url guard and a credential leak. Now uses a no-redirect opener
(_NoRedirectTtsHandler raises on any redirect) via the _tts_open seam. +redirect
rejection regression test. Residual DNS-rebinding TOCTOU (re-resolve at connect)
is a narrower low-severity window noted for follow-up.
The base_url validator accepted any https host; an https URL pointing at an
internal/link-local/loopback/reserved IP (e.g. https://169.254.169.254 cloud
metadata, https://10.x internal) passed the scheme-only check. Now resolves the
host and rejects blocked-target addresses (private/loopback/link-local/reserved/
multicast/unspecified), while still allowing public OpenAI-compatible hosts and
the explicit localhost-over-http dev case. DNS-resolution failure is allowed
(unreachable host can't be an SSRF vector + avoids false-rejecting public hosts
that don't resolve in sandboxed envs). +5 regression vectors.
Add a default-OFF 'project_quick_create_buttons' setting that gates the
per-project '+' quick-create buttons on sidebar project chips. The chip
render site only calls _attachProjectQuickCreateButton when the new
window._projectQuickCreate runtime flag is enabled, so the buttons do
not appear unless the user opts in via Settings.
Mirrors the large_text_paste_as_attachment opt-in plumbing (defaults,
allowlist, settings collect/restore/save, window hydration at both sites,
checkbox row, i18n across all locales).
A visible tab can permanently miss a server-initiated turn (self-wake /
cron / restart hook) when its per-session SSE is momentarily down at the
emit instant. server_turn_started is a fire-and-forget broadcast with no
replay buffer; if the turn also finishes before the tab reconnects, the
on-subscribe self-heal finds no live run in ACTIVE_RUNS and replays
nothing, so the transcript stays stale until a hard refresh.
Extend the self-heal: the (re)subscribing tab reports its last-known
message_count (?known_count); when there is no live run to replay, the
handler compares the persisted count and, if the server is ahead, emits a
lightweight session-updated frame. The frontend syncs incrementally via
the #5189 keepStaleUntilLoaded swap-in-place loadSession path (no
clear+refetch, so the #5177/#5189 blank-gap jump is not reintroduced).
Idle-only and pane-scoped; emits only when the server is strictly ahead.
Extract _turn_transcript_lacks_final_assistant_answer() so settlement
logic can inspect an already-merged transcript. The merge wrapper now
delegates to the pure evaluator without changing behavior.
Fixes#5141
Gate hardening for #5198: the new stream-owner map leaked entries on three
paths that bypass the teardown finally — worker early-return when the stream
was cancelled before startup (streaming.py, gateway_chat.py) and the direct
ACTIVE_RUNS stale-zombie prune (routes.py). Each now calls unregister_stream_owner.
Adds a regression test proving the early-return path no longer leaks.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
Add a user setting 'large_text_paste_as_attachment' (default: on) that lets
users paste large text inline instead of having it auto-converted to .md file
attachments. The toggle lives in Control Center → Chat tab and persists via
settings.json, following the established upstream pattern for boolean settings.
