A threading.Timer per RequestDiagnostics spawns one OS thread per request, held
alive until finish() cancels it. Under sustained /api/sessions poll load this
exhausts the per-process thread cap (RuntimeError: can't start new thread) and
the server stops accepting connections. Replace with a single lazy-init
watchdog daemon thread that scans pending diags every ~1s; public API
unchanged. Caps timeout-tracking at 1 thread/process regardless of request rate.
On the default local in-process backend, an approval card whose stream
ended while still pending (cancel/fork/provider-error/completion) left the
agent queue entry dropped and the _pending mirror reconciled away. Clicking
Approve/Deny then sent an approval_id that matched nothing, so the handler
returned a bare {ok:false}; since #4771 the frontend surfaces that as
'Approval response not accepted.' with a stuck card (reported by
@santastabber on v0.51.666 and b3nw).
Distinguish a genuinely stale card (no pending approval for the session ->
benign {ok:true, stale_cleared:true} so the UI clears the orphan) from a
stale-id click made while a DIFFERENT approval is live (still ok:false so it
can never resolve the wrong command -- #527 guard preserved). Frontend
clears the orphan card on stale_cleared even if the displayed id drifted.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
#4771 added an explicit relay-failure 409 (code: gateway_run_unavailable)
to /api/approval/respond so a gateway approval whose run is gone keeps its
card actionable instead of silently failing. But the 409 was emitted
regardless of backend.
On the DEFAULT local in-process backend, every guarded command parks an
entry in tools.approval._gateway_queues (via _await_gateway_decision), and
the local streaming notify callback mirrors it into _pending tagged with
_GATEWAY_MIRROR_FLAG. There is no gateway run and no _STREAM_RUN_IDS entry
by design. _gateway_pending_approval_without_run_id() therefore returned
True for that purely local approval, so the handler 409'd
("active run unavailable") and refused to resolve an approval that resolves
perfectly well locally. Local-backend users hit this on every approval
click (reported by b3nw on Discord; independently filed as #4948 by
claw-io, which also notes the local-runtime trigger).
Fix: gate the 409 on the WebUI actually running the gateway chat backend
(webui_gateway_chat_enabled). Local approvals fall through to the local
resolution path; the legitimate #4771 gateway behaviour is unchanged.
Tests:
- New tests/test_issue4771_local_approval_regression.py drives the real
_handle_approval_respond handler and asserts: local backend -> 200 +
agent thread unblocked; gateway backend -> 409 preserved. These fail
before the fix (409) and pass after.
- The two existing #4771 tests that asserted the 409 are updated to pin
HERMES_WEBUI_CHAT_BACKEND=gateway at respond time, since the 409 is now
gateway-deployment behaviour.
Reported-by: b3nw
Reported-by: claw-io (#4948)
Opus SHIP-WITH-FIXES (no code change required): the comment/docstring overstated
that the session-list-change listener covers ALL structural mutations. In-app
mutations fire it, but externally-driven changes (scheduled cron completion,
external CLI writes) don't — for those the 30s streaming TTL is the load-bearing
backstop (≤30s bounded, self-healing lag). Comment-only accuracy fix; logic
unchanged (both gates: perf fix correct, freeze effective, staleness bounded).
/api/sessions re-ran the expensive CLI/cron session projection on every
poll while a turn was streaming — a major cause of the multi-second sidebar
latency + 100% CPU on cron-heavy installs (continuing #4672/#4808/#4889).
Root cause: the CLI/cron projection is gated by _CLI_SESSIONS_CACHE, whose
key folds in _sqlite_file_stat_cache_key -> _sqlite_content_fingerprint
(MAX(rowid) FROM messages). During a live turn the gateway writes a message
row per streamed delta, so that fingerprint advances on essentially every
~5s poll — busting the cache and re-running the full candidate-join +
projection (and the lineage-metadata pass) on every poll, contending for the
same SQLite/global lock the streaming worker holds.
The route-level session-list cache already froze its key during streaming
(#4808 _session_list_cache_streaming_freeze_marker), but that freeze never
reached this inner CLI-sessions cache, so the heavy CLI/cron query still
re-ran whenever the outer cache validated.
Fix: while any stream is active, the CLI-sessions cache key folds in the same
stable streaming-freeze marker (keyed only on the set of active stream ids)
instead of the volatile content fingerprint, and the cache TTL widens to a
streaming window. The projection is reused across polls mid-stream and
rebuilt at most once per window. The instant a stream starts/stops the marker
changes, so freshly-finished rows surface promptly. Structural mutations
(cron completion, new/renamed/archived sessions, attention) now also clear
the CLI cache via the existing session-list-changed listener — those signals
never fire per streamed token, which is what makes the freeze safe.
Idle behavior is byte-identical (freeze only engages with active streams).
Applies to both the active-profile and all-profiles cache paths.
Adds tests/test_issue4842_cli_sessions_streaming_freeze.py (6 tests): marker
idle/stable/changes semantics, the core 'key stable across message writes
while streaming' guarantee, idle key still advances with the fingerprint,
streaming TTL > idle TTL, and the structural-mutation listener clears the
CLI cache.
Settings -> Extensions offered an Install button even when extensions were
unconfigured, then dead-ended on 'Extensions not configured' because
install_extension required HERMES_WEBUI_EXTENSION_DIR to be set to an existing
directory. Default to a WebUI-managed root under STATE_DIR/extensions and
bootstrap it on first install so install just works with zero config; the env
var remains an optional override (never auto-created). Gallery-installed
extensions already auto-load via the runtime manifest, so the extension loads
on the next app-shell render with no further setup.
- api/extensions.py: _default_extension_root + _writable_extension_root;
_extension_root/_extension_root_status fall back to the managed default;
install_extension bootstraps via _writable_extension_root; dir-unavailable
warning only fires for an explicitly-set-but-broken env path.
- tests: update disabled-by-default expectations (configured=True, valid=False
pre-install) + new regression proving install bootstraps + auto-loads with
no env.
- docs/EXTENSIONS.md: document zero-config one-click install + env now optional.
- CHANGELOG [Unreleased].
Tool-call args were truncated to 120 chars at three sites (_truncate_tool_args,
the live _tool_args_snapshot in api/streaming.py, and _toolArgsSnapshot in
static/ui.js). For incidental args that's fine, but content/diff-bearing keys
(command/cmd/script/code/patch/diff/old_string/new_string/content/path/
file_path) feed the rendered tool card AND the diff reconstruction in
recovery-rebuilt sessions, so the cap silently corrupted long commands/paths and
produced broken/empty diffs. Exempt those keys with a much larger cap
(_TOOL_ARG_CONTENT_CAP = _TOOL_RESULT_SNIPPET_MAX = 4000) at every site, keyed by
key name; incidental args keep the 120 cap; storage stays bounded. Part of #4925.
format_wakeup_prompt() in api/background_process.py returned None for any event
type other than 'completion'/watch_*, so async_delegation events (produced when
a background delegate_task subagent finishes) were silently dropped — the
subagent showed as perpetually 'running' and its consolidated result never
re-entered the parent conversation.
Add an async_delegation branch that delegates to the agent-side
tools.process_registry.format_process_notification (which already knows the
async_delegation shape), so the subagent result is rendered into the wakeup
prompt and re-enters chat. Scoped to async_delegation specifically (not all
unknown types) so genuinely-unrecognized events are still skipped — preserving
the existing skip-unknown-event contract.
Reported + root-caused by @mydelren.
Co-authored-by: mydelren <mydelren@users.noreply.github.com>
On Windows a reserved-device-name file (nul/con/prn/aux/com1-9/lpt1-9) can land
in the working tree (e.g. a shell command redirecting to `> nul` under Git Bash,
which treats it as a literal filename). git can't delete it via the normal Win32
path, so `git clean -fd` exits non-zero — and force_apply_update treated that as
fatal, aborting the whole force update with 'Failed to remove untracked files
before force reset' and leaving the user stuck.
The clean is now best-effort: a failure is logged (stderr captured for
diagnostics) but non-fatal, because the subsequent `git reset --hard` overwrites
any tracked-file collisions regardless, and residual untracked files git can't
delete are harmless. Updates the prior test that asserted the (buggy) fatal
behavior to the corrected non-fatal contract, plus a new regression test.
Reported + root-caused by @rodboev.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Opus gate finding (BLOCKING): the two #4385 archive-reappears regression tests
mocked Session.load_metadata_only to return an archived stub WITHOUT writing a
real sidecar file, and never set SESSION_DIR. The new stat-gate in
_state_projection_sidecar_metadata short-circuits on SESSION_DIR/{sid}.json
before consulting metadata, so those mocks were never reached -> archived=False.
This is NOT a production regression (in prod, load_metadata_only reads exactly
SESSION_DIR/{sid}.json and returns None when absent, so archived metadata only
exists when the file does — in which case the stat() succeeds and behavior is
identical). The divergence existed only at the tests' mock boundary.
Fix: both tests now write a real archived sidecar JSON under a monkeypatched
SESSION_DIR, exercising the actual production read path (strictly better
coverage than mocking load_metadata_only). Also documented the
SESSION_DIR-as-sole-source assumption on the cache function per Opus's
forward-looking note.
GET /api/sessions pinned CPU to 100% and took multiple seconds on profiles with
many cron sessions. The state.db -> sidebar projection did three pieces of
redundant per-row file I/O, in BOTH the visible pass and the higher-capped
(CRON_PROJECT_CHIP_LIMIT=200) cron-only second pass:
1. _state_projection_sidecar_metadata(sid): uncached open() + 64KB prefix read +
JSON-key scan, once per row.
2. get_last_workspace(): up to two file reads + an is_dir()/remote probe,
returning the SAME workspace for every row.
3. cron/jobs.json: full read + parse per untitled cron row.
On a cron-heavy install that is hundreds of file reads per build, and because
the enclosing _CLI_SESSIONS_CACHE is keyed on a state.db content fingerprint
that advances on every streamed message row, the whole scan was re-paid on
essentially every 5s poll during a live turn (the recurring 100% CPU / slow
/api/sessions from #4672 -> #4808 -> #4842).
Fix:
- Memoize _state_projection_sidecar_metadata by the sidecar's
(path, mtime_ns, size, ctime_ns) stat signature in a bounded module cache, so
a warm build re-stats instead of re-reads. A rename/archive/edit bumps the
signature and invalidates just that one entry (no staleness). This survives
the 5s CLI-cache busting during streaming, which is the actual trigger.
Mirrors the Claude Code parse cache (#4718). Folded into
clear_cli_sessions_cache() for explicit-reset/test parity.
- Build-scoped memos for cron jobs.json (parse once -> job_id->name map) and
get_last_workspace() (resolve once), matching the existing _cron_pid() pattern.
Output is byte-identical. Adds tests/test_issue4842_cron_projection_perf.py
(7 tests: warm-rebuild no re-read, stat-change invalidation, dict-copy isolation,
missing-sidecar default, jobs.json once, workspace once, lifecycle clear).
Dual-gate (Codex + Opus) findings on the Claude Code transcript parse cache:
1. Cache-key staleness hardening: add st_ctime_ns to the (path, mtime_ns, size,
max_messages) key so a rare same-size, same-mtime in-place edit can't serve a
stale parse. Size is already the strong discriminator for append-only JSONL;
ctime_ns is cheap defense-in-depth (a spurious bump costs one harmless re-parse,
never a stale read).
2. Timestamp-fallback bug-for-bug fix: the row-builder guard used identity
(first_ts is None and last_ts is None) but the field assignment uses truthiness
(first_ts or last_ts or _mtime). They disagreed on a falsy 0.0 timestamp
(epoch-0/1970 transcripts), leaving created_at/updated_at/last_message_at = None
instead of the file mtime the pre-cache inline code fell back to. Guard now uses
the same 'not first_ts and not last_ts' test.
Adds 3 regression tests: epoch-0 mtime fallback, the shared-dict read-only
contract (pins the load-bearing invariant), and same-size+mtime ctime invalidation.
The /api/sessions cold build (which a profile switch triggers) was
dominated by get_claude_code_sessions() re-parsing every Claude Code
JSONL transcript line-by-line on each call — 650-1000ms on a 200-file
/ ~130MB tree, repeated on every switch, every 5s CLI-cache expiry, and
every sidebar poll, because the directory is global but the higher
session cache is keyed per active profile.
Memoize the per-file parse by (path, mtime_ns, size) so a warm build
re-stats files instead of re-parsing them; any genuine edit/append
changes the stat signature and invalidates just that file. Also hoist
the loop-invariant get_last_workspace() out of the Claude Code row loop
(it stats config + probes terminal cwd, ~200 redundant calls per build).
Measured /api/sessions cold-rebuild on a 250-session profile: ~670-880ms
-> ~100-200ms (verified in-browser). Output byte-identical.
Credit @rodboev for the browser-level profiling that isolated the real
bottleneck (the prior pre-warm hypothesis could not move it).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex reproduced a SILENT regression: the PR's 4096-BYTE read could miss the
continuation marker the old read_text()[:4096] (4096 CHARS) found. Session.save()
writes compression_anchor_summary (ensure_ascii=False) BEFORE parent_session_id,
so a multi-byte emoji summary puts the marker at char offset ~1230 but byte
offset ~4900 — within the old 4096-char window, past a 4096-byte one → continuation
child no longer detected. Fix: read 16384 bytes (4096 chars worst-case for 4-byte
UTF-8) then slice [:4096] chars, keeping the same needle + bounded I/O. Multibyte
regression test added + proven non-vacuous; updated the read-size assertion to <=16384.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
