CI failed on shard 2 across ALL python versions (deterministic, not a flake):
test_list_profiles_prefers_matching_isolated_default_home did
mock.patch('hermes_cli.profiles.list_profiles', ...), whose string target forces
a REAL import of hermes_cli — not installed on the WebUI CI path (bundled agent
absent) -> ModuleNotFoundError. Passed locally only because hermes_cli is
importable here. Per the never-import-agent/hermes_cli-in-a-WebUI-test rule:
inject a stub hermes_cli + hermes_cli.profiles into sys.modules so
list_profiles_api()'s internal
resolves to the stub at the WebUI boundary. Verified: all 24 tests pass with
hermes_cli import BLOCKED (CI simulation).
The reload-based state-dir tests (importlib.reload(api.config) with mutated env)
leaked recomputed STATE_DIR/SESSION_DIR globals + stale references into later
tests in the full sequential run -> 17 spurious failures in cancel/stream/
health/state-isolation suites (all green in isolation). Restoring env + reloading
couldn't fully undo it (fixture/monkeypatch teardown ordering + other modules
holding by-value STATE_DIR refs). Rewrite the 3 tests to import api.config in a
fresh SUBPROCESS under the target env and read STATE_DIR from stdout — truly
isolated, can't pollute the test process. Same assertions (HERMES_HOME-derived
default, unset->platform default w/ cross-check, STATE_DIR override precedence).
Opus flagged a real upgrade gap on the #4449 state-dir contract folded into
#4454: a deployment that relocated HERMES_HOME to a non-default base WITHOUT
setting HERMES_WEBUI_STATE_DIR sees WebUI state move to $HERMES_HOME/webui on
upgrade (empty app at the new address). Add (1) README note documenting the
move + how to keep the old location, and (2) two backward-compat regression
tests pinning the invariants: HERMES_HOME-unset stays at the platform default
(normal installs don't relocate), and HERMES_WEBUI_STATE_DIR takes precedence.
The /api/profiles handler re-imported _is_isolated_profile_mode locally inside
handle_get, which made the name function-local for the ENTIRE handle_get scope
-> the earlier use at the /api/projects branch (routes.py:8000) hit
UnboundLocalError (F823), failing 6 test shards + lint. It's already a
module-level re-export (routes.py:429), so the local re-import is redundant;
removing it lets every in-function use resolve to the module global. Also drop
an unused 'from pathlib import Path' in the new #4449 state-dir test.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex SHIP-WITH-FIXES: non-done terminal paths (apperror/cancel/stream-error
finalize) didn't clear _STREAM_WAS_HIDDEN, so a stale wasHidden=true could be
inherited by a later same-sid reconnect -> spurious notification. Make each
entry stream-owned ({streamId, wasHidden}) so a stale entry from a prior stream
is never trusted (reconnect keeps state only when streamId matches; done reads
via the owned entry), and add _clearStreamHidden(activeSid, streamId) on the
apperror/cancel/finalize-fallback terminal paths (belt-and-suspenders).
A background tab's SSE is throttled by Chromium, so the stream 'done' event
arrives late (after the user returns, document.hidden=false) and the
response-complete notification was suppressed by the live visibility check.
Track per-stream whether the tab was hidden at ANY point during the stream
(_STREAM_WAS_HIDDEN, set at attach, OR'd by one idempotent visibilitychange
listener, read+cleared at done) and pass forceHidden accordingly.
sendBrowserNotification bypasses ONLY the live visibility gate on forceHidden,
still honoring the user's notifications-enabled setting — so a watched stream
stays silent (matches Slack/Discord/Gmail/Claude) but a backgrounded one
notifies. Correct fix for the #4416 root cause; supersedes #4421's always-fire
approach (which would also fire when the tab was never hidden).
Co-authored-by: bergeouss <bergeouss@users.noreply.github.com>
Gate-found (Opus SHIP-WITH-FIXES; Codex SAFE — split resolved by reading the
contract: no dropped-turn, but a real transient start-race flicker). #4424
removed the _sendInProgress guards entirely; that re-opened the #2689 start-race
SYMPTOM for the one session mid-send (a /api/sessions poll landing during the
/api/chat/start round-trip would blank the just-sent bubble + queue a spurious
force-reload). Re-add the guard SCOPED to sid===_sendInProgressSid only, in both
_reconcileActiveSessionIdleStateFromList and _purgeStaleInflightEntries — a
long-hung session has _sendInProgress===false so it still gets unstuck (#4354),
only the in-flight start window is protected. Also fixes Opus's test-masking
finding: test_inflight_send_start_race now strips JS comments before asserting +
requires a real , so a comment can't green-light the #2689 guard, and
adds coverage for the reconcile path.
Co-authored-by: bergeouss <bergeouss@users.noreply.github.com>
Gate-found (Codex CORE, repro'd): a hardlink at a clean *.md page name can
carry an arbitrary inode (incl. one outside the wiki) through the O_NOFOLLOW +
inode-identity read check — O_NOFOLLOW does not block hardlinks. Reject any
page file with st_nlink > 1 in BOTH the allowlist walk and the read-path
revalidation. Pre-existing on master (the inode recapture predates this PR's
cache); closing it here while on the read path. +1 toothed regression test
(prime cache -> hardlink listed page to outside secret -> assert 404, no leak).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Gate-found: reaping on started_at age alone could pop a genuinely-live worker
that's mid-teardown past the 180s ceiling (active_stream_id cleared during final
writeback, STREAMS entry still present) — ACTIVE_RUNS is worker-lifecycle truth
for health/background-wakeup/agent-cache consumers. Add a 'not in STREAMS'
guard (snapshot taken sequentially before ACTIVE_RUNS_LOCK, no nesting) and pop
by the real dict key. +1 regression test (toothed). Keeps the anti-permanent-409
age behavior intact.
Co-authored-by: ai-ag2026 <ai-ag2026@users.noreply.github.com>
The pre-session toolset comment pushed the #2518 provenance comment past the
test_issue2518 4000-char window assertion. Trim the comment to one line and
widen the test window to 5000 (the fallback block it actually guards is well
within range; newSession() has legitimately grown).
Gate-found: staged _pendingSessionToolsets could leak into a later New Chat
started from an existing session (newSession forwarded it whenever it was an
array; loadSession never cleared it). Gate the forward on !S.session and clear
the staged value when a real session loads. +2 regression tests.
Co-authored-by: claw-io <claw-io@users.noreply.github.com>
Codex gate caught that the new load_pool() path runs under the process-default
profile (sibling endpoints /api/providers + /api/models/live already wrap).
Without it a multi-profile client sees+seeds the default profile's pool
(#4247/#4067 profile-isolation class).
Codex+Opus gate caught that the forward-scan-only keep produced [user, _recovered
user, assistant] -> adjacent users -> strict-provider 400 once the anchoring
assistant's prev kept neighbour was a user. Keep a recovered user ONLY when it
separates two assistants (prev_kept==assistant AND next==assistant); mirror in
_api_safe_message_positions. + 3 regression tests.
6-round RCE-class security hardening, converged. Codex SAFE (live exploit probes) + Opus SAFE (flag-off-reachability enumeration) + full suite 9508 on the byte-identical gated head. Nathan signed off.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Closes#3879Closes#3777
Codex CORE findings on the fingerprint: (1) default busy timeout could stall
/api/sessions ~10s on a locked state.db; (2) COUNT(*) forced a full messages
table SCAN (~16-22ms/refresh on 200k rows). Fix: open file:...?mode=ro with
timeout=0.05 + PRAGMA busy_timeout=50 (returns fast, falls back to stat stamp on
lock), and use index-only MAX(rowid) per table (no COUNT, no scan). Verified:
EXPLAIN shows SEARCH not SCAN, 0.36ms on 200k msgs, 0.16ms while EXCLUSIVE-locked.
Fingerprint shape is now (sessions_max_rowid, messages_max_rowid); regression
tests updated.
Codex root-cause consult confirmed: the CLI-session cache (_CLI_SESSIONS_CACHE,
5s TTL) and session-list cache were keyed on (st_mtime_ns, st_size) of state.db
+ WAL sidecars. Under WAL-mode writes those stamps collide (same mtime-ns bucket
+ WAL frame size after checkpoint), serving a stale cache -> freshly-committed
gateway/CLI session intermittently absent from /api/sessions (the recurring
test_gateway_sync flake AND a real up-to-5s sidebar delay for gateway writes).
PRAGMA data_version is NOT usable here (fresh per-request connection always
reports its own initial value, never advances — verified). Fix = a cheap
content fingerprint (COUNT/MAX(rowid) over sessions+messages) read on a fresh
connection, which DOES advance on every commit and is immune to mtime
granularity. Wired into _sqlite_file_stat_cache_key (models) + the routes-layer
source stamp. + 3 regression tests + corrected the earlier false root-cause
comments.
ROOT FIX (product bug): POST /api/settings never invalidated the session-list
cache, so toggling show_cli_sessions/show_cron_sessions could serve a stale
sidebar for up to the cache TTL when the settings-file mtime stamp collided with
a cached entry. The handler now clears the cache directly on those toggles. This
is also the root cause of the recurring test_gateway_sync CI flake (session
absent from /api/sessions right after the visibility toggle).
DEFENSE-IN-DEPTH: cleanup_test_sessions fixture now resets show_cli_sessions to
default BEFORE each test too, not only in teardown.
FLAKE FIX (test infra): bump 31 node-subprocess correctness-harness timeouts from
10s -> 30s across 23 test files. Node spawn + V8 warmup under parallel CI shard
load can exceed 10s, causing TimeoutExpired flakes (e.g. test_1325 on #4445).
Only real node subprocess.run() sites changed — mock signatures, HTTP urlopen,
and deliberate timeout-behavior tests (test_api_timeout, timeout=0/1/2) untouched.
Codex re-gate found two issues in the uncapped-read fix: (1) open_anchored_fd
opens blocking O_RDONLY, which HANGS on a workspace FIFO/special file swapped in
at a regular-file path; (2) the non-regular fstat branch returned without
closing the fd (leak). Fix: lstat the symlink-resolved target BEFORE any open to
skip non-regular leaves (no blocking open on a FIFO), and close the fd on the
non-regular fstat branch. Adds a threaded FIFO-no-hang regression test.
Codex regression gate found that routing workspace reads through
read_file_content() introduced a 400KB MAX_FILE_BYTES cap not present on master
(Path.read_text), so a large but legitimate workspace file raised ValueError ->
caught -> None -> the rollback diff falsely reported it as DELETED. Replace with
an uncapped anchored O_NOFOLLOW read (safe_resolve_ws + open_anchored_fd +
S_ISREG fstat check) that keeps the symlink-escape protection but has no size
cap. Adds a >400KB regression test (RED on the capped path, GREEN with the fix).
Co-authored-by: Hinotoi-agent <Hinotoi-agent@users.noreply.github.com>
Absorbs #4412 (tomas-forgac) rebased onto fresh master, plus two
Codex-regression-gate fixes: (1) use the scheme that actually answered the
probe for the ready/already-up URL + browser-open, (2) honor an explicit
HERMES_WEBUI_HEALTH_URL override in the WSL autostart probe.
Co-authored-by: tomas-forgac <tomas-forgac@users.noreply.github.com>
Closes#4412
Codex regression gate found that _session_model_identity_matches normalized
only the @provider:model form, not provider/model (slash). A session stored as
'deepseek/deepseek-v4-1m' resolving to bare 'deepseek-v4-1m' was treated as a
model change -> the 256k fallback bypassed the accept-guard and clobbered the
persisted 1M window (the same #4248 bug on slash-qualified IDs). Mirror the
slash-aware split already used by _model_matches_configured_default. Adds a
route-level regression + unit test for the slash shape.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Restore previously used pathname-based shutil.copy2() which follows an in-tree
symlink whose target escapes the workspace. Route restore writes through the
workspace-anchored file helpers so a restored file always lands inside the
workspace tree. +regression test (escaping symlink refused, normal restore ok).
Co-authored-by: hinotoi-agent <paperlantern.agent@gmail.com>
