Steering a live gateway-owned run had no process-local agent in SESSION_AGENT_CACHE,
so it errored with "No agent cached" and dropped the typed text. Now detects the live
gateway ACTIVE_RUNS stream and queues the steer as the next turn (model/provider/profile
snapshotted first), updates the queue badge, clears the draft, shows a toast — without
cancelling the stream. Gateway sibling of the shipped #5597 fallback.
Codex SAFE + Fable SHIP-UX; byte-identical to PR head; 66 steer/busy-input tests pass;
Nathan visual-approved (desktop + mobile screenshots).
Co-authored-by: t <a@b>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Phase 1 low-risk backend batch (on top of Batch A). Both byte-verified vs PR head,
Codex SAFE, 64 own + 749 regression tests pass, coexists with Batch A's #5833.
- #5828 (ai-ag2026): frame CORS OPTIONS preflight with Content-Length: 0 (keep-alive hang fix)
- #5834 (ai-ag2026): parse session _index.json from bytes not decoded str (all 10 sites)
Co-authored-by: t <a@b>
Co-authored-by: ai-ag2026 <ai-ag2026@users.noreply.github.com>
Both gateway SSE read loops used urlopen(timeout=600) and re-checked cancel only
BETWEEN lines, so a half-open gateway (TCP open, zero bytes) pinned the worker
for the full 10 minutes and ignored Stop.
Route raw-line iteration through _iter_sse_lines_cancellable and bound the socket
to a byte-silence budget (_gateway_read_timeout_secs, default 120s, env
HERMES_WEBUI_GATEWAY_READ_TIMEOUT). A read timeout is TERMINAL — CPython's
socket.makefile latches _timeout_occurred on the first socket.timeout, after
which every read raises a bare OSError, so the connection can't be resumed. On a
read timeout (or that poisoned-socket OSError) the generator surfaces the user's
Stop (yield b"" -> caller emits cancel) or re-raises so the caller reports the
stall. A stream that keeps emitting bytes within the window never trips it, so a
genuinely alive (if slow) turn is untouched; only >budget of total byte-silence
is treated as a dead gateway. Both loops (runs-API + legacy) changed identically.
The test fake now models the timeout-poison latch (a bare OSError after the first
timeout) so it reflects real socket semantics; the previous stateless re-raise
had made a multi-window "recovery" path look green that cannot occur in
production.
Refs #2476
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-token hot path mirrors partial text, reasoning, and live tool
calls into STREAM_PARTIAL_TEXT / STREAM_REASONING_TEXT /
STREAM_LIVE_TOOL_CALLS without holding STREAMS_LOCK, while snapshot
readers (_snapshot_and_append_partial_on_error, cancel_stream) read them
under STREAMS_LOCK. This is safe under the CPython GIL for two reasons —
NOT because `+=` is one bytecode (it is not: `d[k] += s` compiles to
load / add / STORE_SUBSCR):
1. Single writer — only the streaming thread ever writes a given
stream_id's buffers, so there is no writer/writer race to tear.
2. Reader/writer atomicity — `+=` builds a complete new immutable str,
then the final STORE_SUBSCR binding it into the dict is a single
atomic bytecode, so a concurrent reader's dict.get returns either
the old or new *complete* string (strings are immutable), never a
half-built one. list.append / single-key dict writes are atomic too.
So a reader sees a complete-but-possibly-stale value, never a torn one,
and the snapshot is a best-effort partial reconciled by later journal/SSE
events.
Chose documentation over locking: pulling the writes under STREAMS_LOCK
would add per-token contention against readers copying large buffers and
entangle the documented LOCK -> STREAMS_LOCK ordering (deadlock risk) for
only a recoverable staleness window. Make the assumption explicit at the
write sites and the reader instead. No behaviour change.
Rollback: revert this commit (comments only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three fixes for the CI failures (lint + test shard 1):
1. B009 lint fix (api/routes.py:6281): Replace getattr(session, 'profile')
with session.profile — the guard on line 6275 already ensures it exists.
2. B905 lint fix (api/routes.py:13843): Add strict=True to zip() of
_draft_stages slices. Both sides are always the same length (shifted
by one), so strict=True is correct.
3. _emit_slow format-arg fix (api/request_diagnostics.py:186-195):
Keep the prefix in the format string and pass the JSON payload as a
separate positional argument. The test at
test_issue1855_request_diagnostics.py:33 does
json.loads(caplog.records[0].args[0]) and the old code embedded the
prefix in the payload string, breaking the JSON parse. Now
self.logger.warning("%s %s", prefix, payload) — wait, actually
self.logger.warning(f"{prefix} %s", payload) — so args[0] is the
raw JSON string, same as the pre-_emit_slow code.
_probe_remote_gateway held _remote_probe_lock only around the cache
read and cache write; the network probe itself (_http_probe per path,
each up to ~2s) ran unlocked. On a cold cache a dashboard that fans out
to N panels fired N concurrent probe sets at the (possibly dead)
gateway, each blocking up to ~6s (#5455, #2476).
Wrap the existing lock in a Condition and add an in-flight set: the
first 'leader' thread marks the base_url in-flight and runs the probe
(via extracted _run_remote_probe), while latecomers wait on the
Condition for the leader's cached result instead of probing themselves.
The leader clears the marker and notify_all()s in a finally block, so
waiters are woken even if the probe raises (no deadlock); a bounded wait
budget lets a waiter fall back to probing itself if a leader hangs.
_reset_remote_probe_cache_for_tests also clears the in-flight set.
Expire the cache from the probe's COMPLETION time, not the leader's
entry time: walking every path of a hung gateway takes
len(paths) * timeout (~6s), which exceeds the 5s TTL, so
`entry_time + TTL` would store an already-expired cache line. Waiters
woken right after would then miss the cache and each re-probe the dead
gateway — collapsing single-flight and regressing latency to worse-than-
serial. A regression test drives a walk longer than the TTL and asserts
still exactly one probe set (the shipped happy-path test kept the walk
well under the TTL and so never exercised this).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
append_run_event() without an explicit seq called _next_seq(path), which
re-read and re-parsed the entire journal file on every append — O(n) per
append, O(n^2) over a run. Direct callers therefore paid a full file
reparse per event; only RunJournalWriter avoided it via its own in-memory
counter.
Introduce a module-level _SEQ_CACHE keyed by run-journal file path. The
per-path _lock_for(path) serializes same-path reserve->append so seqs
stay monotonic and file order matches; both RunJournalWriter and the free
append_run_event now draw from this one cache (_reserve_next_seq), so
seqs stay gapless even when both write the same path — the writer's
private counter is removed to keep a single source of truth. An explicit
caller-supplied seq advances the cache (_note_assigned_seq) so it can
never be re-issued. delete_run_journal() now also evicts cache entries
for the removed session so a run re-created at the same path restarts
numbering at 1.
Guard every structural _SEQ_CACHE access (reserve/note/evict) with a
dedicated _SEQ_CACHE_LOCK. The eviction sweep in delete_run_journal
iterates the whole dict; the per-path locks the append path holds do NOT
serialize against it, so a concurrent append on another session inserted
a fresh key mid-iteration and raised "dictionary changed size during
iteration" (the caller swallows it, but the eviction then aborts and
leaks stale seqs, so a re-created run resumed a stale number instead of
restarting at 1). A stress test races deletes against cross-session
appends and asserts no such error.
Rollback: revert this commit; _next_seq(path) re-read behaviour returns.
No on-disk format change, so existing journals stay compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When cached session has no anchor_activity_scenes and the prefix reader
returns None (e.g. sidecar too large for prefix), the empty-scenes path
did not mark _scene_check_inconclusive = True. This meant an inactive
session with matching message counts would return False (cache fresh)
even though disk may have gained its first scene record.
Fix: the empty-scenes else-branch now sets _scene_check_inconclusive
when disk_meta_quick is None, matching the existing behavior in the
cached-scenes branch (line 3173). The code then falls through to the
full metadata comparison instead of serving a stale cache.
discussion_r3548650345
The connectivity probe schedules retries with setInterval(probe, 3000)
but tore the timer down with clearTimeout(retryTimer). Browsers share a
single numeric handle space between the timer families, so clearing an
interval handle via clearTimeout only worked by coincidence and is
semantically wrong. Match the teardown to how the timer was created.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
