Codex re-gate caught 2 defects in the prior UX commit:
1. streaming.py:8921 — the disclosure fired for ANY process_wakeup failure (e.g.
rate-limit) even when no pause was recorded -> falsely claimed 'retries paused'
while process_wakeup_pause stayed empty. Now gated on the record_* return value.
2. streaming.py:10130 — the raised-exception credential_pool_empty path (exercised
by the existing agent test) recorded the pause but omitted the disclosure. Added
the same conditional disclosure there.
Both paths now keep _error_payload['hint'] in sync with the persisted bubble, and
the disclosure appears only when a pause was actually recorded. 69 tests green.
Fable UX-gate follow-up on #5732: after the first credential-exhausted error,
automatic wakeups are silently suppressed. Disclose it in the wakeup-source
error card ('Automatic retries for this conversation are paused until you send
a message, switch the model/provider, or fix the credentials.') so the silence
reads as intentional, not a stuck agent — Gmail/Dropbox account-needs-attention
convention. Scoped to the process_wakeup + credential_pool_empty seam only.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
WebUI hard token-cap on protected-tail tool-result payloads in the MODEL CONTEXT
only, applied once after a confirmed compression event, to prevent post-compression
context blowup. Deep-copy content-replace only (no message removal/reorder → no
orphaned tool_result); visible transcript (s.messages) untouched.
Deep-review re-gate: Codex SAFE (pairing-safety + context-only + idempotent verified
by reproduction), 65 own + 992 regression tests green. Crown-jewel streaming path.
Co-authored-by: t <a@b>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Steering a live gateway-owned run had no process-local agent in SESSION_AGENT_CACHE,
so it errored with "No agent cached" and dropped the typed text. Now detects the live
gateway ACTIVE_RUNS stream and queues the steer as the next turn (model/provider/profile
snapshotted first), updates the queue badge, clears the draft, shows a toast — without
cancelling the stream. Gateway sibling of the shipped #5597 fallback.
Codex SAFE + Fable SHIP-UX; byte-identical to PR head; 66 steer/busy-input tests pass;
Nathan visual-approved (desktop + mobile screenshots).
Co-authored-by: t <a@b>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
The per-token hot path mirrors partial text, reasoning, and live tool
calls into STREAM_PARTIAL_TEXT / STREAM_REASONING_TEXT /
STREAM_LIVE_TOOL_CALLS without holding STREAMS_LOCK, while snapshot
readers (_snapshot_and_append_partial_on_error, cancel_stream) read them
under STREAMS_LOCK. This is safe under the CPython GIL for two reasons —
NOT because `+=` is one bytecode (it is not: `d[k] += s` compiles to
load / add / STORE_SUBSCR):
1. Single writer — only the streaming thread ever writes a given
stream_id's buffers, so there is no writer/writer race to tear.
2. Reader/writer atomicity — `+=` builds a complete new immutable str,
then the final STORE_SUBSCR binding it into the dict is a single
atomic bytecode, so a concurrent reader's dict.get returns either
the old or new *complete* string (strings are immutable), never a
half-built one. list.append / single-key dict writes are atomic too.
So a reader sees a complete-but-possibly-stale value, never a torn one,
and the snapshot is a best-effort partial reconciled by later journal/SSE
events.
Chose documentation over locking: pulling the writes under STREAMS_LOCK
would add per-token contention against readers copying large buffers and
entangle the documented LOCK -> STREAMS_LOCK ordering (deadlock risk) for
only a recoverable staleness window. Make the assumption explicit at the
write sites and the reader instead. No behaviour change.
Rollback: revert this commit (comments only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The streaming worker called meter().begin_session(stream_id) but never a
paired end_session. A turn that produced zero output tokens (pre-flight
cancel, or a setup exception before the first token) left a _SessionMeter
in GlobalMeter._sessions forever: get_stats() only prunes sessions with
first_token_ts > 0, so a zero-token session is never reclaimed, and each
leaked entry inflates the `active` count sent over the SSE metering event.
begin_session() and the metering ticker's .start() are now registered as
the first statements inside the worker's outer `try`, and paired with an
idempotent meter().end_session(stream_id, 0) plus a deterministic
_metering_stop.set() in that try's outer `finally` (the same block that
pops STREAMS/CANCEL_FLAGS) — so every exit path tears the session down.
Deferring .start() until after put() is defined also closes a latent
start-before-put ordering window in the ticker closure.
The metering payload is unchanged; end_session only pops the session.
Verification: tests/test_metering_session_lifecycle.py (leak reproduction,
end_session reclaim, idempotency, begin->cancel-before-token->empty) plus a
plain-assert run on python3.11 (conftest gates pytest to 3.11-3.13; system
python is 3.14). py_compile clean.
WARNING streaming-contract: adjusts the streaming worker's begin/end
teardown ordering. Needs RFC/contract review against docs/rfcs/
(session-sse-contract-v1.md) before merge — draft PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strict providers (DeepSeek v4, newer OpenAI) reject an assistant message
carrying tool_calls: [] with HTTP 400. _sanitize_messages_for_api already
strips orphaned tool results, but not an assistant message that literally
stores an empty tool_calls list.
Drop the empty key on ALL THREE sanitized-dict sites that must agree:
_sanitize_messages_for_api, _api_safe_message_positions, and the
_safe_projection inside _restore_reasoning_metadata (so a tool_calls: [] row
projects identically on both sides of the metadata-carry-forward alignment and
doesn't lose its reasoning/id/timestamp every turn — Fable gate follow-up,
folded in).
Added mutation-checked regression tests: empty tool_calls dropped + populated
chain preserved (both sanitizer paths), and metadata carry-forward survives a
tool_calls: [] row.
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
Per Greptile's review on #5717: the previous
`if result.get('messages') is not None` guard created an asymmetric
condition where the silent-failure classifier could miss the injected
fallback when the agent result had `messages: None` (which never
actually happens in practice since `finalize_turn` always returns a
list, but the asymmetry read as code smell). Drop the guard; the
writeback is now unconditional on `isinstance(result, dict)`.
Per the maintainer's review on #5717: there may be a
`finalize_turn` invariant on the agent branch this WebUI ships
against that already appends the fallback as an assistant row, in
which case the live bug is gone and this PR becomes a regression
guard. Reframe the test docstrings as dual-purpose pins of the user-
visible closure-text contract so they read correctly under both
interpretations.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
The WebUI already surfaces the model-generated summary when the agent's
tool-iteration budget is exhausted. The remaining gap: when the agent
hits the limit but its summary call yields no usable assistant content
(common with reasoning-only responses), the user sees a bare
`tool_limit_reached` error instead of any closure text.
`hermes-agent.handle_max_iterations()` always returns a non-empty
`final_response` (either the summary or a graceful fallback), and
`AIAgent` exposes that string on the result dict. The WebUI, by
contrast, only reads `result['messages']`, so an empty summary
left no final assistant answer and routed to the error path.
Inject `result['final_response']` as a final assistant turn when
`max_iterations_reached` fires and no usable answer exists in
`messages`. The existing `_mark_latest_assistant_tool_limit_status`
flow then attaches the status card and the user sees the closure
text instead of the bare error. Hermes-agent parity preserved (it's
literally the same string); the bare-error path stays for the case
where the agent died entirely (no fallback either).
Tests cover: fallback-injection surfaces closure text + status card,
no double-injection when a real summary already exists, the existing
'died entirely' apperror path is unchanged, and three unit cases for
the helper's skip conditions.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
The streaming settlement path could show a misleading "No response from provider"
error card AFTER a completed answer, because a stale terminal/partial result flag
still forced the generic no_response error path even when the merged current turn
already had a final assistant answer. Now the generic no_response terminal error
is suppressed ONLY when the already-merged current turn has a completed assistant
reply (scoped to soft `partial` results per the issue-thread recommendation),
letting normal `done` settlement continue. Real silent failures, partial
failures, auth/quota errors, cancellation, replayed rows, and terminal
tool/compression cases all keep their existing error behavior.
Self-built stage of contributor PR; gate-pass. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
'All 0 credential(s) exhausted for <provider>' did not match _is_quota_error_text
so it fell through to a generic 'Error' with no hint (the exception path's else
branch discarded the hint entirely). Adds a distinct credential_pool_empty
classification (ordered before the quota check) + an explicit exception-path
branch + an actionable pool-specific hint, so the user sees why the turn failed
and what to do. Does not over-match (doesn't swallow real quota errors) and
doesn't block partial-harvest; downstream tolerates the new type.
NOTE: this is the CLASSIFICATION half of #3929 only — the data-loss half
(reasoning/partial-work restore on reload-reconcile) is a scoped follow-up that
needs the reporter's server logs.
Self-built (nesquena-hermes); nesquena independent review APPROVED.
Codex round-3 gate finding: _dedupe_replayed_context_messages deep-copies the
stale-user repaired boundary row (streaming.py:4501) into the context array. When
the mint ran AFTER dedupe, that deep-copied context row was no longer the shared
result dict, so it stayed id-less while the display copy got the minted id —
re-opening the cross-array id divergence for the stale-repair path. Fix: move
_assign_stable_message_ids() to run immediately after _restore_reasoning_metadata
and BEFORE _dedupe_replayed_context_messages at all four commit sites (streaming
main / in-band self-heal / except-path self-heal + routes runs/MoA), so the shared
result rows are stamped before any deep-copy — both arrays inherit the id.
Codex/Opus gate finding: in eager session-save mode the current user turn is
checkpointed into s.messages before the agent runs (no id yet); after
_assign_stable_message_ids stamps the returned result row, the display merge
keeps the durable eager checkpoint and skips the stamped result — leaving the
display row id-less while its context twin carries the minted id, silently
defeating id-based fork/truncate alignment for eager-mode users. The merge now
copies the minted id onto the kept checkpoint when it lacks one (guarded so it
can't overwrite or duplicate). Opus verified cross-array id pairing holds across
the real 3-turn transform pipeline.
WebUI keeps two parallel arrays per session: messages (display transcript) and
context_messages (what's sent to the model). In large/compacted sessions they
diverge. Forking / in-place truncation copies a prefix of both and relies on
truncate_context_for_display_keep() to translate a display keep-index into the
matching context index — an aligner that prefers a stable per-row `id`. But the
model-context rows carried neither id nor timestamp (a 989-session live scan
found 0 with any id), so alignment fell back to fragile content-signature
matching that goes ambiguous on repeated tool calls / empty assistant turns.
This closes the DATA gap (the alignment logic itself already prefers `id` and
handles the compacted case via the matcher, shipped in #5563): mint a monotonic,
session-unique integer `id` on the per-turn result rows AFTER the context restore
and BEFORE both arrays are built, so the display and model-context copies of a
logical row share the same id.
- api/streaming.py: new _assign_stable_message_ids(); _restore_reasoning_metadata
carries `id` forward across turns (as it already does timestamp); wired into all
three streaming commit sites (main turn, retry, self-heal).
- api/routes.py: same mint on the runs/MoA _handle_chat_sync commit path.
- api/gateway_chat.py: mints ids on the two new rows of a gateway turn.
- The `id` is stripped before the provider API call (not in _API_SAFE_MSG_KEYS),
same treatment as timestamp — nothing new reaches the provider.
session_ops.py alignment was already delivered by #5563 (id-preferring matcher +
compacted-case fall-through), so no change is needed there now; tests updated to
assert the post-#5563 behavior (id-bearing = exact cut; id-less = errs toward
under-keeping, never the old raw-index mis-cut).
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Root cause of the silent-crash-after-hours cluster (#4765/#2233/#4633): the
global in-memory SESSIONS LRU evicted with a blind popitem(last=False), which
could drop an actively streaming or not-yet-persisted session (data loss) and
was capped only via an env var. On long-running installs the effective result
was unbounded RAM growth until segfault.
- Add _session_is_evictable(): a session is evictable ONLY when it is not
streaming (no active_stream_id), has no in-flight turn (no pending_user_message
/ pending_started_at), and its full state is proven on disk (sidecar
message_count >= in-memory count; metadata-only stubs and zero-message shells
are trivially safe).
- Add _evict_sessions_over_cap(): replaces all 8 blind popitem loops across
models.py, routes.py, streaming.py. Walks the LRU oldest-first and removes only
provably-safe entries; never acquires LOCK/stream locks itself (caller holds
LOCK) so no lock-ordering deadlock. May briefly exceed the cap rather than ever
evict an active/unsaved session.
- Make the cap configurable via config.yaml webui.sessions_cache_max
(get_sessions_cache_max()); precedence config.yaml -> HERMES_WEBUI_SESSIONS_MAX
(legacy) -> DEFAULT_SESSIONS_CACHE_MAX=300. No new HERMES_* env var. Invalid or
<1 values fall back so a typo can never disable the bound.
- Evicted sessions lazily reload from their JSON sidecar via the existing
get_session() accessor; no call sites changed. _index.json sidebar behavior
unchanged (the sidebar reads the index, not SESSIONS).
- Add tests/test_issue4765_sessions_lru_eviction.py (8 tests): eviction past
cap, active/streaming never evicted, unsaved/stale-tail never evicted, lazy
reload with identical content, and no-data-loss under heavy churn.
- README: document the config.yaml key + safety semantics.
Fixes#4765.
