Codex round-3 gate finding: _dedupe_replayed_context_messages deep-copies the
stale-user repaired boundary row (streaming.py:4501) into the context array. When
the mint ran AFTER dedupe, that deep-copied context row was no longer the shared
result dict, so it stayed id-less while the display copy got the minted id —
re-opening the cross-array id divergence for the stale-repair path. Fix: move
_assign_stable_message_ids() to run immediately after _restore_reasoning_metadata
and BEFORE _dedupe_replayed_context_messages at all four commit sites (streaming
main / in-band self-heal / except-path self-heal + routes runs/MoA), so the shared
result rows are stamped before any deep-copy — both arrays inherit the id.
Codex/Opus gate finding: in eager session-save mode the current user turn is
checkpointed into s.messages before the agent runs (no id yet); after
_assign_stable_message_ids stamps the returned result row, the display merge
keeps the durable eager checkpoint and skips the stamped result — leaving the
display row id-less while its context twin carries the minted id, silently
defeating id-based fork/truncate alignment for eager-mode users. The merge now
copies the minted id onto the kept checkpoint when it lacks one (guarded so it
can't overwrite or duplicate). Opus verified cross-array id pairing holds across
the real 3-turn transform pipeline.
WebUI keeps two parallel arrays per session: messages (display transcript) and
context_messages (what's sent to the model). In large/compacted sessions they
diverge. Forking / in-place truncation copies a prefix of both and relies on
truncate_context_for_display_keep() to translate a display keep-index into the
matching context index — an aligner that prefers a stable per-row `id`. But the
model-context rows carried neither id nor timestamp (a 989-session live scan
found 0 with any id), so alignment fell back to fragile content-signature
matching that goes ambiguous on repeated tool calls / empty assistant turns.
This closes the DATA gap (the alignment logic itself already prefers `id` and
handles the compacted case via the matcher, shipped in #5563): mint a monotonic,
session-unique integer `id` on the per-turn result rows AFTER the context restore
and BEFORE both arrays are built, so the display and model-context copies of a
logical row share the same id.
- api/streaming.py: new _assign_stable_message_ids(); _restore_reasoning_metadata
carries `id` forward across turns (as it already does timestamp); wired into all
three streaming commit sites (main turn, retry, self-heal).
- api/routes.py: same mint on the runs/MoA _handle_chat_sync commit path.
- api/gateway_chat.py: mints ids on the two new rows of a gateway turn.
- The `id` is stripped before the provider API call (not in _API_SAFE_MSG_KEYS),
same treatment as timestamp — nothing new reaches the provider.
session_ops.py alignment was already delivered by #5563 (id-preferring matcher +
compacted-case fall-through), so no change is needed there now; tests updated to
assert the post-#5563 behavior (id-bearing = exact cut; id-less = errs toward
under-keeping, never the old raw-index mis-cut).
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Root cause of the silent-crash-after-hours cluster (#4765/#2233/#4633): the
global in-memory SESSIONS LRU evicted with a blind popitem(last=False), which
could drop an actively streaming or not-yet-persisted session (data loss) and
was capped only via an env var. On long-running installs the effective result
was unbounded RAM growth until segfault.
- Add _session_is_evictable(): a session is evictable ONLY when it is not
streaming (no active_stream_id), has no in-flight turn (no pending_user_message
/ pending_started_at), and its full state is proven on disk (sidecar
message_count >= in-memory count; metadata-only stubs and zero-message shells
are trivially safe).
- Add _evict_sessions_over_cap(): replaces all 8 blind popitem loops across
models.py, routes.py, streaming.py. Walks the LRU oldest-first and removes only
provably-safe entries; never acquires LOCK/stream locks itself (caller holds
LOCK) so no lock-ordering deadlock. May briefly exceed the cap rather than ever
evict an active/unsaved session.
- Make the cap configurable via config.yaml webui.sessions_cache_max
(get_sessions_cache_max()); precedence config.yaml -> HERMES_WEBUI_SESSIONS_MAX
(legacy) -> DEFAULT_SESSIONS_CACHE_MAX=300. No new HERMES_* env var. Invalid or
<1 values fall back so a typo can never disable the bound.
- Evicted sessions lazily reload from their JSON sidecar via the existing
get_session() accessor; no call sites changed. _index.json sidebar behavior
unchanged (the sidebar reads the index, not SESSIONS).
- Add tests/test_issue4765_sessions_lru_eviction.py (8 tests): eviction past
cap, active/streaming never evicted, unsaved/stale-tail never evicted, lazy
reload with identical content, and no-data-loss under heavy churn.
- README: document the config.yaml key + safety semantics.
Fixes#4765.
Extract _turn_transcript_lacks_final_assistant_answer() so settlement
logic can inspect an already-merged transcript. The merge wrapper now
delegates to the pure evaluator without changing behavior.
Fixes#5141
Gate hardening for #5198: the new stream-owner map leaked entries on three
paths that bypass the teardown finally — worker early-return when the stream
was cancelled before startup (streaming.py, gateway_chat.py) and the direct
ACTIVE_RUNS stale-zombie prune (routes.py). Each now calls unregister_stream_owner.
Adds a regression test proving the early-return path no longer leaks.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
Prevent historical assistant reasoning_content from being replayed to
generic/local providers by default while preserving current behavior for
known reasoning-capable provider protocols.
Adds a webui.reasoning_content_replay policy read by the sanitizer:
auto, strip, and preserve. Auto strips for local/generic backends such as
LM Studio/Ollama/llama.cpp and preserves for explicitly known provider
protocols that may require reasoning replay.
This only affects provider-facing sanitized history. It does not modify
stored session/display reasoning metadata.
WebUI's api/streaming.py:_resolve_image_input_mode was a LOCAL
re-implementation of agent/image_routing.py:decide_image_input_mode that
had DIVERGED from the canonical function. The local copy never consulted
the active model's vision capability — it only checked a handful of config
heuristics — so it returned "native" (embedding the image as an image_url
part) even for models KNOWN to lack vision. That silently sends pixels to a
blind model (#21160-class failure) instead of routing through the text
(vision_analyze) pipeline.
Fix: delegate to the canonical agent.image_routing.decide_image_input_mode
(single source of truth), with a try/except fallback that preserves the
historical WebUI behaviour when the agent package is not importable (e.g.
the WebUI standalone test environment).
Carve-out preserved: the canonical router conservatively returns "text" for
UNKNOWN/custom models (no models.dev capability data). WebUI historically
forwards those NATIVELY and relies on the agent's strip-and-retry guard
(_try_shrink_image_parts_in_messages / _strip_images_from_messages) to
downgrade on a provider rejection. We keep that: a canonical "text" verdict
is only honoured when there's a real signal — an explicit user choice
(image_input_mode: text or a configured auxiliary.vision backend) or a model
KNOWN to lack vision. Otherwise we forward native.
Tests:
- Existing gateway test (forwards_image_attachments_as_multimodal_parts)
stays green — unknown "test-model" still forwards native.
- New non-vacuous regression: a model KNOWN to be text-only
(supports_vision == False) now routes "text" — this FAILS against master,
where the old local copy returned "native" for the same config.
- Added coverage for the unknown-model carve-out, known-vision native path,
explicit-text-signal precedence, and the agent-unavailable fallback.
Salvage of #4113 by @gkd2323c. The original PR delegated fully to canonical
but did not add the unknown-model carve-out, which would have regressed the
WebUI native-forwarding behaviour the gateway test encodes.
Co-authored-by: gkd2323c <gkd2323c@users.noreply.github.com>
Adds a dedicated GET /api/commands/moa/resolve endpoint + resolve_moa_config()
that reads the MoA preset/usage server-side. The /moa send-path passes only a
moa_config:true boolean to /api/chat/start; the server re-resolves the real
config itself (never trusts a browser-supplied dict) and threads it per-turn into
the live agent run. Fails closed (409) on gateway-backed sessions.
Replaces the originally-proposed broad 43-command subprocess approach (bounced)
with this narrow, security-reviewed /moa route. Maintainer fixes on top of the
contributor's converged design: moa_config is added to run_conversation only
when not None (older-agent TypeError safety) at the main call AND both auth
self-heal retry sites, and forwarded through the legacy-journal runtime adapter.
Gate: Codex SAFE (after the maintainer fixes) + full suite 10784 pass + the
activity-stream regression gate clean.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
When a turn is steered, the agent embeds an
[OUT-OF-BAND USER MESSAGE]...[/OUT-OF-BAND USER MESSAGE] control block, and that
block gets persisted into the session's messages/context_messages (appended onto
a tool-result entry — verified present in real prod session JSON, in both
messages[] and context_messages[]). WebUI then replays context_messages into the
provider-facing conversation history every turn, so the model kept re-seeing the
internal control marker: wasted tokens + stale control data.
Adds a non-mutating, recursive _strip_oob_blocks() (regex, DOTALL+IGNORECASE,
non-greedy) wired into every model-facing chokepoint:
_sanitize_messages_for_api, _api_safe_message_positions,
_restore_reasoning_metadata projection, and the gateway runs-API
conversation_history (context_messages + prefill). The compression path
(/session/compress) inherits the strip via _sanitize_messages_for_api.
The marker stays intact in the persisted transcript (display/reload/audit
fidelity); it's removed only on the way to the provider. Idempotent, only strips
a complete open+close pair, and does not mutate persisted session objects (the
content key is reassigned to a freshly-built value).
Maintainer hardening (Opus gate): relaxed the opening pattern from a mandatory
`]\r?\n` to `]\s*?` so a single-line OOB block is also stripped (DOTALL .*?
already spans newlines) — robustness against a format that omits the newline.
Co-authored-by: Stacey2911 <Stacey2911@users.noreply.github.com>
Cancelling a running turn briefly collapsed the already-streamed reasoning,
tool cards, and text to a bare "Task cancelled" marker before a follow-up
GET /api/session repopulated them — a visible race between the terminal
cancel SSE event and the refetch (#4076, #1361).
The terminal cancel event now carries a canonical, redacted session snapshot
captured under the per-session agent lock right after the cancelled turn is
persisted (including the recovered partial-assistant message). The frontend
applies that embedded snapshot immediately via _applyCancelSessionPayload and
skips the GET round-trip. The snapshot is a fully detached deep copy (redaction
rebuilds every container) so it cannot alias or be staled by concurrent writes;
it is only applied when its session_id matches the active view; and a
missing/failed snapshot degrades cleanly to the existing GET fallback.
The seven other cancel call sites are standardized via _cancel_event_payload()
(message + type + status), backward-compatible (no session key when none).
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Tool-call args were truncated to 120 chars at three sites (_truncate_tool_args,
the live _tool_args_snapshot in api/streaming.py, and _toolArgsSnapshot in
static/ui.js). For incidental args that's fine, but content/diff-bearing keys
(command/cmd/script/code/patch/diff/old_string/new_string/content/path/
file_path) feed the rendered tool card AND the diff reconstruction in
recovery-rebuilt sessions, so the cap silently corrupted long commands/paths and
produced broken/empty diffs. Exempt those keys with a much larger cap
(_TOOL_ARG_CONTENT_CAP = _TOOL_RESULT_SNIPPET_MAX = 4000) at every site, keyed by
key name; incidental args keep the 120 cap; storage stays bounded. Part of #4925.
Codex re-gate 2: post-run flush is bypassed on exception (jumps to outer except) + retry
paths, so a buffered reasoning tail was lost before apperror there. Added a guarded
_flush_reasoning_buffer() at the top of the inner finally (guaranteed exit for
normal/exception/retry) so the tail always reaches the live Thinking view before teardown.
+ regression asserting the finally flush.
Co-authored-by: wlknight <wlknight@users.noreply.github.com>
Codex re-gate caught a SILENT gap: the tail-flush only ran on on_reasoning(text=None),
but the agent never calls reasoning_callback(None) — so the last sub-100ms coalesced
buffer was silently lost, and a reasoning→tool/output transition stranded the tail.
Added a shared _flush_reasoning_buffer() helper called at every boundary that closes or
reorders the live reasoning stream: on_token (visible output starting), on_tool (tool
boundary), the defensive on_reasoning(None), and right after agent.run_conversation()
returns (terminal catch-all for a turn that ends on reasoning with no trailing token/tool).
Each flush emits the buffered chunk then clears it (since-last-flush delta semantics, so
the frontend's reasoningText += text stays correct). + regression test for all sites.
Co-authored-by: wlknight <wlknight@users.noreply.github.com>
Module-level _compact_for_echo_compare (strips ALL whitespace, \s+ -> '') so _is_visible_output_echo's
endswith() suffix-match detects interim_assistant prose that only differs from visible token prose by
line-break/paragraph formatting. Hoisted to module scope (importable) + behavior-based test (real
token vs interim text equality). Codex SAFE (exact suffix match, not substring). 103 echo/journal tests.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
A failed /steer (or busy-mode steer) no longer silently falls back to interrupt+queue
(which cancelled the active stream). _trySteer now returns delivered:bool, restores the
draft + leaves the stream running on failure; send() only clears staged files when
_steerDelivered===true. Backend hunk comment-only (failure paths already returned
{accepted:False}, never touched CANCEL_FLAGS). i18n value-only on pre-existing keys
(locale-parity preserved, 173 locale tests pass). 48 own tests pass.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: dso2ng <dso2ng@users.noreply.github.com>
In-chat cronjob tool reached cron.jobs whose path constants snapshot at import, so a
chat under a named profile hit the DEFAULT profile's cron dirs. Adds a per-turn
contextvar (_STREAMING_CRON_PROFILE_HOME, reset on every exit) + a tool-call-boundary
wrapper that enters cron_profile_context_for_home() only while the cronjob handler runs
(cron lock, restored immediately — no whole-turn global mutation). Verified: agent
executor's propagate_context_to_thread uses unfiltered contextvars.copy_context() so the
contextvar reaches the sync handler on the ThreadPool worker; reset-token symmetry intact.
Code byte-faithful from PR head c3fd6ab22f8c. 12 own tests pass.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: roian6 <roian6@users.noreply.github.com>
* stage #4618 (allenliang2022): broaden streaming stale-compressor guard to any model-window mismatch + #4618 regression tests + CHANGELOG
Broadens #3256's default-only live-usage guard: the streaming SSE snapshot now
always resolves the real per-model window via the same helper GET /api/session
hydration uses (_context_length_lookup_inputs_for_model + get_model_context_length)
and corrects whenever it differs from the compressor's cached value, with a
TypeError fallback to the legacy 2-arg form. Fixes 'refresh shows 1M, send reverts
to stale 168k + early auto-compress' on model-switched sessions. Per-stream cache
preserved (one lookup/stream). Code byte-identical to PR head 3beb18e92d73.
Adds 4 source-structure regression tests (RED-proven on master).
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
* fix#4618 gate findings: profile-scoped config (Codex cross-profile) + #4248 256k acceptance gate (Opus downward-clobber)
Codex SHIP-WITH-FIXES: live-snapshot used ambient get_config() which in the
detached streaming worker resolves the process-global/default profile (#3294) ->
for a non-default profile pinning a different per-model context_length it would
surface the WRONG profile's window. Now resolves via get_config_for_profile_home
on the session's own profile home (mirrors the worker's _cfg resolution).
Opus SHIP-WITH-FIXES: broadened guard aligned resolution w/ hydration but not its
#4248 acceptance gate -> a transient low-confidence 256k metadata probe could
clobber a LARGER cached window mid-stream. Now reuses the exact hydration helper
_should_accept_session_context_length_refresh on both modern + legacy paths.
+ regression tests for both. Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
* fix#4618 Codex re-gate findings: broaden save + SSE-done stale-compressor guards too
Codex re-gate found the broadened live-snapshot guard fixed metering but the two
SIBLING paths still used the old default-only exact-cap test:
- api/streaming.py final session-save: persisted stale other-model window (168k)
to s.context_length -> wrong window on reload.
- api/streaming.py terminal SSE: emitted stale window -> indicator REVERTS
on stream end (messages.js overwrites S.lastUsage) = the exact 'send reverts to
168k' symptom.
Both now resolve the real per-model window via the same hydration helper and honor
the #4248 acceptance gate (no 256k downward-clobber), with legacy 2-arg fallback.
This is the root-cause completion across all 3 paths (live/save/SSE-done).
+ 2 regression tests. Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
* docs(#4618): note model_changed-omission rationale (Opus) + broaden CHANGELOG to all-3-paths
* Release v0.51.561 — Release TT (context-window indicator stays correct after model switch; #4618)
---------
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
Codex+Opus gate caught that the forward-scan-only keep produced [user, _recovered
user, assistant] -> adjacent users -> strict-provider 400 once the anchoring
assistant's prev kept neighbour was a user. Keep a recovered user ONLY when it
separates two assistants (prev_kept==assistant AND next==assistant); mirror in
_api_safe_message_positions. + 3 regression tests.
Opus review note: the substring match in _stale_prefix_matches_prior_user_context
is intentionally loose, but correctness depends on the caller's
suffix-equals-entire-submitted-turn invariant (cleaning can only ever rewrite a
flagged row to the user's actual current turn — never drop legitimate content).
Comment-only; no behavior change.
When the agent's role-sequence repair concatenates the prior context-tail
user row with the submitted current turn as <stale>\n\n<current>, that
polluted pair was persisted into both the model context and the visible
transcript. Add a _detect_stale_user_merge detector (requires the literal
\n\n repair boundary + exact tail/current match after sentinel stripping)
and funnel context-merge and display-merge through one
_strip_stale_user_merge_from_messages so a single rule governs persistence.
The strip is scoped to the new-turn candidate slice only — historical rows
in the already-committed previous_context prefix are never rewritten.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
The backfill inner loop in _merge_display_messages_after_agent_result
re-ran _message_identity (json.dumps) for every future display row on
every outer iteration and used an O(N) list-slice membership test
(in context_keys[_cursor:]), making it O(D²·C). On long tool-call
sessions this pegged a core, held the GIL, and stalled /api/sessions
5-11s — the WebUI appeared hung. Precompute display keys once and keep
a multiset (count-keyed dict) mirror of context_keys[_cursor:] for O(1)
membership → O(D²).
Salvaged from #4314 (the original PR bundled an unrelated, undescribed
HermesWebUIContext feature). Reimplemented the membership mirror as a
count-keyed dict instead of the PR's set: _message_identity returns
DUPLICATE keys for identical-content turns and None for empty rows, so
a plain set diverges from the original list-slice semantics (verified
over 3M adversarial trials); a multiset including None is exact. Added
a differential regression test guarding the equivalence.
Co-authored-by: psanger <psanger@users.noreply.github.com>
_drain_webui_process_notifications now drops (consumes, never requeues) a
completion whose completed_at is older than a configurable cap
(HERMES_WEBUI_STALE_COMPLETION_MAX_AGE_SECONDS, default 6h; 0 disables).
Fail-open: events without completed_at (legacy agent) are never dropped.
Additive (+192/-0). Closes#4029.
Co-authored-by: allenliang2022
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
* Release PK (v0.51.450): preserve custom provider namespace prefixes in model normalization (#4278)
Token-boundary check in _normalize_provider_id (exact token or prefix
followed strictly by ':' or '/'); explicit aliases for hyphenated
first-party names (google-gemini, google-ai-studio, claude-code); bare-prefix
family detection no longer fires on namespaced model IDs.
Closes#4278. Supersedes #4280 (rodboev) whose '-' boundary re-collapsed
claude-relay->anthropic.
Co-authored-by: b3nw
* gate fix (Codex CORE): add openai-api to _PROVIDER_ALIASES so the :/ token boundary doesn't drop the registered openai-api provider id to '' (would silently repair OpenAI API sessions to the profile default) + 2 regression tests
* gate fix (Opus): gate the streaming send-path bare-prefix loop on '/' not in model so custom namespaces (gemini_cli/, gpt_proxy/, claude-relay/) survive the send path — same #4278 collision Opus found in _apply_profile_provider_context_to_streaming_model; + regression test + CHANGELOG
---------
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Consolidates #4162 + #3944. Aux path: skip the reasoning-disable inject for
routes that 400 on it via _route_rejects_reasoning_extra — hostname-matched
OpenAI / Azure OpenAI / Azure AI Foundry (+ provider aliases openai*/azure*/
azure-foundry/azure-ai*) and OpenRouter Anthropic mandatory-reasoning; keep it
for local/OpenRouter-non-anthropic/other reasoning routes (#4161). Agent path:
rely on the existing agent.reasoning_config={enabled:False} + _build_api_kwargs()
(route-correct per provider profile) rather than re-injecting a generic disable
on top of it; only MiniMax reasoning_split is added (unchanged from master).
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Resolve custom provider API keys from the matched config snapshot and pass them through session hydration plus streaming fallback context-length probes. This prevents authenticated /v1/models endpoints from falling back to the default 256K window and clobbering larger persisted session metadata.
Codex caught a regression: the contributor's _runtime_preferred_base_url preferred
the runtime base_url for ALL non-custom providers, which would clobber an explicit
providers.<id>.base_url override pointing at a different host/port (LM Studio at a
LAN IP, an OpenRouter mirror). Now only prefer the runtime URL when it's the SAME
scheme+host+port as the configured one (the #3895 /v1-dedup case is path-only on
the same host); a genuinely different configured endpoint wins. Added
_same_base_url_endpoint() + 2 regression tests (different-endpoint preserved,
same-endpoint normalization).
Extract _build_partial_message() shared by cancel_stream() and the new
_snapshot_and_append_partial_on_error(); the two provider/exception error paths
in _run_agent_streaming now snapshot accumulated text/reasoning/tool_calls under
STREAMS_LOCK and append a _partial assistant message instead of discarding it.
Closes#3929.
Co-authored-by: b3nw <150195942+b3nw@users.noreply.github.com>
* fix(session): retire stale truncation watermark on new committed turn (#3831)
retry_last / undo_last / the Edit-truncate handler set truncation_watermark
to suppress the *replaced* tail from the append-only state.db merge.
Session.save() deliberately never auto-clears it (#2914), but nothing retired
it when the user then sent a genuinely NEW turn either — so it froze at the old
edit boundary. A frozen watermark then dropped post-watermark state.db rows
whenever the sidecar was later reconstructed empty (recovery/reconcile),
permanently losing the turns sent after the edit (state.db still had them).
Retire a POSITIVE watermark to None once the new user turn is COMMITTED to
session.messages — at the success-merge (3 sites), eager-checkpoint, error/
recovery materialization, and cold-load repair commit points. Not at chat-start:
in deferred mode the new row isn't in messages yet, so a merge in that window
would resurrect the replaced tail (the max-sidecar guard hasn't risen past the
old boundary). Once committed, max_sidecar_timestamp rises past the replaced
tail and the merge suppresses it without the watermark, so retiring is safe.
Cleared to None, never 0.0 — 0.0 is the truncate-to-empty sentinel (#2914) that
must keep blocking all state replay, so the clear is falsy-gated.
Closes#3831
* chore(changelog): clarify watermark-retirement timing to commit-time
Greptile review noted the original phrase "retires the watermark at the
start of a new user turn" was timing-imprecise. The retirement actually
fires when the new turn is durably committed to session.messages —
at the agent-result merge, the eager user-message checkpoint, or the
cold-load recovery commit. Reword for accuracy; semantics unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test(#3831): add regression tests for the two inline watermark-clear paths (greptile P2)
Cover the error/cancel materialization path (_materialize_pending_user_turn_before_error)
and the eager first-turn checkpoint path (_checkpoint_user_message_for_eager_session_save),
which inline the falsy-gated watermark clear instead of calling the tested helper.
The error path is precisely the #3831 failure mode (recovery/reconcile after a
crash), so a dedicated regression test closes that gap. Both assert a positive
watermark clears to None while the 0.0 truncate-to-empty sentinel (#2914) is
preserved.
---------
Co-authored-by: nesquena-hermes <[email protected]>
Co-authored-by: Nathan Esquenazi <nesquena@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* stage bg_task trio combined (#2979 superset) on master for deep review
* fix(bg_task): unsubscribe SessionChannel on header-write failure (Codex deep-review catch) + regression test
* test: realign on-subscribe-recovery anchor to subscribe_to_session_channel after leak fix
* CHANGELOG: bg_task trio as v0.51.340 LD (HELD pending independent review)
* bg_task trio: apply 3 independent-review (greptile) fixes
1. start_session_turn now threads the session PROFILE model defaults
(_read_profile_model_config) into the wakeup model-resolve, so a brand-new
session with an empty model falls back to the profile default not global
DEFAULT_MODEL. Updated the white-box spy test signature accordingly.
2. /api/session/stream omits the Connection header (HTTP/1.1 keep-alive
default) to match the #3103 long-lived-SSE pattern.
3. Reaper now prunes _LAST_EMIT_TS for collected sessions so the coalesce
timestamp map can't grow one permanent entry per session forever.
nesquena APPROVED the PR; these are the 3 non-blocking greptile suggestions.
* test: realign _start_session_turn adapter stub lambda to new profile-defaults signature
* fix(streaming): make per-token inline-thinking extraction linear (#3633 follow-up)
Codex post-merge perf catch on #3633: _parseStreamState() and
syncInflightAssistantMessage() call _extractInlineThinkingFromContent on the
FULL accumulated assistantText on EVERY streamed token. The #3633 rewrite made
that a full char-by-char walk, so cost was O(n^2) over a stream — a Node harness
measured ~88s (no-tag) / ~103s (leading <think> block) for 2000x100-char tokens,
which would freeze the main thread on long reasoning-model responses.
Two fixes (Python api/streaming.py + JS static/messages.js twin, line-for-line parity):
1. Fast path: if the text contains no complete thinking opener AND (when
streaming) its tail is not a prefix of an opener, return unchanged without the
char walk — two cheap substring scans. Handles the common no-tag case.
2. Bulk-skip plain trailing content: track the next complete opener via
str.find/indexOf (_next_inline_thinking_opener / _nextThinkingOpener); once no
opener remains ahead, append the remainder and stop instead of walking it
(streaming still suppresses a trailing partial-opener prefix). Handles the
leading-block-then-long-answer case.
Result: ~88s/103s → ~0.5s/0.9s (Python), ~0.18s/0.21s (JS). All behavioral cases
(persist + streaming, code-awareness, position-aware unclosed, leading whitespace)
verified unchanged in both twins. Added a per-token streaming perf regression test
and wired _nextThinkingOpener into the node driver harness.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* fix(streaming): perf bulk-skip must respect code context for partial-opener tails
Codex catch on the perf fix: the no-complete-opener bulk-skip suppressed a
trailing partial opener (e.g. '<thi') unconditionally during streaming, but a
partial opener INSIDE inline-backtick / fenced / indented code must stay visible
(master parity). Now, when streaming and the tail is a partial opener, fall
through to the code-aware char walk (bounded — a partial tail is a transient
single token) instead of bulk-skipping; only a PLAIN-text partial opener is
suppressed as a forming block. Added _text_tail_is_partial_opener /
_textTailIsPartialOpener (Python + JS parity) + regression tests for the
inside-code vs plain partial-tail cases.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* docs(changelog): v0.51.336 KZ — inline-thinking streaming perf fix
---------
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Co-authored-by: Hermes Agent <hermes-agent@nesquena-hermes.local>
* fix(streaming): normalize inline thinking extraction across live and persisted turns (#3599)
# Conflicts:
# api/streaming.py
# static/messages.js
# static/ui.js
* fix(streaming): code-aware inline-thinking extraction + position-aware unclosed handling
Codex deep-review caught two regressions in the leading-only -> full-scan
rewrite (both silent data-mangling on the persist/reload path):
1. Code-span unawareness: the scanner only protected triple fences, so a
literal <think> in an inline single-backtick code span or an indented
(>=4-space/tab) code block got silently extracted into reasoning. Added
_inline_thinking_indented_code_at + inline-backtick tracking (Python +
the JS twin _thinkingIndentedCodeAt), so all three code contexts now keep
thinking tags visible.
2. Unclosed-tag truncation: any unmatched open tag moved the trailing prose
into reasoning. Now position-aware — a LEADING unclosed block (cut off
mid-thought) is still reasoning (#3455 intent), but an unclosed tag AFTER
visible content stays visible so literal typed tags don't truncate prose.
Gated partial handling on the previously-unused options.streaming param
(live streaming keeps 'still thinking' behavior; persist/reload does not).
Updated 2 tests that pinned the buggy behavior + added 4 regression tests
(inline-backtick, indented-code, mid-body-unclosed-visible, leading-unclosed-
extracted). Updated the node driver harness to include the new helper.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* fix(streaming): recognize fenced code blocks indented 1-3 spaces
Codex round-3: a fence indented 1-3 spaces is valid Markdown but the fence
detector only matched at column 0, so a literal think tag inside such a fence
(not 4+-space indented code either) was still extracted. Both detectors
(_inline_thinking_fence_marker_at / _thinkingFenceMarkerAt) now walk back over
up to 3 leading spaces to a line start. Added backtick + tilde indented-fence
regression tests.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* fix(streaming): O(n) inline-thinking scan + merge separate reasoning on reload
Round-4 Codex deep-review caught two real issues in my own fixes:
1. PERF (O(n^2)): the indented-code check (_inline_thinking_indented_code_at /
_thinkingIndentedCodeAt) scanned to line boundaries at EVERY character index,
plus the leading check sliced+stripped the whole prefix per unclosed tag. On
long no-newline content this was quadratic (~8.4s @ 200k, called repeatedly
on the streaming path). Replaced with incremental O(1)-per-iteration line
state (_line_is_indented_code / _lineIsIndentedCode evaluated only at line
starts) + a seen_nonspace flag. 200k now extracts in ~55-140ms.
2. RELOAD reasoning-drop: renderMessages() seeded the shared extractor with ''
so a message with BOTH an inline <think> block AND a separate m.reasoning
payload showed only the inline part — the separate payload was dropped
because the !thinkingText worklog resolution was then skipped. Now seeds with
the message's direct reasoning (m.reasoning_content||m.reasoning||...) so the
two MERGE (deduped); separate-only reasoning is preserved without promoting
it into visible prose.
Python + JS twins kept line-for-line parity. Added merge + perf + reload
regression tests; updated the reload structure test and the node driver harness
for the renamed helper.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* fix(streaming): revert reload reasoning-seed; keep O(n) perf fix
Codex round-4 finding #2 (seed renderMessages' inline extractor with
m.reasoning so a separate payload merges) turned out to VIOLATE a deliberate
architectural invariant pinned by test_issue2565 +
test_sprint42: the reload content-extraction path must NOT touch
m.reasoning/m.reasoning_content — reasoning metadata is owned exclusively by
the Worklog Thinking Card path (_worklogReasoningTextFromMessage /
_assistantReasoningPayloadText), never conflated with inline-content
extraction (which would risk promoting provider reasoning into final-answer
prose). Reverted the ui.js seed to the PR's original `thinkingText` arg.
The inline+separate merge is still a genuine extractor capability (exercised
by the live streaming path via liveReasoningText) and is covered by a unit
test, just not invoked from the reload render path by design.
The O(n) perf fix (finding #1) and the code-awareness + position-aware
unclosed handling (rounds 1-3) are all retained.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* fix(streaming): only lstrip extracted content when a leading block was removed
Codex round-5 catch: the extractor unconditionally lstripped the final content
(.lstrip() / .replace(/^\s+/,'')) even when NO thinking block was extracted, so
an assistant reply that legitimately starts with an indented code block or blank
lines lost its leading whitespace on live display, reload, and persistence. This
was a real regression vs master (master returned non-thinking content unchanged).
Now track leading_removed (set only when a LEADING thinking block/prefix is
actually extracted) and lstrip only in that case. Mid-body / no-thinking content
keeps its exact leading whitespace. Python + JS twins kept in parity; added
backend regression tests (indented-first preserved, leading-blank preserved,
leading-think still strips).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* fix(streaming): reconnect restore prefers raw inflight accumulator
Codex round-6 CORE catch: on reconnect, the single-live-message restore used
(_liveInflightAssistant.content || ''). Because the PR now splits a leading
unclosed <think> into empty content, restoring from the split content dropped
the open tag — so a later </think> token leaked into the visible reply and
corrupted the live accumulator. Restore from
(_fullInflightAssistant || _liveInflightAssistant.content || '') so the raw
open tag survives reconnect and the accumulator stays correct. Added a
reconnect-restore regression test.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* Release v0.51.335 — Release KY (normalize inline thinking extraction, #3633)
Unify inline-thinking (<think>/<|channel>/<|turn|>) extraction across live,
reload, and persisted turns (#3599/#3633, @rodboev). Deep-reviewed: Opus +
6 Codex rounds; maintainer fixes resolved every Codex finding — code-awareness
(inline-backtick/indented/1-3-space fences keep literal tags visible),
position-aware unclosed handling, O(n) line scanning (was O(n^2) on long
content), conditional lstrip (preserve leading whitespace when no leading block
removed), and a reconnect-restore CORE fix (raw accumulator preferred so an open
<think> tag survives reconnect). Python + JS twins in parity. Full suite 8330,
Opus SHIP-SAFE, Codex SAFE-TO-SHIP, ESLint/scope-undef/ruff clean.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
---------
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Co-authored-by: Hermes Agent <hermes-agent@nesquena-hermes.local>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
@rodboev. providers.<name>.models.<model>.context_length overrides (standard provider,
no base_url) were invisible to the session context resolver → wrong window shown/persisted,
could trip auto-compression at the wrong threshold. New _context_length_lookup_inputs_for_model
helper resolves provider config / base_url / custom_providers across route-load, session-save,
and SSE-usage paths; provider-scoped overrides match by provider name and forward as
config_context_length (returned before any base-url-gated probe).
Maintainer pre-merge items both already satisfied in PR head: no-base_url regression test
(test_route_resolver_uses_provider_model_context_length_without_base_url) present; session-save
_cfg_base_url assigned before the helper call (safe-bound, no NameError on TypeError fallback).
Verified api code byte-identical to PR head; 14 context-length tests pass. + CHANGELOG v0.51.300.
Co-authored-by: nesquena-hermes <[email protected]>
* Harden interrupted recovery control filtering
* Redesign live-to-final assistant replies
* Fix live activity anchor test fixture
* Fix CI lint issues for live reply tests
* Strengthen live progress prompt contract
* Recover PR #3401 refresh on origin/master
* Repair live-to-final refresh regressions
* Fix live worklog refresh regressions
* Show live footer timer on initial stream start
* Restore live stream shell after reload
* Preserve per-frame live SSE replay cursors
* Preserve reasoning as Worklog Thinking cards
* Quiet Worklog Thinking card styling
* Align Worklog Thinking card styling
* Scope live Worklog Thinking cards by segment
* Suppress exact duplicate settled Thinking
* Close#3401 merge review test gaps
* fix(#3401): resolve 4 deep-review regressions (inline-think, reconnect-dup, neon skin, busy-gate worklog)
Deep review (Codex diff-vs-master + live-browser drive) of the live-to-final refactor
surfaced 4 regressions vs master that the rewritten suite no longer guarded:
1. Inline <think>…</think>answer reasoning vanished — _assistantReasoningPayloadText
used $-anchored regexes so a leading think block + visible answer extracted nothing
and the Thinking card never rendered. Removed the 3 $ anchors to match the
(non-anchored) display stripper. Live: inline-think thinking-only turn now renders.
2. (CORE) reconnect/reload duplicated the live reply — _rememberRunJournalCursor advanced
a closure-local seq but never wrote INFLIGHT[activeSid].lastRunJournalSeq, so a reload
replayed the journal from after_seq=0 over restored lastAssistantText. Now mirrors the
cursor onto INFLIGHT + schedules a throttled persist.
3. Neon skin silently broke — PR deleted the :root[data-skin="neon"] CSS but left Neon in
the picker. Restored the neon CSS block from master.
4. Settled tool-worklog rebuild gated purely on !S.busy — dropped every prior settled
turn's worklog when renderMessages re-ran during an active stream (switch-back to an
in-progress session). Restored master's !S.busy || (S.toolCalls && S.toolCalls.length).
Live: busy re-render now preserves tool cards (4→4, was 4→0).
Live-verified all 4 + confirmed #3709/#3592 invariants still hold (1 thinking card, none
below footer; distinct siblings preserved). + tests/test_issue3401_deep_review_fixes.py (7).
* test(#3401): realign 3 stale source-shape assertions to the deep-review fixes
Fix commit changed two source literals that existing stage tests scanned for:
- test_live_activity_timeline.py (x2): split anchor 'if(!S.busy){' → the restored
'if(!S.busy || (S.toolCalls&&S.toolCalls.length)){' guard (fix 4).
- test_run_journal_frontend_static.py: 'after_seq=0' not in source — fix 2's comment
contained that literal; rephrased the comment to 'the zero floor (after_seq of 0)'.
Intent of all three assertions unchanged; only the matched string updated. No code
behavior change.
* docs(changelog): v0.51.294 — Release JJ (stage-3401, #3401 live-to-final redesign)
---------
Co-authored-by: Frank Song <franksong2702@gmail.com>
Co-authored-by: Nathan-Hermes <nesquena-hermes@users.noreply.github.com>
Co-authored-by: nesquena-hermes <[email protected]>
* fix(#3315): surface compression-exhausted/no-final-answer turns as errors (#3316)
When Hermes Agent exhausts context compression in a long tool-heavy turn, the
streamed result can end on a tool result / assistant(tool_calls) turn with no
final assistant answer. WebUI was finalizing that as a completed response.
Now _session_lacks_final_assistant_answer() + _agent_result_terminal_failure()
classify these as terminal failures and surface an apperror instead. The
compression session-id migration + pre-compression snapshot now run BEFORE the
terminal-failure return (ordering bug from the prior hold) so state stays
consistent when exhaustion fires after the agent rotated session_id.
Co-authored-by: Frank Song <franksong2702@gmail.com>
* docs(changelog): v0.51.292 — Release JH (stage-s4, #3316fixes#3315)
---------
Co-authored-by: Frank Song <franksong2702@gmail.com>
Co-authored-by: nesquena-hermes <[email protected]>
* fix(#3405): respect profile provider/model in session resolution (#3448)
Profile-bound sessions now resolve their provider/model from the profile
instead of silently falling back to the global active provider — fixes wrong
credentials/billing and silent context truncation. Repairs stale models under
the profile provider (incl. the openai-codex + openai/ slash-model case) while
preserving native slash IDs on openrouter/custom.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
* docs(changelog): v0.51.290 — Release JF (stage-s1, #3448fixes#3405)
---------
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Co-authored-by: nesquena-hermes <[email protected]>
* feat(sessions): skip adaptive auto-rename for manually-named sessions (#3542, #3230)
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* docs(changelog): v0.51.276 — Release IR (stage-p3e)
* fix(sessions): clear manual_title lock on /api/session/clear (#3542)
Codex regression-gate follow-up: the clear endpoint reset the title to
Untitled directly, stranding manual_title=True so the reused session never
auto-named again. Route the reset through apply_session_title_rename (which
clears the lock for auto-labels) + add a behavioral and a static-guard test.
---------
Co-authored-by: nesquena-hermes <[email protected]>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
## Release v0.51.266 — Release IH (stage-r16)
One agent-authored APPROVED fix + two un-held streaming/SSE fixes.
### Fixed
| Issue/PR | Author | Fix |
|----------|--------|-----|
| #3635 (#3637) | @nesquena-hermes (nesquena APPROVED) | Composer profile chip reads `S.activeProfile` again — a #3331 regression keyed it on the loaded session's profile, so opening a cross-profile session made the chip disagree with the dropdown checkmark and misrepresent where the next message routes. #3331's project/session-op scoping is unaffected. |
| #3587 (#3605) | @rodboev | Reasoning persists to the correct intermediate assistant message in multi-turn tool flows. The index only advanced in `on_interim_assistant` (suppressed for contentless tool-call messages) → post-tool reasoning was mis-attributed; it now also advances at the `on_tool` boundary, guarded against over-increment. **(un-held — finding resolved)** |
| #2660 (#3558) | @franksong2702 | Session-event SSE no longer wakes every tab across profiles and never drops a relevant refresh — profile attached when known, root/`default` aliases stay unscoped, and the `maxsize=1` queue falls back to unscoped refresh-all on a profile-mismatch coalesce. **(un-held — both findings resolved)** |
### Gate
- Full pytest suite: **7770 passed, 0 failed**
- ESLint: CLEAN · ruff: CLEAN · browser-smoke: CLEAN
- Codex (regression): **SAFE TO SHIP** — chip matches dropdown/routing (no #3331 scoping regression), reasoning-index advance composes with the agent's tool/interim callback ordering, session-events coalesce safely with no dropped refresh and no profile data leak (`/api/sessions` still server-side filtered).
Co-authored-by: nesquena <nesquena@users.noreply.github.com>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
## Release v0.51.262 — Release ID (stage-r12)
Phase-3 light slice (no-screenshot items) — 3 PRs.
### Fixed
| Issue/PR | Author | Fix |
|----------|--------|-----|
| #3432 (#3532) | @franksong2702 | Normalize the recall-prefill terminal `user` turn so WebUI doesn't send adjacent `user` roles to strict chat templates (Mistral/Gemma/Jinja). `_normalize_prefill_messages_before_user_turn()` in both `streaming.py` + `gateway_chat.py`; drops only the terminal user tail, preserves assistant/system/mid-list context. **Rebased onto master** (was CONFLICTING). |
| #2558 (#3516) | @rodboev | "Reveal in file manager" now translates container workspace paths (`/workspace`) back to the host mount path for Docker deployments (traversal-safe via `safe_resolve` + sibling-prefix guard). |
### Changed
| Issue/PR | Author | Change |
|----------|--------|--------|
| (#3539) | @Lyr-GW | Completed the Chinese (Simplified) `zh` localization (MCP controls, tool-list pagination) with all interpolations preserved, and the language dropdown now applies the locale **instantly** on change. |
### Review fix absorbed (Codex)
#3539 also added an `allowed=['en','zh']` filter to the Settings language dropdown, which dropped the 9 other shipped locales (it/ja/ru/es/de/pt/ko/fr/tr) — and since save falls back to `en` when the select has no matching option, an existing user of those locales would be **silently reset to English** on a Settings save. Removed the filter (dropdown enumerates all `LOCALES` again, matching master); partially-translated locales fall back per-key to English at render. + regression test `test_issue3539_language_dropdown_all_locales.py`.
### Gate
- Full pytest suite: **7701 passed, 0 failed**
- ESLint: CLEAN · ruff: CLEAN · browser-smoke: CLEAN
- Codex (regression): SHIP-ONLY-WITH-FIXES (dropdown drops-locales) → fixed → **SAFE TO SHIP** (verified prefill drops only terminal user tail in both paths, Docker path-translation traversal-safe, zh interpolations preserved)
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Co-authored-by: Lyr-GW <Lyr-GW@users.noreply.github.com>
## Release v0.51.261 — Release IC (stage-r11)
Live Todos panel via an explicit `todo_state` SSE contract.
### Fixed
| Issue/PR | Author | Fix |
|----------|--------|-----|
| #3373 follow-up (#3454) | @v2psv | The Todos side panel now tracks `todo` tool state **live during an active run** instead of staying stale until settle / rolling back on a mid-stream reload. A dedicated `todo_state` SSE event sends a full, redacted, idempotent snapshot on todo-tool completion (no more truncated `tool_complete.preview`); the same `api.todo_state` parser feeds live + cold-load; live snapshots persist into INFLIGHT so reload/reattach restores the panel; cold-load vs INFLIGHT reconciled by timestamp (incl. the `coldTs===0` compressed-session edge); legacy reverse-scan kept as fallback for old servers. |
### Gate
- Full pytest suite: **7692 passed, 0 failed**
- ESLint: CLEAN · ruff: CLEAN · browser-smoke: CLEAN
- Codex (regression): **SAFE TO SHIP** — verified the new `todo_state` SSE handler composes with existing dispatch (no double-subscribe), INFLIGHT persistence is cleared on terminal/cancel (composes with discard_session + turn-journal), timestamp reconciliation can't let a stale local snapshot win, redaction holds, the legacy reverse-scan fallback still works with no double-render, and the `models.py` change is todo-scoped (no CLI-classification interaction).
Co-authored-by: v2psv <v2psv@users.noreply.github.com>
## Release v0.51.259 — Release IA (stage-r7)
Two ship-ready @rodboev bug-fixes from today (the clean subset of the prioritized 6).
### Fixed
| Issue | Fix |
|-------|-----|
| #3582 | **Edge-TTS playback no longer has a ~31s delay / playback error** — `_handle_tts` streamed audio without `Content-Length` on an HTTP/1.0 server; audio is now buffered and sent with an exact `Content-Length`. |
| #3583 | **CLI-bridge message reconstruction strips orphaned `tool_calls`** (assistant `tool_calls` with no matching `tool` response, left by an aborted bridge) so the next request no longer 400s on strict providers. |
### Held back from the 6-PR batch (Codex regression gate caught a real defect in each)
- **#3586/#3603** (`is_cli_session_row` reclassification) — CORE: messaging rows become non-CLI, but the sidebar open path only imports when `is_cli_session`, so opening a Discord/Telegram session shows a transient stub and the next send 404s on `/api/chat/start`. Needs a client import-gate fix + live verify. **Held.**
- **#3585/#3604** (cron-overflow) — removing `exclude_sources=None` also re-excludes `source='webui'` rows, dropping sidecarless WebUI session recovery from `/api/sessions`. Needs a separate webui recovery pass. **Held.**
- **#3587** (intermediate reasoning) — `on_interim_assistant` is suppressed upstream for contentless tool-call assistant messages (`run_agent.py:3834`), so advancing the reasoning index there never fires at tool-call boundaries → mis-attribution. **Held.**
- **#3538** (self-update stash-pop) — BRICK data-loss (`git reset --merge` + `git stash drop` discards user mods), still unaddressed. **Held.**
### Gate
- Full pytest suite: **7645 passed, 0 failed**
- ruff: CLEAN
- Codex (regression): 3 rounds — 4 PRs dropped/held for real regressions → **SAFE TO SHIP** on the clean 2
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
## Release v0.51.257 — Release HY (stage-r5)
Two rebased ★★★ fixes.
### Fixed
| Issue | Author | Fix |
|-------|--------|-----|
| #3546 | @rodboev | **"Refresh Models" on a provider card no longer returns "Error: Not found".** Sent `POST /api/models/refresh` but no route matched (404). Added the route, wired to the existing `invalidate_provider_models_cache(provider_id)`. |
| #3548 | @franksong2702 | **Credential self-heal no longer writes to a dead `SessionDB` handle.** A credential-refresh evicted/closed the cached agent, but the retry rebuilt a new agent from kwargs still holding the old closed `SessionDB` → persistence silently targeted a dead handle. Per-request `SessionDB` construction centralized + refreshed on the retry. |
### Dropped from this batch
- **#2660** (session-event SSE profile scoping) — the Codex regression gate found **two SILENT dropped-refresh bugs** the scoping introduced: (1) the `maxsize=1` subscriber-queue coalescing overwrites a pending profile-A event with a profile-B event → A-tabs filter B out and never refresh for the A change; (2) renamed-root/`default` alias mismatch (backend `_profiles_match` treats them equal, the client filter uses a literal `!==`). A dropped refresh (stale sidebar) is worse than the extra refreshes the PR removes. Held with `changes-requested` + repros + the fail-safe fix (coalesce to unscoped on a profile mismatch; normalize root aliases).
### Gate
- Full pytest suite: **7622 passed, 0 failed**
- ESLint: CLEAN · ruff: CLEAN · browser-smoke: CLEAN
- Codex (regression): SHIP-ONLY-WITH-FIXES (#2660 dropped-refresh bugs) → #2660 dropped + held → **SAFE TO SHIP**
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
## Release v0.51.256 — Release HX (stage-r4)
Performance — bound WebUI memory growth & idle CPU on large installs.
### Fixed
| Issue | Author | Fix |
|-------|--------|-----|
| #3506 | @nesquena-hermes (reported w/ profiling by @djenttleman) | On a large install (~615 sessions / 40k messages / 454 MB state.db) the WebUI process climbed ~100 MB → ~1.5 GB RSS over days and held high idle CPU. Three root causes fixed: (1) `session_lifecycle._sessions` grew unbounded → new `discard_session()` drops the entry at agent-eviction boundaries, only when no in-flight commit / no uncommitted memory work (retry invariant preserved); (2) cache caps now operator-tunable (`HERMES_WEBUI_AGENT_CACHE_MAX` default 50→25, `HERMES_WEBUI_SESSIONS_MAX`); (3) GatewayWatcher computes a cheap fingerprint before the expensive per-session `MAX(messages.timestamp)` projection and only re-projects on change. |
### Rebase + review notes
- Rebased onto current master; the code diff was verified **byte-identical to the nesquena-APPROVED head** at rebase time (only CHANGELOG re-resolved).
- The Codex regression gate then surfaced **two correctness gaps** the approval didn't catch, both fixed here with regression tests:
1. **Watcher fingerprint missed same-count transcript rewrites.** `/retry`,`/undo`,`/compress` (`SessionDB.replace_messages`) rewrite messages with new timestamps but can leave `message_count` unchanged → stale sidebar `last_activity`. Fixed with a **per-session** grouped message aggregate (`id, count, user_count, MAX(timestamp)`) over the same non-excluded sessions (a global MAX would miss a rewrite of an older, non-newest session); cron/webui stay excluded so idle churn still doesn't re-project.
2. **LRU agent-cache eviction could close a live worker's agent** (`popitem(last=False)`, liveness-blind — pre-existing, but the lower 50→25 cap made it more likely). Eviction now snapshots `ACTIVE_RUNS` session_ids (before the cache lock — no nested lock) and skips live sessions, deferring (temporarily exceeding cap) rather than closing a live agent.
### Gate
- Full pytest suite: **7612 passed, 0 failed** (one boot-cascade flake re-run; clean on re-run)
- ruff: CLEAN · Codex (regression): 4 rounds → both gaps + a stale test fixed → **SAFE TO SHIP**
Co-authored-by: nesquena <nesquena@users.noreply.github.com>
## Release v0.51.247 — Release HO (stage-q19)
Backend correctness fix.
### Fixed
| Issue | Author | Fix |
|-------|--------|-----|
| #3505 | @franksong2702 | **Reasoning effort is coerced to a level the active model/provider actually supports** before each request, instead of being sent verbatim and rejected. `openai-codex` `gpt-5` no longer gets `max` (→ `xhigh`); `o1`/`o3`/`o4` clamp to `low`/`medium`/`high`. Coercion only steps *down* (never escalates); `none`/unset preserved. The capability filter is applied across heuristic / models.dev / Copilot / LM Studio paths. |
This is the narrow, correct fix for the detection gap that #3431 tried to address by removing the chip-visibility gate (which we shelved). The chip-visibility gate is **untouched** (Codex confirmed) — `get_reasoning_status`/`_applyReasoningChip` still hide the chip for unconfirmed models.
### Review fix absorbed (Codex + self-flagged)
The first cut **dropped** a configured effort for *unrecognized* models, because capability detection returns `[]` for both "known-unsupported" and "simply-unknown" (custom providers, aggregator-rewritten ids, new releases) — that's a behavior change vs master (which sent it verbatim) and would silently disable reasoning. Fixed: an **empty** capability set now **preserves** the configured effort (provider stays the final authority; worst case = the same rejected request master already produces, i.e. no regression). Known-bad clamps return *non-empty* filtered sets, so they still degrade correctly. Nathan chose this "preserve-for-unknown" behavior. + regression test.
### Gate
- Full pytest suite: **7548 passed, 0 failed**
- ruff: CLEAN · 48 reasoning tests pass (incl. preserve-for-unknown + codex-clamp + never-escalate)
- Codex (regression): SHIP-ONLY-WITH-FIXES (unknown-model drop) → fixed → **SAFE TO SHIP**
- Verified empirically: gpt-5/codex max→xhigh, o3 max/xhigh→high, unknown high→high (preserved), none/unset preserved
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
## Release v0.51.237 — Release HE (stage-q8)
Phase 3 MEDIUM-ring pick (3-factor framework): a real concurrency/state-consistency fix from a regular contributor (@franksong2702, ★★★ 38 merges). The hardest review of the sweep — the dual gate caught **two** silent data-loss bugs across two review rounds.
### Fixed
| PR | Author | Fix |
|----|--------|-----|
| #3475 | @franksong2702 | Cancelling a live turn immediately after send now reliably stops the worker and settles the session to a cancelled state (was: spinner over a blank page). `cancel_stream()` falls back to the live active-run registry (`ACTIVE_RUNS`) + session agent cache when `STREAMS` has already detached, so the worker still receives `interrupt("Cancelled by user")`. `/api/session` reports run-journal active state from the live registry instead of trusting a persisted `active_stream_id`. |
### Pre-release dual gate caught TWO silent data-loss bugs (both fixed + regression-tested)
The PR's refactor moved `agent.interrupt()` ahead of the partial-text/reasoning/tool-call snapshot, and that snapshot was no longer under `streams_lock`:
1. **Codex round 1** — the worker's `finally` (which pops `STREAM_PARTIAL_TEXT`/`STREAM_REASONING_TEXT`/`STREAM_LIVE_TOOL_CALLS` under `STREAMS_LOCK`) could clear those buffers the instant `interrupt()` wakes it, so a cancelled turn **silently lost its already-streamed text**. (Opus reviewed the original and said ship — it assumed the snapshot was still lock-protected; the stale comment claimed so but the code wasn't. Verified against the actual code → Codex was right.)
2. **Codex round 2** — my first fix only snapshotted on the `STREAMS`-present path; the detached `ACTIVE_RUNS`-only path (the case this PR adds) still lost text. Fixed by hoisting the snapshot above the `if stream_present` branch so it runs unconditionally under the lock.
Both fixes have regression tests **verified to fail against the buggy versions** (`test_cancel_preserves_partial_text_when_interrupt_pops_buffers` + `test_cancel_preserves_partial_text_on_detached_active_run_path`).
### Gate results
- **Full pytest suite**: 7481 passed, 9 skipped, 3 xpassed, **0 failed**
- **ruff forward gate**: CLEAN
- **browser-smoke gate**: CLEAN
- **Codex (regression)**: SHIP ONLY WITH FIXES (×2 rounds) → both MUST-FIXes applied → re-reviewed **SAFE TO SHIP**
- **Opus (correctness)**: reviewed the original (SAFE); the shipped version is strictly safer (adds the under-lock snapshot Opus deemed unnecessary)
- Deadlock concern cleared: no path takes `ACTIVE_RUNS_LOCK` then `STREAMS_LOCK`.
Rebased onto current master (streaming.py/routes.py merged clean — no overlap with the #3468 dedup change shipped in HC).
Closes#3475.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
## Release v0.51.235 — Release HC (stage-q5)
Single fix in the data-sensitive transcript-merge path, with a Codex-found MUST-FIX applied.
### Fix
| PR | Author | Fix |
|----|--------|-----|
| #3468 | @jasonjcwu | `_find_current_user_turn` returns the **last** matching user turn instead of the first, so post-compression `result_messages` (which carries the full history) no longer replays the entire conversation when the user repeats a similar question (137-msg session → 89 duplicate replays). |
### Pre-release review caught a CORE issue (fixed before ship)
Codex's regression gate found that a naive last-match could be overridden by a **synthetic `role:"user"` continuation prompt** (the agent loop injects "Continue"/empty-recovery nudges — verified at `conversation_loop.py:1763/4183/4356`) that only *substring*-matches the user text — anchoring the merge **past** the real turn and dropping the assistant/tool output in between. Applied the fix: track **strong** (exact `_looks_like_current_user_turn`) and **weak** (substring) matches separately and return `last_strong → last_weak → fallback`. The real turn (strong) always wins over a later synthetic continuation (weak).
Opus reviewed the original and said ship; the applied fix is strictly safer than what it reviewed.
### Gate results
- **Full pytest suite**: 7465 passed, 8 skipped, 3 xpassed, **0 failed**
- **ruff forward gate**: CLEAN
- **browser-smoke gate**: CLEAN
- **Codex (regression)**: SHIP ONLY WITH FIXES → fix applied → re-reviewed **SAFE TO SHIP**
- **Opus (correctness)**: SHIP IT (on the pre-fix code; applied fix is strictly safer)
- **Regression test** (`tests/test_issue3468_duplicate_after_compression.py`, 7 cases): pins the last-match behavior, the strong-beats-later-weak invariant, and the end-to-end no-duplicate-replay invariant — each **verified to fail against the pre-fix logic**.
Co-authored-by: jasonjcwu <jasonjcwu@users.noreply.github.com>
* Release v0.51.230 (stage-p14): extract <think> to m.reasoning #3455 + LLM Wiki last-writer (#1257)
Salvage of #3455 (@gsurenull): dropped the stale api/config.py bits (MiniMax-M3 +
SCHEMA_VERSION 3->4 — both already on master via #3374). Kept the two genuine fixes:
(1) _splitThinkFromContent persist-path extraction of inline <think> blocks into
m.reasoning (fixes 30-50% session bloat for reasoning-only providers like MiniMax-M3);
(2) LLM Wiki status Last-writer 3-tier fallback (was always 'Not available' since #1257).
Added 9 Node-driven think-split regression tests (data-loss guards: content-before/after
preserved, unclosed blocks intact, lookalike tags not extracted).
* fix(#3455): renderer-matching think extraction + wiki symlink/bounded-read guards (Codex review)
Codex review of stage-p14 found 3 SILENT bugs, all fixed:
(1) DATA-LOSS: _splitThinkFromContent's Pass-2 whole-body scan extracted a CLOSED literal
<think>...</think> from visible prose/code (e.g. inside a fenced code block) into m.reasoning,
emptying it — more aggressive than the renderer (which only strips LEADING blocks). Removed
Pass 2; extraction now matches _streamDisplay semantics (leading-only, loop captures
consecutive leading blocks). +fenced-code regression test.
(2) PRIVACY: _llm_wiki_last_writer followed symlinked .md pages resolving OUTSIDE the wiki
(is_file follows symlinks), leaking external frontmatter. Now requires resolved path under
wiki_root. +symlink-containment regression test.
(3) CONTRACT/PERF: replaced full read_text() with bounded line-by-line reads (frontmatter
block only / capped log-heading scan), never page bodies.
* fix(#3455): think-split is leading-single (renderer-matching) + fix 2 stale source-match tests
Codex re-review finding #2: looping consecutive leading blocks diverged from the renderer
(_streamDisplay/_parseStreamState strip ONE leading block). Now extracts exactly one leading
block. Also updated 2 tests that asserted pre-split implementation strings:
test_live_stream_tokens_persist (content:assistantText -> content:split.content, invariant
preserved) and the consecutive-blocks test. NOTE: Codex finding #1 (client-only split doesn't
persist server-side) is a separate architectural decision pending Nathan.
* feat(#3455): split inline <think> server-side before s.save() so persisted file is compacted (Codex #1)
Codex finding #1: the think-split was client-only, so the SAVED session file still
carried inline <think> blocks (bloat) — the fix only compacted the browser copy.
Added _split_thinking_from_content (api/streaming.py), a server-side twin of the JS
helper with identical leading-only/single-block semantics, applied to the final
assistant message before s.save() (extended the existing reasoning-persist block).
Merges with on_reasoning-stream reasoning. +8 backend-parity regression tests covering
the mid-body-code-block data-loss guard, unclosed-intact, single-leading, none-content.
* test: update 3 save-path source-assertion tests for #3455 server-side think-split
The backend think-split (api/streaming.py reasoning-persist block) changed the literal
code shape + grew the pre-save block, breaking 8 source-assertion tests that anchor on it:
- test_sprint42: assert _rm['reasoning']=_reasoning_text -> now _merged_reasoning/_existing_reasoning
+ _split_thinking_from_content present (intent preserved: reasoning persisted before save).
- test_pr1318 (6) + test_pr1341: re-anchored the locator from the changed 'if _reasoning_text
and s.messages:' line to the stable 'Persist reasoning trace in the session' comment marker;
bumped the 1341 byte-distance limit 15000->16000 (the test self-documents bumping on legit
pre-save growth). All behavioral invariants (reasoning persisted + context fields before save)
unchanged.
---------
Co-authored-by: nesquena-hermes <[email protected]>
* fix: respect profile toolset/skill config on WebUI streaming worker (#3294)
The streaming agent runs on a detached worker thread that does not inherit
the per-request thread-local profile context (set from the hermes_profile
cookie on the HTTP handler thread). On that worker, the ambient get_config()
resolves via get_active_profile_name() which falls back to the process-global
_active_profile (usually 'default'). A session under a non-default profile
with an empty platform_toolsets.cli therefore loaded the DEFAULT profile's
full toolset list, inflating a tools-disabled profile's prompt from ~400 to
~15K input tokens.
Add api.config.get_config_for_profile_home() — a race-free direct disk read
of an explicit profile home's config.yaml (no shared-cache mutation), which
defers to get_config() when the requested home matches the ambient path so
in-memory test overrides are preserved. The streaming worker now resolves
_cfg from the session's own profile home, fixing toolsets, prefill context,
and fallback chains in one place.
Closes#3294
Co-authored-by: gottipx <gottipx@users.noreply.github.com>
* docs(changelog): v0.51.224 — #3294 profile toolset config (dropped #3405 to hold per Codex stale-model-repair finding)
---------
Co-authored-by: nesquena-hermes <[email protected]>
Co-authored-by: gottipx <gottipx@users.noreply.github.com>
* fix: reject cross-script drifted auto-generated session titles (#3293)
The title-language mismatch guard only knew two states: German (de) or empty,
and _title_language_mismatch early-returned False whenever the user start
wasn't German. So an English conversation whose LLM-generated title came back
in Chinese / Spanish / Russian sailed through and persisted with
llm_title_generated=true. The German case was the only one covered because
that's the one prior report it was built for.
Generalize from a German-specific binary to a language-agnostic cross-script
check. Add _script_counts() + _dominant_script() (cheap, dependency-free
Unicode-block classification: latin / cjk / cyrillic / arabic / hebrew / greek
/ devanagari). _title_language_mismatch now rejects a title that introduces a
substantial amount (>=35% of alphabetic chars, min 2) of a script different
from the conversation start's dominant script — so short titles that embed a
borrowed Latin technical term still trip, while an English title with a single
foreign place-name does not. The legacy German->English same-script heuristic
is preserved verbatim.
Kept api/streaming.py ASCII-only (the test_title_generation_source_has_no_cjk_
literals guard) — all CJK examples live in the test file, not the source.
Closes#3293
Co-authored-by: andrewkangkr <andrewkangkr@users.noreply.github.com>
* fix: prune orphaned imported-CLI sidecars from the WebUI sidebar (#3238)
When a CLI/agent session is opened in WebUI it gets a WebUI-owned sidecar
(webui/sessions/<id>.json + _index.json row) so it can render and reopen;
all_sessions() then returns it independently of the agent state.db. If the
user later deletes that session from the CLI / local Hermes storage, nothing
pruned the sidecar — the merge loop only overlays CLI metadata when a matching
state.db row exists and otherwise continues, so the stale row lingered in the
sidebar indefinitely (there is no WebUI delete affordance for CLI rows).
Add api.models.agent_session_row_exists(): an exact, uncapped existence probe
against the state.db sessions table. The sidebar merge loop now drops a row
that is_cli_session_row + not WebUI-native + absent from cli_by_id + whose
state.db row is genuinely gone, and calls prune_session_from_index() so
_index.json self-heals.
The state.db probe is deliberate: get_cli_sessions() caps at
CLI_VISIBLE_SESSION_LIMIT (20), so a still-existing session can fall out of
that window and look deleted — pruning on cli_by_id absence alone would delete
live sessions. WebUI-native rows with a CLI ancestor are never pruned, and any
probe error degrades to keep-the-row so a transient failure can't lose data.
Closes#3238
Co-authored-by: Luxciax <Luxciax@users.noreply.github.com>
* fix: count pin quota by visible session lineage
* docs(changelog): v0.51.222 — backend bugfix batch (#3293 title drift, #3238 sidecar prune, #3288 pin lineage)
* fix(pins): forks count as own pin lineage, not collapsed to parent (#3288 Codex follow-up)
Codex review of the batch found a pin-limit UNDERCOUNT: _session_row_lineage_root_id
followed any parent_session_id to the root, but /api/session/branch creates
independent visible fork sessions that also carry parent_session_id (session_source=
'fork'). Two pinned forks of the same parent collapsed to one quota lineage, letting
a user exceed pinned_sessions_limit with no 400. Fix: a fork returns its own id as
its lineage root (it's a separately-visible session); only compression/continuation
rows still collapse to a shared root. Adds a regression test with two pinned forks +
the parent counting as three distinct lineages, and confirms the existing
pre-compression-snapshot collapse case still passes.
* test(pins): update #2508/#2821 source-match tests for #3288 lineage rename
#3288 replaced the raw-session-id pin counter (pinned_ids set) with a
visible-lineage counter (pinned_lineage_ids via _visible_pinned_lineage_ids over
persisted_rows/candidate_rows). Two pre-existing source-string-matching tests
asserted the OLD implementation literals (pinned_ids = {, _session_field(existing,
session_id...), len(pinned_ids) >=). Updated both to assert the new mechanism while
preserving the invariants they actually guard: snapshot computed BEFORE LOCK (no
all_sessions()-inside-LOCK deadlock), quota filtering via the shared
_session_counts_toward_pin_quota helper, and the limit/400 guard. Behaviour
unchanged; these were implementation-detail assertions, not behaviour tests.
---------
Co-authored-by: nesquena-hermes <[email protected]>
Co-authored-by: andrewkangkr <andrewkangkr@users.noreply.github.com>
Co-authored-by: Luxciax <Luxciax@users.noreply.github.com>
Co-authored-by: Andy Kang <andrewkang.kr@gmail.com>
Manual title regeneration (POST /api/session/title/regenerate) and background
aux title generation failed with 422 / llm_error_aux when
auxiliary.title_generation.model in config.yaml used the WebUI-internal
@provider:model picker format (e.g. @gemini:gemini-3.1-flash-lite) — the raw
@-qualified id was forwarded to the provider API verbatim. Normalize it through
the canonical _split_webui_provider_model_value() helper before the aux call.
Co-authored-by: pamnard <pamnard@users.noreply.github.com>
Consecutive user turns (session-context prefill + actual message) made models
with strict chat templates (Mistral, Gemma via llama.cpp) reject the request
with a Jinja 500. Move the platform/delivery context (connected platforms, home
channels, scheduled-task delivery hints) from _webui_session_context_message
(a prefill user message) into _webui_delivery_context_prompt, appended to the
ephemeral system prompt. Session framing stays in _webui_surface_context_prompt.
Context is preserved — just role-alternation-safe. Closes#3276.
Co-authored-by: aether-agent <aether-agent@users.noreply.github.com>
