## Release v0.51.256 — Release HX (stage-r4)
Performance — bound WebUI memory growth & idle CPU on large installs.
### Fixed
| Issue | Author | Fix |
|-------|--------|-----|
| #3506 | @nesquena-hermes (reported w/ profiling by @djenttleman) | On a large install (~615 sessions / 40k messages / 454 MB state.db) the WebUI process climbed ~100 MB → ~1.5 GB RSS over days and held high idle CPU. Three root causes fixed: (1) `session_lifecycle._sessions` grew unbounded → new `discard_session()` drops the entry at agent-eviction boundaries, only when no in-flight commit / no uncommitted memory work (retry invariant preserved); (2) cache caps now operator-tunable (`HERMES_WEBUI_AGENT_CACHE_MAX` default 50→25, `HERMES_WEBUI_SESSIONS_MAX`); (3) GatewayWatcher computes a cheap fingerprint before the expensive per-session `MAX(messages.timestamp)` projection and only re-projects on change. |
### Rebase + review notes
- Rebased onto current master; the code diff was verified **byte-identical to the nesquena-APPROVED head** at rebase time (only CHANGELOG re-resolved).
- The Codex regression gate then surfaced **two correctness gaps** the approval didn't catch, both fixed here with regression tests:
1. **Watcher fingerprint missed same-count transcript rewrites.** `/retry`,`/undo`,`/compress` (`SessionDB.replace_messages`) rewrite messages with new timestamps but can leave `message_count` unchanged → stale sidebar `last_activity`. Fixed with a **per-session** grouped message aggregate (`id, count, user_count, MAX(timestamp)`) over the same non-excluded sessions (a global MAX would miss a rewrite of an older, non-newest session); cron/webui stay excluded so idle churn still doesn't re-project.
2. **LRU agent-cache eviction could close a live worker's agent** (`popitem(last=False)`, liveness-blind — pre-existing, but the lower 50→25 cap made it more likely). Eviction now snapshots `ACTIVE_RUNS` session_ids (before the cache lock — no nested lock) and skips live sessions, deferring (temporarily exceeding cap) rather than closing a live agent.
### Gate
- Full pytest suite: **7612 passed, 0 failed** (one boot-cascade flake re-run; clean on re-run)
- ruff: CLEAN · Codex (regression): 4 rounds → both gaps + a stale test fixed → **SAFE TO SHIP**
Co-authored-by: nesquena <nesquena@users.noreply.github.com>
## Release v0.51.247 — Release HO (stage-q19)
Backend correctness fix.
### Fixed
| Issue | Author | Fix |
|-------|--------|-----|
| #3505 | @franksong2702 | **Reasoning effort is coerced to a level the active model/provider actually supports** before each request, instead of being sent verbatim and rejected. `openai-codex` `gpt-5` no longer gets `max` (→ `xhigh`); `o1`/`o3`/`o4` clamp to `low`/`medium`/`high`. Coercion only steps *down* (never escalates); `none`/unset preserved. The capability filter is applied across heuristic / models.dev / Copilot / LM Studio paths. |
This is the narrow, correct fix for the detection gap that #3431 tried to address by removing the chip-visibility gate (which we shelved). The chip-visibility gate is **untouched** (Codex confirmed) — `get_reasoning_status`/`_applyReasoningChip` still hide the chip for unconfirmed models.
### Review fix absorbed (Codex + self-flagged)
The first cut **dropped** a configured effort for *unrecognized* models, because capability detection returns `[]` for both "known-unsupported" and "simply-unknown" (custom providers, aggregator-rewritten ids, new releases) — that's a behavior change vs master (which sent it verbatim) and would silently disable reasoning. Fixed: an **empty** capability set now **preserves** the configured effort (provider stays the final authority; worst case = the same rejected request master already produces, i.e. no regression). Known-bad clamps return *non-empty* filtered sets, so they still degrade correctly. Nathan chose this "preserve-for-unknown" behavior. + regression test.
### Gate
- Full pytest suite: **7548 passed, 0 failed**
- ruff: CLEAN · 48 reasoning tests pass (incl. preserve-for-unknown + codex-clamp + never-escalate)
- Codex (regression): SHIP-ONLY-WITH-FIXES (unknown-model drop) → fixed → **SAFE TO SHIP**
- Verified empirically: gpt-5/codex max→xhigh, o3 max/xhigh→high, unknown high→high (preserved), none/unset preserved
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
## Release v0.51.237 — Release HE (stage-q8)
Phase 3 MEDIUM-ring pick (3-factor framework): a real concurrency/state-consistency fix from a regular contributor (@franksong2702, ★★★ 38 merges). The hardest review of the sweep — the dual gate caught **two** silent data-loss bugs across two review rounds.
### Fixed
| PR | Author | Fix |
|----|--------|-----|
| #3475 | @franksong2702 | Cancelling a live turn immediately after send now reliably stops the worker and settles the session to a cancelled state (was: spinner over a blank page). `cancel_stream()` falls back to the live active-run registry (`ACTIVE_RUNS`) + session agent cache when `STREAMS` has already detached, so the worker still receives `interrupt("Cancelled by user")`. `/api/session` reports run-journal active state from the live registry instead of trusting a persisted `active_stream_id`. |
### Pre-release dual gate caught TWO silent data-loss bugs (both fixed + regression-tested)
The PR's refactor moved `agent.interrupt()` ahead of the partial-text/reasoning/tool-call snapshot, and that snapshot was no longer under `streams_lock`:
1. **Codex round 1** — the worker's `finally` (which pops `STREAM_PARTIAL_TEXT`/`STREAM_REASONING_TEXT`/`STREAM_LIVE_TOOL_CALLS` under `STREAMS_LOCK`) could clear those buffers the instant `interrupt()` wakes it, so a cancelled turn **silently lost its already-streamed text**. (Opus reviewed the original and said ship — it assumed the snapshot was still lock-protected; the stale comment claimed so but the code wasn't. Verified against the actual code → Codex was right.)
2. **Codex round 2** — my first fix only snapshotted on the `STREAMS`-present path; the detached `ACTIVE_RUNS`-only path (the case this PR adds) still lost text. Fixed by hoisting the snapshot above the `if stream_present` branch so it runs unconditionally under the lock.
Both fixes have regression tests **verified to fail against the buggy versions** (`test_cancel_preserves_partial_text_when_interrupt_pops_buffers` + `test_cancel_preserves_partial_text_on_detached_active_run_path`).
### Gate results
- **Full pytest suite**: 7481 passed, 9 skipped, 3 xpassed, **0 failed**
- **ruff forward gate**: CLEAN
- **browser-smoke gate**: CLEAN
- **Codex (regression)**: SHIP ONLY WITH FIXES (×2 rounds) → both MUST-FIXes applied → re-reviewed **SAFE TO SHIP**
- **Opus (correctness)**: reviewed the original (SAFE); the shipped version is strictly safer (adds the under-lock snapshot Opus deemed unnecessary)
- Deadlock concern cleared: no path takes `ACTIVE_RUNS_LOCK` then `STREAMS_LOCK`.
Rebased onto current master (streaming.py/routes.py merged clean — no overlap with the #3468 dedup change shipped in HC).
Closes#3475.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
## Release v0.51.235 — Release HC (stage-q5)
Single fix in the data-sensitive transcript-merge path, with a Codex-found MUST-FIX applied.
### Fix
| PR | Author | Fix |
|----|--------|-----|
| #3468 | @jasonjcwu | `_find_current_user_turn` returns the **last** matching user turn instead of the first, so post-compression `result_messages` (which carries the full history) no longer replays the entire conversation when the user repeats a similar question (137-msg session → 89 duplicate replays). |
### Pre-release review caught a CORE issue (fixed before ship)
Codex's regression gate found that a naive last-match could be overridden by a **synthetic `role:"user"` continuation prompt** (the agent loop injects "Continue"/empty-recovery nudges — verified at `conversation_loop.py:1763/4183/4356`) that only *substring*-matches the user text — anchoring the merge **past** the real turn and dropping the assistant/tool output in between. Applied the fix: track **strong** (exact `_looks_like_current_user_turn`) and **weak** (substring) matches separately and return `last_strong → last_weak → fallback`. The real turn (strong) always wins over a later synthetic continuation (weak).
Opus reviewed the original and said ship; the applied fix is strictly safer than what it reviewed.
### Gate results
- **Full pytest suite**: 7465 passed, 8 skipped, 3 xpassed, **0 failed**
- **ruff forward gate**: CLEAN
- **browser-smoke gate**: CLEAN
- **Codex (regression)**: SHIP ONLY WITH FIXES → fix applied → re-reviewed **SAFE TO SHIP**
- **Opus (correctness)**: SHIP IT (on the pre-fix code; applied fix is strictly safer)
- **Regression test** (`tests/test_issue3468_duplicate_after_compression.py`, 7 cases): pins the last-match behavior, the strong-beats-later-weak invariant, and the end-to-end no-duplicate-replay invariant — each **verified to fail against the pre-fix logic**.
Co-authored-by: jasonjcwu <jasonjcwu@users.noreply.github.com>
* Release v0.51.230 (stage-p14): extract <think> to m.reasoning #3455 + LLM Wiki last-writer (#1257)
Salvage of #3455 (@gsurenull): dropped the stale api/config.py bits (MiniMax-M3 +
SCHEMA_VERSION 3->4 — both already on master via #3374). Kept the two genuine fixes:
(1) _splitThinkFromContent persist-path extraction of inline <think> blocks into
m.reasoning (fixes 30-50% session bloat for reasoning-only providers like MiniMax-M3);
(2) LLM Wiki status Last-writer 3-tier fallback (was always 'Not available' since #1257).
Added 9 Node-driven think-split regression tests (data-loss guards: content-before/after
preserved, unclosed blocks intact, lookalike tags not extracted).
* fix(#3455): renderer-matching think extraction + wiki symlink/bounded-read guards (Codex review)
Codex review of stage-p14 found 3 SILENT bugs, all fixed:
(1) DATA-LOSS: _splitThinkFromContent's Pass-2 whole-body scan extracted a CLOSED literal
<think>...</think> from visible prose/code (e.g. inside a fenced code block) into m.reasoning,
emptying it — more aggressive than the renderer (which only strips LEADING blocks). Removed
Pass 2; extraction now matches _streamDisplay semantics (leading-only, loop captures
consecutive leading blocks). +fenced-code regression test.
(2) PRIVACY: _llm_wiki_last_writer followed symlinked .md pages resolving OUTSIDE the wiki
(is_file follows symlinks), leaking external frontmatter. Now requires resolved path under
wiki_root. +symlink-containment regression test.
(3) CONTRACT/PERF: replaced full read_text() with bounded line-by-line reads (frontmatter
block only / capped log-heading scan), never page bodies.
* fix(#3455): think-split is leading-single (renderer-matching) + fix 2 stale source-match tests
Codex re-review finding #2: looping consecutive leading blocks diverged from the renderer
(_streamDisplay/_parseStreamState strip ONE leading block). Now extracts exactly one leading
block. Also updated 2 tests that asserted pre-split implementation strings:
test_live_stream_tokens_persist (content:assistantText -> content:split.content, invariant
preserved) and the consecutive-blocks test. NOTE: Codex finding #1 (client-only split doesn't
persist server-side) is a separate architectural decision pending Nathan.
* feat(#3455): split inline <think> server-side before s.save() so persisted file is compacted (Codex #1)
Codex finding #1: the think-split was client-only, so the SAVED session file still
carried inline <think> blocks (bloat) — the fix only compacted the browser copy.
Added _split_thinking_from_content (api/streaming.py), a server-side twin of the JS
helper with identical leading-only/single-block semantics, applied to the final
assistant message before s.save() (extended the existing reasoning-persist block).
Merges with on_reasoning-stream reasoning. +8 backend-parity regression tests covering
the mid-body-code-block data-loss guard, unclosed-intact, single-leading, none-content.
* test: update 3 save-path source-assertion tests for #3455 server-side think-split
The backend think-split (api/streaming.py reasoning-persist block) changed the literal
code shape + grew the pre-save block, breaking 8 source-assertion tests that anchor on it:
- test_sprint42: assert _rm['reasoning']=_reasoning_text -> now _merged_reasoning/_existing_reasoning
+ _split_thinking_from_content present (intent preserved: reasoning persisted before save).
- test_pr1318 (6) + test_pr1341: re-anchored the locator from the changed 'if _reasoning_text
and s.messages:' line to the stable 'Persist reasoning trace in the session' comment marker;
bumped the 1341 byte-distance limit 15000->16000 (the test self-documents bumping on legit
pre-save growth). All behavioral invariants (reasoning persisted + context fields before save)
unchanged.
---------
Co-authored-by: nesquena-hermes <[email protected]>
* fix: respect profile toolset/skill config on WebUI streaming worker (#3294)
The streaming agent runs on a detached worker thread that does not inherit
the per-request thread-local profile context (set from the hermes_profile
cookie on the HTTP handler thread). On that worker, the ambient get_config()
resolves via get_active_profile_name() which falls back to the process-global
_active_profile (usually 'default'). A session under a non-default profile
with an empty platform_toolsets.cli therefore loaded the DEFAULT profile's
full toolset list, inflating a tools-disabled profile's prompt from ~400 to
~15K input tokens.
Add api.config.get_config_for_profile_home() — a race-free direct disk read
of an explicit profile home's config.yaml (no shared-cache mutation), which
defers to get_config() when the requested home matches the ambient path so
in-memory test overrides are preserved. The streaming worker now resolves
_cfg from the session's own profile home, fixing toolsets, prefill context,
and fallback chains in one place.
Closes#3294
Co-authored-by: gottipx <gottipx@users.noreply.github.com>
* docs(changelog): v0.51.224 — #3294 profile toolset config (dropped #3405 to hold per Codex stale-model-repair finding)
---------
Co-authored-by: nesquena-hermes <[email protected]>
Co-authored-by: gottipx <gottipx@users.noreply.github.com>
* fix: reject cross-script drifted auto-generated session titles (#3293)
The title-language mismatch guard only knew two states: German (de) or empty,
and _title_language_mismatch early-returned False whenever the user start
wasn't German. So an English conversation whose LLM-generated title came back
in Chinese / Spanish / Russian sailed through and persisted with
llm_title_generated=true. The German case was the only one covered because
that's the one prior report it was built for.
Generalize from a German-specific binary to a language-agnostic cross-script
check. Add _script_counts() + _dominant_script() (cheap, dependency-free
Unicode-block classification: latin / cjk / cyrillic / arabic / hebrew / greek
/ devanagari). _title_language_mismatch now rejects a title that introduces a
substantial amount (>=35% of alphabetic chars, min 2) of a script different
from the conversation start's dominant script — so short titles that embed a
borrowed Latin technical term still trip, while an English title with a single
foreign place-name does not. The legacy German->English same-script heuristic
is preserved verbatim.
Kept api/streaming.py ASCII-only (the test_title_generation_source_has_no_cjk_
literals guard) — all CJK examples live in the test file, not the source.
Closes#3293
Co-authored-by: andrewkangkr <andrewkangkr@users.noreply.github.com>
* fix: prune orphaned imported-CLI sidecars from the WebUI sidebar (#3238)
When a CLI/agent session is opened in WebUI it gets a WebUI-owned sidecar
(webui/sessions/<id>.json + _index.json row) so it can render and reopen;
all_sessions() then returns it independently of the agent state.db. If the
user later deletes that session from the CLI / local Hermes storage, nothing
pruned the sidecar — the merge loop only overlays CLI metadata when a matching
state.db row exists and otherwise continues, so the stale row lingered in the
sidebar indefinitely (there is no WebUI delete affordance for CLI rows).
Add api.models.agent_session_row_exists(): an exact, uncapped existence probe
against the state.db sessions table. The sidebar merge loop now drops a row
that is_cli_session_row + not WebUI-native + absent from cli_by_id + whose
state.db row is genuinely gone, and calls prune_session_from_index() so
_index.json self-heals.
The state.db probe is deliberate: get_cli_sessions() caps at
CLI_VISIBLE_SESSION_LIMIT (20), so a still-existing session can fall out of
that window and look deleted — pruning on cli_by_id absence alone would delete
live sessions. WebUI-native rows with a CLI ancestor are never pruned, and any
probe error degrades to keep-the-row so a transient failure can't lose data.
Closes#3238
Co-authored-by: Luxciax <Luxciax@users.noreply.github.com>
* fix: count pin quota by visible session lineage
* docs(changelog): v0.51.222 — backend bugfix batch (#3293 title drift, #3238 sidecar prune, #3288 pin lineage)
* fix(pins): forks count as own pin lineage, not collapsed to parent (#3288 Codex follow-up)
Codex review of the batch found a pin-limit UNDERCOUNT: _session_row_lineage_root_id
followed any parent_session_id to the root, but /api/session/branch creates
independent visible fork sessions that also carry parent_session_id (session_source=
'fork'). Two pinned forks of the same parent collapsed to one quota lineage, letting
a user exceed pinned_sessions_limit with no 400. Fix: a fork returns its own id as
its lineage root (it's a separately-visible session); only compression/continuation
rows still collapse to a shared root. Adds a regression test with two pinned forks +
the parent counting as three distinct lineages, and confirms the existing
pre-compression-snapshot collapse case still passes.
* test(pins): update #2508/#2821 source-match tests for #3288 lineage rename
#3288 replaced the raw-session-id pin counter (pinned_ids set) with a
visible-lineage counter (pinned_lineage_ids via _visible_pinned_lineage_ids over
persisted_rows/candidate_rows). Two pre-existing source-string-matching tests
asserted the OLD implementation literals (pinned_ids = {, _session_field(existing,
session_id...), len(pinned_ids) >=). Updated both to assert the new mechanism while
preserving the invariants they actually guard: snapshot computed BEFORE LOCK (no
all_sessions()-inside-LOCK deadlock), quota filtering via the shared
_session_counts_toward_pin_quota helper, and the limit/400 guard. Behaviour
unchanged; these were implementation-detail assertions, not behaviour tests.
---------
Co-authored-by: nesquena-hermes <[email protected]>
Co-authored-by: andrewkangkr <andrewkangkr@users.noreply.github.com>
Co-authored-by: Luxciax <Luxciax@users.noreply.github.com>
Co-authored-by: Andy Kang <andrewkang.kr@gmail.com>
Manual title regeneration (POST /api/session/title/regenerate) and background
aux title generation failed with 422 / llm_error_aux when
auxiliary.title_generation.model in config.yaml used the WebUI-internal
@provider:model picker format (e.g. @gemini:gemini-3.1-flash-lite) — the raw
@-qualified id was forwarded to the provider API verbatim. Normalize it through
the canonical _split_webui_provider_model_value() helper before the aux call.
Co-authored-by: pamnard <pamnard@users.noreply.github.com>
Consecutive user turns (session-context prefill + actual message) made models
with strict chat templates (Mistral, Gemma via llama.cpp) reject the request
with a Jinja 500. Move the platform/delivery context (connected platforms, home
channels, scheduled-task delivery hints) from _webui_session_context_message
(a prefill user message) into _webui_delivery_context_prompt, appended to the
ephemeral system prompt. Session framing stays in _webui_surface_context_prompt.
Context is preserved — just role-alternation-safe. Closes#3276.
Co-authored-by: aether-agent <aether-agent@users.noreply.github.com>
_merge_display_messages_after_agent_result shared one _emitted set between
context-backfill rows and the visible-display backbone; _message_identity
ignores timestamps, so a second identical visible user turn was suppressed as
already-emitted. Now the display backbone is always appended in order; a
separate _context_inserted set + the _display_id_set guard dedupe only the
spliced-in context rows. Adds a regression test with two identical visible
user turns + a context-only gap.
Backfill model-context turns into the visible transcript before saving the
latest reply while keeping hidden [context compaction] markers out of the
visible transcript, so sidebar/header message counts no longer collapse a
compacted gateway session to a 2-message conversation. Resolves the
transcript-loss regression the v0.51.197 Codex gate flagged on the original.
Co-authored-by: AJV20 <24819659+AJV20@users.noreply.github.com>
Closes#3106. Adds /api/session/title/regenerate endpoint + session-action
menu item. Preserves chronology (touch_updated_at=False), guards read-only
and imported sessions, syncs to state.db when Insights sync enabled.
Maintainer refinement (Opus SHOULD-FIX): scope the is_imported guard to the
regenerate action only instead of broadening the shared _isReadOnlySession()
helper, which also gates rename/pin/archive/move/fork. Matches the backend
403 guard. Test updated to assert the scoped shape.
The default-only context_length guard compared model.default to the session
model with exact string equality. But model.default and the session model can
be stored in equivalent-but-different shapes (bare 'claude-opus-4.8',
provider-prefixed 'anthropic/claude-opus-4.8', or '@anthropic:claude-opus-4.8').
An exact compare wrongly treats the actual default model as non-default and
drops its configured context_length cap for provider-prefixed configs.
Add api/routes._model_matches_configured_default(session_model, cfg_default,
provider) that normalizes all three shapes, and use it at all 6 guard sites
(routes resolver + the 5 api/streaming.py sites: live-usage snapshot, persistence
_skip_cc_cl, persistence fallback _apply_cfg_ctx, SSE-done _dropped_stale_cap_sse,
SSE fallback _apply_cfg_ctx). Imported function-scoped in streaming to avoid the
routes<->streaming module-level circular import. 10 helper unit tests + a
behavioral test that a prefixed default still receives its cap.
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
The threshold-rescale block runs unconditionally after the fallback and
references _skip_cc_cl/_cc_cl, which were only defined inside 'if _cc_for_save:'.
On the no-compressor path (fresh agent / interrupted stream) that raised
UnboundLocalError (caught by test_issue1857_usage_overwrite). Hoist both inits
above the block (no-op rescale when no compressor). Also widen the #1318
source-assertion test to accept the widened fallback gate (still asserts the
falsy-check invariant).
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
Codex regression gate (+Opus, both independently) found the default-only guard
dropped the stale compressor cap but two sibling paths stayed inconsistent:
1. Per-turn persistence: fallback resolver only ran when context_length was
falsy, so a previously-persisted stale 232K survived forever on non-default
sessions. Now also runs when _skip_cc_cl, and rescales threshold_tokens to
the recomputed real cap (or clears it).
2. Terminal SSE 'done' payload: re-emitted the stale compressor threshold, so
messages.js overwrote S.lastUsage and the indicator reverted on stream end.
Now rescales threshold to the resolved window when the stale cap was dropped.
Added 3 source-structure regression tests pinning both fixes; bumped the brittle
test_pr1341 distance limit 13000→15000 (+ noted it should become structural).
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
The default-only guard corrected context_length to the real per-model cap
(e.g. 1M for claude-opus-4.7-1m) but left threshold_tokens pointing at the
ContextCompressor's stale value (computed from the global 232K cap → 197.2k
@ 85%). UI then showed 'auto-compress at 197.2k / 1M' which is misleading.
Rescale threshold_tokens by the real/orig ratio so the displayed trigger
reflects the actual window (e.g. ~850k @ 1M).
NOTE: this only corrects the SSE display payload. The real auto-compress
trigger lives inside ContextCompressor in hermes-agent (agent_init.py:1446
constructs it with the global cap). A full fix requires a parallel change
upstream — tracked separately.
_live_usage_snapshot() runs on every metering tick (~10x/sec while streaming).
The #3256 default-only guard recomputed get_model_context_length() there on
every tick for non-default models, which does a config read + potential
metadata/network probe — freezing claude-opus-4.7-1m streams while the default
model (4.8) stayed fast (guard not triggered for it). Resolve the real cap at
most once per stream via _real_ctx_cache. Backend-only, no frontend changes.
Re-applies cb0065eb + b34311b3 (context_length default-only guard) which were
dropped by the upgrade reset to v0.51.185. Fixes 4.7-1m context window showing
as stale global cap (232K) instead of real 1M metadata. Backend-only: touches
api/routes.py + api/streaming.py, zero frontend/render changes.
