`_lock_for` caches one threading.Lock per (dir, file, pid) in the module-global
`_WRITER_LOCKS`, but nothing evicted those entries: `delete_run_journal` rmtree'd
the on-disk `_run_journal/{sid}/` directory yet left the cached lock objects
behind, so a long-lived gateway leaked one entry per deleted run forever.
Delete now evicts every cached lock whose parent directory matches the removed
session (pid-independent) under `_WRITER_LOCKS_GUARD`, leaving unrelated
sessions' locks intact.
Refs #4633, #2097.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
start_drain_thread and start_session_channel_reaper checked is_alive()
and then created + started the daemon thread without holding a lock, a
check-then-act race: two concurrent callers could both see "not alive"
and each spawn a thread. The loser's thread was never stored in the
module global and ran forever, un-joinable by the matching stop_*. Both
check-then-start sequences now run under a dedicated
_THREAD_LIFECYCLE_LOCK (kept separate from the purpose-bound
SESSION_CHANNELS_LOCK / _EMIT_COALESCE_LOCK), so exactly one thread is
ever created.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BG_TASK_COMPLETE_EVENTS_SEEN gained a session_id -> set[process_id] entry the
first time a bg task completed for a session and was never deleted anywhere, so
it grew unbounded for the server lifetime.
The entry is created in _process_one for EVERY completion, whether or not any
SSE channel/tab exists, so pruning it only when the SessionChannel is reaped
would miss the dominant headless case (task fires, tab closed or never opened —
no channel to collect). Instead the reaper now sweeps the map by DELIVERY: once
a completion is drained (its session_id removed from PENDING_BG_TASK_COMPLETIONS)
the short _move_to_finished dedup window is closed and the entry is swept, every
tick, regardless of any channel. The registry's per-process_id
_completion_consumed gate remains the primary idempotency backstop, so sweeping a
delivered session's set can never resurrect an already-delivered completion.
Session deletion also prunes the entry (new forget_bg_task_completion_dedup),
covering a session deleted while a completion is still pending (which the
delivery-gated sweep deliberately retains).
Refs #4633
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The session-listing path already opens the live agent state.db read-only
(file:...?mode=ro) so a write-capable handle doesn't add checkpoint/lock
surface while the agent streams into the same WAL DB (#5455). Three pure-read
projections were missed and still opened a read-WRITE connection:
- read_session_lineage_report
- read_session_lineage_metadata
- the gateway-watcher fingerprint projection (a 5s poll)
Route all three through a shared open_state_db_readonly() helper that mirrors
the listing path: read-only file: URI with a writable fallback that warns. The
self-heal write path (missing idx_messages_session) is intentionally left
writable. No behavior change beyond open mode; reads perform zero writes.
Refs #5455
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MUST-FIX: /api/updates/check now accepts an explicit 'channel' in the POST body;
the Settings dropdown passes the just-picked value to checkUpdatesNow(channel) so
the immediate re-check can't race the debounced autosave PUT and answer for the
previous channel (verified live: body channel wins over stale saved setting).
SHOULD-FIX:
- copy: 'soaked' -> 'soak-tested'; append 'Applies to WebUI updates only.' to the
channel helper text (all 15 locales, native translations)
- mobile: .settings-version-badge gets max-width/overflow/ellipsis in the <=768px
query so the '· Experimental' suffix can't overflow a 320px viewport
(#5 badge-suffix i18n key deferred — non-blocking fast-follow per Fable, matches
the existing English-only version badge on master.)
Codex round-2 gate: apply_clear_lock re-entered _apply_update_inner(target)
without the channel, so an experimental WebUI lock-recovery retry silently fell
back to stable. Now passes _read_update_channel() through. + regression test.
Codex gate CORE fix: check_for_updates passed the user's WebUI channel into the
Agent repo resolution, so on 'experimental' the Agent ignored its v* tags and
fell back to origin/master. The Agent is a separate project (plain v* tags,
tracks master past tags) and must ALWAYS use DEFAULT_UPDATE_CHANNEL. Forced in
all 3 sites: check_for_updates agent leg + _apply_update_inner + apply_force_update
(target=='agent' -> channel=DEFAULT_UPDATE_CHANNEL). Added a real-git regression
proving Agent release/apply resolution is identical under both WebUI channels.
Add an update_channel setting (stable|experimental) that selects which git-tag
stream the self-updater tracks on the single linear master line:
stable -> 'v*' promoted, soaked releases (default; unchanged glob)
experimental -> 'exp-v*' every release batch (opt-in testers)
A channel is only a tag glob — no branches, no divergence — so every ff-only
guarantee (#2653/#2846/#3140) is preserved. Channel governs the WebUI repo only;
the Agent repo keeps its historical branch fall-through.
Correctness (advisor-reviewed, Codex + Fable):
- describe uses --match <glob> so a commit tagged both v* and exp-v* resolves
to the channel-correct tag
- stable never falls through to origin/master when HEAD contains the latest
stable tag (the firehose-suppression that makes channels work)
- apply_force_update refuses to reset --hard onto an ancestor ref (rewind guard)
- update cache + in-progress guard keyed by (channel, include_agent)
- channel display badge is a SEPARATE field; WEBUI_VERSION stays channel-neutral
(asset cache-busting / SW cache / stale-client skew all do exact-string equality)
Settings UX: channel dropdown with risk-forward copy + the ff-only asymmetry
helper ('switching back to Stable keeps your current version until Stable catches
up'), channel chip on the version badge, i18n keys, re-check on switch.
Tests: new tests/test_update_channels.py (12 real-git-fixture tests) + existing
update suite updated for the channel-aware signatures.
Strict providers (DeepSeek v4, newer OpenAI) reject an assistant message
carrying tool_calls: [] with HTTP 400. _sanitize_messages_for_api already
strips orphaned tool results, but not an assistant message that literally
stores an empty tool_calls list.
Drop the empty key on ALL THREE sanitized-dict sites that must agree:
_sanitize_messages_for_api, _api_safe_message_positions, and the
_safe_projection inside _restore_reasoning_metadata (so a tool_calls: [] row
projects identically on both sides of the metadata-carry-forward alignment and
doesn't lose its reasoning/id/timestamp every turn — Fable gate follow-up,
folded in).
Added mutation-checked regression tests: empty tool_calls dropped + populated
chain preserved (both sanitizer paths), and metadata carry-forward survives a
tool_calls: [] row.
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
A valid session:// deep link to a session owned by a DIFFERENT Hermes
profile used to look identical to a deleted session (404 -> frontend
self-heals away). Now GET /api/session returns a structured 409
session_profile_mismatch envelope (error/code/session_id/profile ONLY,
no transcript) ONLY when the owning profile is KNOWN, and loadSession()
catches it, switches to the owning profile, and retries once. Truly
missing/deleted or legacy None-profile sessions keep the 404 self-heal.
Gate fixes applied (Codex + Fable):
- Codex CORE: added a post-await stale-load guard after
_switchProfileForSessionLoad so a navigation during the switch can't
hijack the UI back to the old session.
- Fable Finding 1: only emit 409 when _session_profile is truthy;
a None-profile (missing/legacy) session under a non-default active
profile now keeps 404 instead of a useless profile=null 409 (which
skipped self-heal + spun the SSE reconnect against a dead sid). Both
detail branches. + 2 regression tests.
- Fable Finding 2: _switchProfileForSessionLoad now clears the sidebar
skeleton + re-renders from cache on switch-POST failure (mirrors the
#4671 canonical-switch catch), then rethrows, so a failed switch can't
strand the sidebar on the skeleton.
Reconciled tests/test_issue1611_session_profile_filtering.py (4 tests) to
the 409 contract while preserving the no-leak boundary assertion.
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
Add creationflags to _run_git() and _config_names_for_scope() so git
child processes don't accumulate hidden cmd.exe/conhost windows on Windows.
Uses a local module-level _windows_hide_flags() helper (mirrors the
api/updates.py pattern: CREATE_NO_WINDOW on win32, 0 elsewhere) rather than
importing from the OPTIONAL hermes_cli package — a standalone/agent-less
WebUI (and the repo's own CI runners, which don't pip-install hermes-agent)
must keep full workspace-git functionality. windows_hide_flags() is a safe
no-op (0) on non-Windows. Scoped to workspace git commands.
Adds mutation-checked tests asserting the kwarg is wired on both sites.
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
Per Greptile's review on #5717: the previous
`if result.get('messages') is not None` guard created an asymmetric
condition where the silent-failure classifier could miss the injected
fallback when the agent result had `messages: None` (which never
actually happens in practice since `finalize_turn` always returns a
list, but the asymmetry read as code smell). Drop the guard; the
writeback is now unconditional on `isinstance(result, dict)`.
Per the maintainer's review on #5717: there may be a
`finalize_turn` invariant on the agent branch this WebUI ships
against that already appends the fallback as an assistant row, in
which case the live bug is gone and this PR becomes a regression
guard. Reframe the test docstrings as dual-purpose pins of the user-
visible closure-text contract so they read correctly under both
interpretations.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
The WebUI already surfaces the model-generated summary when the agent's
tool-iteration budget is exhausted. The remaining gap: when the agent
hits the limit but its summary call yields no usable assistant content
(common with reasoning-only responses), the user sees a bare
`tool_limit_reached` error instead of any closure text.
`hermes-agent.handle_max_iterations()` always returns a non-empty
`final_response` (either the summary or a graceful fallback), and
`AIAgent` exposes that string on the result dict. The WebUI, by
contrast, only reads `result['messages']`, so an empty summary
left no final assistant answer and routed to the error path.
Inject `result['final_response']` as a final assistant turn when
`max_iterations_reached` fires and no usable answer exists in
`messages`. The existing `_mark_latest_assistant_tool_limit_status`
flow then attaches the status card and the user sees the closure
text instead of the bare error. Hermes-agent parity preserved (it's
literally the same string); the bare-error path stays for the case
where the agent died entirely (no fallback either).
Tests cover: fallback-injection surfaces closure text + status card,
no double-injection when a real summary already exists, the existing
'died entirely' apperror path is unchanged, and three unit cases for
the helper's skip conditions.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Add a complete Czech (cs) locale to hermes-webui: full key parity with the
English reference (1642/1642 keys), real Czech translations for all string and
function-valued keys, Slavic plural helpers for tool/worklog summaries, the cs
login-screen locale in api/routes.py, and a dedicated tests/test_czech_locale.py
parity+placeholder+diacritics guard mirroring the other per-locale tests. All
15 locale-count assertions bumped 14->15.
Co-authored-by: ostravajih <ostravajih@users.noreply.github.com>
Address round-2 gate cert (issuecomment-4896321817) on sha:dcec232d:
1. BRICK (Codex strace-verified): v2's fcntl-flock holder check on
.git/index.lock was wrong. Codex straced git 2.43.0 and proved
that git uses O_CREAT|O_EXCL + rename(2), NOT advisory locking,
so an fcntl-flock probe returns False for a lock file a live git
process is holding open. The 'fail-closed' claim was wrong:
auto-delete could still race and corrupt the index.
2. CORE (Codex): if .git/index.lock vanished between _is_lock_held
and os.remove, the FileNotFoundError surfaced as a 'remove-failed'
diagnostic instead of 'already gone' and prevented the apply retry.
Fix: stop deleting lock files from the server entirely. The only thing
that removes a lock is the user, on the host, via the manual command
surfaced in the response. Once the lock is gone, the user re-clicks
Update Now and the normal non-destructive apply path runs.
Implementation:
api/updates.py:
- Removed _is_lock_held (fcntl-flock probe) -- was provably unsafe.
- Removed _try_remove_lock (os.remove path) -- was provably unsafe.
- Removed _GIT_LOCK_FILES_REMOVABLE (no longer enumerable by name).
- Added _inventory_locks(path) -> dict: pure inventory of well-known
+ other lock files under .git/. Never touches anything.
- Rewrote apply_clear_lock to inventory-only + manual-instruction.
When .git/index.lock is absent, runs the normal non-destructive
apply path; when present, returns ok=False with the exact 'rm -f'
command + an explanation of why the server cannot do this safely
(O_CREAT|O_EXCL cannot be detected with advisory probes).
CORE-1 race evaporates: there is no os.remove to race against.
api/routes.py:
- Updated the /api/updates/clear_lock comment to reflect v2.2.
static/ui.js:
- applyClearUpdateLock: when res.lock_held, call new
_renderLockManualInstruction(target, res) which surfaces the exact
'rm -f <path>' command in a copyable code block plus an
'I've removed the lock -- retry update' button that POSTs the
same endpoint again; the second call takes the success branch
and re-runs the normal apply path. Also lists any other lock
files present so the operator can investigate.
tests/test_updates.py:
- Removed all 4 tests for the deleted v2 helpers (_is_lock_held
Returns*, _try_remove_lock*).
- Added test_v2_probe_helpers_removed: regression guard that
fails loud if either helper is ever reintroduced.
- Rewrote test_apply_clear_lock_removes_unheld_lock_and_runs_normal_update
and test_apply_clear_lock_refuses_when_lock_held for the v2.2
contract. The 'refuses' test monkeypatches os.remove to record
any deletion attempt and asserts the call list is empty -- a
strict runtime guard against future regressions to auto-delete.
- Added 7 new tests: test_inventory_locks_* (4), test_apply_clear
lock_with_no_lock_runs_normal_update, test_apply_clear_lock_with
lock_present_returns_manual_instruction, test_apply_clear_lock
listing_includes_other_locks, test_apply_clear_lock_rejects
unknown_target, test_apply_clear_lock_rejects_not_git_repo.
Verification:
./scripts/test.sh tests/test_updates.py -> 73 passed
./scripts/test.sh tests/test_update_banner_fixes.py -> 102 passed
./scripts/test.sh tests/test_api_timeout.py -> 8 passed
python3 scripts/ruff_lint.py --diff upstream/master -> 0 new violations
Full ./scripts/test.sh -> 12137 passed, 18 failed (all 18 pre-existing
flakes in unrelated subsystems; verified not v2.2 regressions)
Greptile rerun and the maintainer review both raised the same concern:
state.db and the sidecar self.messages can diverge by hundreds of
messages during recovery / mid-flight writes / pending_user_message.
On the test corpus:
0db167553ac7 db=11 sidecar=12
295592fd560e db=813 sidecar=889
7478cae31f01 db=2730 sidecar=2405
The previous patch's SQL path was reading state.db. The pre-patch
inline walk was reading self.messages (the sidecar). Sidebar stale-row
detection (_looks_like_stale_zero_message_row,
_row_may_need_sidecar_metadata_refresh) consumes this field as if the
sidecar were the source of truth, so swapping the source silently flips
the field's semantics.
Drop the SQL path entirely. The helper now does the same in-memory
walk the pre-patch code did, extracted into a named method for
visibility. The inline role check (dict.get('role') with default
empty string) is 1.7x faster than calling _message_role() per
iteration on the test corpus (2.57ms -> 1.49ms across 8 sidecars,
2405 msgs total).
Bench (2405-msg sidecar): 0.76ms per call, vs ~1.5s for the pre-patch
inline walk on the same sidecar (the original perf hotspot). The
1.5s figure included JSON parse, dict construction, and the SQLite
overhead of the now-removed SQL path; the bare walk is the
0.76ms shown here.
Correctness: 27/27 webui sidecars produce identical counts vs the
pre-patch inline walk. No behavior change for any caller.
Addresses Greptile findings (P1 silent-zero, P2 docstring, P2 dead
allowlist) by removing the SQL path entirely rather than patching
around its divergent-source issue.
Greptile flagged that _compute_user_message_count_lazy silently returns 0
when the SQLite query fails (DB locked, missing file, schema drift).
The pre-patch compact() walked self.messages in Python and returned the
correct count without ever touching the DB, so the silent zero
fallback is a regression that breaks _looks_like_stale_zero_message_row
under contention.
Fix: helper now takes an optional messages argument. On SQLite failure
it falls back to the same in-memory walk the original code used, so
callers see the same number they'd have seen before for any session
where Session.load() populated self.messages.
Also drop the dead ('GET', '/api/session') allowlist entry flagged by
Greptile — the /api/session handler already does its own per-stage
logging via the _t0.._t6 block (with auto-log on slow requests), and
adding RequestDiagnostics on top would just duplicate the slow-request
journal entries without adding information.
Address the RED gate cert on PR #5688 (2 BRICK + 1 CORE) and the
incoming Greptile review (1 P1 + 3 P2):
- BRICK-1 (race-safety): drop the mtime heuristic and any age-based
lock removal. Add /api/updates/clear_lock with a fail-closed
fcntl.flock holder probe (refuses if any process still holds the
file, on POSIX). On non-POSIX, fails closed. Touches only
.git/index.lock (the only well-known short-lived lock git creates).
- BRICK-2 (destructive path coupling): remove lock_conflict from the
force-button condition. Add a separate btnClearUpdateLock that calls
/api/updates/clear_lock -- never apply_force_update. The recovery
path never runs checkout/clean/reset --hard.
- CORE-1 (unconditional pre-cleanup): apply_force_update no longer
iterates .git/**/*.lock. Lock cleanup is exclusively the clear_lock
endpoint's job.
- Greptile P1: when a pull-lock error fires after a stash was pushed,
run git stash pop (with apply+drop fallback) so the user's local
modifications aren't stranded in the stash silently. v2 tests
cover the stashed and the clean cases.
- Greptile P2 (broad 'lock file' substring): tightened _GIT_LOCK_SIGNATURES
to specific git error strings ('index.lock': file exists, '.lock':
file exists, 'another git process seems to be running',
'unable to create .git/index.lock') so unrelated ref-transaction
lock-loss messages no longer trigger a lock-conflict response.
- Greptile P2 (redundant git_dir.exists / magic number): both gone as a
side effect of removing the apply_force_update cleanup loop entirely.
Refactor: rebased onto current upstream/master so PR is no longer
'behind' 130 commits.
Tests:
- 14 new v2 tests covering holder probe (true/false/NOGIL probe),
_try_remove_lock (refuse-on-held/success-on-unheld),
apply_clear_lock (success/refused), signature set parameter table
including false-positive class, apply_force_update no-touch
contract, pull-lock stash restore, pull-lock no-stash-when-clean.
- 2 v1 tests removed (they encoded the unsafe behavior).
- 2 v1 parametrize rows dropped (their positive cases relied on the
broad 'lock file' substring).
Closes#5687
- Detect lock file errors (lock_conflict flag) on fetch/status/pull
failures in _apply_update_inner (#1)
- Expose Force Update button on frontend when lock_conflict is
received (#3)
- Proactively clean stale lock files (>30s old) in apply_force_update
using rglob discovery instead of a hardcoded path list (#2)
- Add 10 tests: _is_git_lock_error parametrized unit tests,
_apply_update_inner lock path coverage, and apply_force_update
stale/recent lock behavior
Session.compact() called _last_message_timestamp(self.messages) which
reversed-walked all 2,730 messages on every /api/session response.
The messages array is chronologically ordered, so the last non-tool
message sits at the very end. Scan only the last 8 messages; fall
back to a full scan only if no timestamp is found in the window
(unusual sessions with >8 trailing tool rows).
Reverts to identical behavior when the window hits, since the most
recent user/assistant message's timestamp is what compact() emits.
Bench (single-shot, live Chromebook eMMC):
before: get_session=1107ms compact=1725ms total=2880ms
after: get_session= 648ms compact= 288ms total=1313ms
(-80% on compact, -54% on total)
Revert: git checkout master && systemctl --user restart hermes-webui
Adds auto-logging when total /api/session latency exceeds 2s, so we don't
need HERMES_DEBUG_SLOW env var to diagnose regressions in production.
Env var still forces logging on every request for development.
Baseline data on the Chromebook (Celeron N3350, eMMC storage):
- long_session_idle (2,445 msgs) p50=5.8s, p95=9.1s, max=9.7s, 2/10 timeouts
- stage breakdown for a 2.8s request:
get_session=1041ms, compact=1717ms, redact=31ms, json_write=8ms
- compact stage iterates all messages for user_message_count and runs
recursive redact_session_data + json.dumps -- biggest single cost
Ref: perf/session-load-latency
The slow-request journal showed /api/profiles, /api/models, and /api/session
fire on every session click but had no per-stage timing. Add:
- RequestDiagnostics coverage for GET /api/profiles, /api/models, /api/session
- Stage markers inside /api/profiles (list / active lookup / isolated check)
- Stage markers inside /api/models (freshness routing, serialize)
- Per-stage stage-log inside get_available_models_for_session_visit that
emits a [SLOW] line when total wall time crosses HERMES_DEBUG_SLOW ms
(default 500ms) so we can pinpoint which sub-step is blocking
No behavior change. Existing _t0.._t6 timing on /api/session and the
2.5s/10s TTL on /api/sessions are unchanged. Reversible: revert commit.
Relaxes the /api/session/branch read-only gate so a canonical-cron session can be forked into a new WebUI-owned session (source never saved). Gate hardening folded in: the read-only branch gate now covers BOTH the synthesized/missing-sidecar path AND the persisted/loaded path (Codex found a stored read-only non-cron session could be branched + saved); relaxation keys on the server-authoritative resolved source (source_tag/raw_source/source == 'cron'), never the id prefix. Full gate: Codex adversarial SAFE, 84 branch/claim tests + suite 12175/0, browser clean.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
get_state_db_session_messages() returned archived (active=0) rows that in-place
compaction had marked inactive. WebUI reconciliation feeds this reader straight
into the next model-facing context (reconciled_state_db_messages_for_session,
prefer_context=True), so those archived rows got pulled back in — resurrecting
pre-compaction history and making every later turn re-trigger compression. Now,
WHEN the messages table exposes an `active` column, inactive rows are excluded by
default (`AND (active IS NULL OR active != 0)` — NULL/legacy rows preserved), with
an `include_inactive=True` escape hatch for explicit recovery/audit callers.
Schemas without the column are unaffected (guarded by `'active' in available`).
Contributor stage; ungated on arrival. Co-authored-by: ai-ag2026 <ai-ag2026@users.noreply.github.com>
_webui_sidecar_lineage_messages_for_display stitches a session's pre-compression
snapshot parents for full-transcript display. A cleared fork child could resurrect
its ORIGINAL (non-fork) parent's transcript when that parent was later compressed.
Now, when the root session is itself a fork (session_source=='fork'), the stitch
walk allows fork-sourced snapshot parents but stops at the first non-fork
ancestor — so a compressed fork continuation still shows its own pre-compression
turns, while an ordinary fork child stays isolated from the original parent.
Additive guard (one extra break condition); non-fork lineage display unchanged.
Contributor stage; gate-pass. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
OpenCode Go's /v1/models endpoint returns models from the full public catalog that
are NOT enabled on the Go tier; selecting any such probe-only model 404s
('model not found') when the chat request is sent. get_available_models now skips
the live probe for opencode-go (bare branch) so the next `if not raw_models` falls
through to the curated static _PROVIDER_MODELS['opencode-go'] list — the actual
Go-tier models. Narrow, provider-scoped; no other provider affected.
Contributor stage; gate-pass. Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
The initial fix added ~16 lines to server.py, tripping test_server_py_under_750_lines
(server.py must stay a thin dispatcher; logic belongs in api/). Moved the whole
preflight-response into apply_cors_preflight_headers(handler) in api/routes.py;
server.py's do_OPTIONS is now a 4-line dispatcher. server.py 764 -> 746 lines.
Same security behavior (same-origin/allowlist echo, Vary: Origin, no wildcard,
header-less denial); test rewritten to the header-emitting API + added Vary and
no-header-on-denial assertions and a routes.py wildcard regression guard.
do_OPTIONS answered every CORS preflight with Access-Control-Allow-Origin: *,
advertising broader cross-origin access than a real request is granted — on a
password-less deployment that lets any site read authenticated responses. It now
echoes the request Origin only when _check_same_origin_browser_request approves
it (the exact same-origin / HERMES_WEBUI_ALLOWED_ORIGINS policy the CSRF gate
enforces for real requests), adds Vary: Origin, and never emits *. A disallowed
origin gets a 200 with no CORS headers (browser treats as preflight denial).
Default deployments (no allowlist) are unaffected.
Contributor stage; gate-pass. Co-authored-by: mo7al876any <mo7al876any@users.noreply.github.com>
Codex gate CORE: the exact-empty sentinel only suppressed restore for a bit-empty
cleared sidecar. If the user cleared, then sent ONE post-clear message, and the
stale pre-clear .json.bak still existed (unlink failed), recovery saw
bak_count>live_count and RESTORED the pre-clear transcript on top of the new
message. Added _live_supersedes_backup_by_clear_generation: suppress restore when
the live sidecar carries a clear_generation the backup lacks AND has a non-empty
transcript AND still shows the clear boundary reset (watermark==boundary==0.0).
Scoped to non-empty live so empty clear-shaped sidecars keep their existing
recovery semantics (active/pending/partial variants still restore). Fails open on
unreadable/partial reads. +2 regression tests (post-clear-message not restored;
post-clear-after-real-compaction still restores).
Closes the crash-window resurrection corner in the #5532 clear-conversation
data-loss fix. /api/session/clear writes a .json.bak before truncating, and
because the cleared sidecar's truncation_watermark is 0.0, startup recovery saw
bak_count > live_count and could restore the backup — resurrecting the cleared
transcript after a crash/restart in the clear window.
Fix: stamp a unique per-clear `clear_generation` UUID on the sidecar when
clearing a session that had messages (persisted via Session model). Recovery's
_session_records_clear_sentinel() returns no_action when the live sidecar carries
a clear_generation the backup doesn't share AND the live sidecar matches the
exact cleared-empty shape (messages/context empty, watermark/boundary 0.0,
pending fields cleared). Same-generation backups stay recoverable; unreadable or
partial matches FAIL OPEN (normal recovery), so a real crash-loss is never
suppressed.
Contributor stage; gate-pass. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
The streaming settlement path could show a misleading "No response from provider"
error card AFTER a completed answer, because a stale terminal/partial result flag
still forced the generic no_response error path even when the merged current turn
already had a final assistant answer. Now the generic no_response terminal error
is suppressed ONLY when the already-merged current turn has a completed assistant
reply (scoped to soft `partial` results per the issue-thread recommendation),
letting normal `done` settlement continue. Real silent failures, partial
failures, auth/quota errors, cancellation, replayed rows, and terminal
tool/compression cases all keep their existing error behavior.
Self-built stage of contributor PR; gate-pass. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
'All 0 credential(s) exhausted for <provider>' did not match _is_quota_error_text
so it fell through to a generic 'Error' with no hint (the exception path's else
branch discarded the hint entirely). Adds a distinct credential_pool_empty
classification (ordered before the quota check) + an explicit exception-path
branch + an actionable pool-specific hint, so the user sees why the turn failed
and what to do. Does not over-match (doesn't swallow real quota errors) and
doesn't block partial-harvest; downstream tolerates the new type.
NOTE: this is the CLASSIFICATION half of #3929 only — the data-loss half
(reasoning/partial-work restore on reload-reconcile) is a scoped follow-up that
needs the reporter's server logs.
Self-built (nesquena-hermes); nesquena independent review APPROVED.
An expired-auth 401 on the login page fed the login redirect its own address:
each of the three redirect guards (_safeNextPath in login.js,
_safe_login_redirect_path in routes.py, the api() 401 redirect in workspace.js,
_safe_login_inner_next in auth.py) validated against open-redirect but none
rejected a `next` pointing back at the login page, so every auth bounce wrapped +
re-encoded the previous login URL one level deeper — growing the URL exponentially
until the browser's length limit broke the tab. All guards now collapse/reject a
`next` whose decoded leading PATH resolves to the login route (bounded
percent-decoding up to 8 levels, fail-closed at the cap), while preserving the
existing open-redirect protections AND legitimate non-login paths that merely
carry their own `next=` query key. Length cap as belt-and-suspenders.
Self-built (nesquena-hermes); nesquena independent review APPROVED. Gate (Codex)
found + fixed an over-broad nested-next reject (regressed next-carrying paths) and
a deep-encode decode off-by-one; both root-fixed, login guards verified clean.
profile_env_for_background_worker mirrored the profile home into process-global
os.environ["HERMES_HOME"] and yielded the worker body OUTSIDE the setup lock, so a
concurrent cross-profile worker (e.g. background title generation on another
profile) could clobber os.environ mid-body — the agent config reader then resolved
the WRONG profile's config (intermittent turn-init failures citing an unused
provider; b3nw's v0.51.849 repro). Now installs hermes-agent v0.18.0's
context-local home override (set_hermes_home_override, a ContextVar that
get_hermes_home() consults before os.environ) for the resolved non-default profile
home — set before yield, reset in finally — so a config read in the worker body
resolves THIS profile's home from task-local state, immune to a concurrent
os.environ clobber. No worker serialization, no os.environ mutation, no detection
heuristic (both earlier WebUI-only mitigations were rejected as regression-unsafe).
Graceful degradation: the override module is resolved lazily+optionally
(_resolve_hermes_home_override, mirroring _resolve_secret_scope_module). On agents
< v0.18.0 the symbol is absent -> returns None -> pre-existing os.environ-mirror
behavior unchanged (no regression, no new failure mode).
Self-built (nesquena-hermes); nesquena independent review APPROVED.
Codex round-3 gate finding: _dedupe_replayed_context_messages deep-copies the
stale-user repaired boundary row (streaming.py:4501) into the context array. When
the mint ran AFTER dedupe, that deep-copied context row was no longer the shared
result dict, so it stayed id-less while the display copy got the minted id —
re-opening the cross-array id divergence for the stale-repair path. Fix: move
_assign_stable_message_ids() to run immediately after _restore_reasoning_metadata
and BEFORE _dedupe_replayed_context_messages at all four commit sites (streaming
main / in-band self-heal / except-path self-heal + routes runs/MoA), so the shared
result rows are stamped before any deep-copy — both arrays inherit the id.
Codex/Opus gate finding: in eager session-save mode the current user turn is
checkpointed into s.messages before the agent runs (no id yet); after
_assign_stable_message_ids stamps the returned result row, the display merge
keeps the durable eager checkpoint and skips the stamped result — leaving the
display row id-less while its context twin carries the minted id, silently
defeating id-based fork/truncate alignment for eager-mode users. The merge now
copies the minted id onto the kept checkpoint when it lacks one (guarded so it
can't overwrite or duplicate). Opus verified cross-array id pairing holds across
the real 3-turn transform pipeline.
WebUI keeps two parallel arrays per session: messages (display transcript) and
context_messages (what's sent to the model). In large/compacted sessions they
diverge. Forking / in-place truncation copies a prefix of both and relies on
truncate_context_for_display_keep() to translate a display keep-index into the
matching context index — an aligner that prefers a stable per-row `id`. But the
model-context rows carried neither id nor timestamp (a 989-session live scan
found 0 with any id), so alignment fell back to fragile content-signature
matching that goes ambiguous on repeated tool calls / empty assistant turns.
This closes the DATA gap (the alignment logic itself already prefers `id` and
handles the compacted case via the matcher, shipped in #5563): mint a monotonic,
session-unique integer `id` on the per-turn result rows AFTER the context restore
and BEFORE both arrays are built, so the display and model-context copies of a
logical row share the same id.
- api/streaming.py: new _assign_stable_message_ids(); _restore_reasoning_metadata
carries `id` forward across turns (as it already does timestamp); wired into all
three streaming commit sites (main turn, retry, self-heal).
- api/routes.py: same mint on the runs/MoA _handle_chat_sync commit path.
- api/gateway_chat.py: mints ids on the two new rows of a gateway turn.
- The `id` is stripped before the provider API call (not in _API_SAFE_MSG_KEYS),
same treatment as timestamp — nothing new reaches the provider.
session_ops.py alignment was already delivered by #5563 (id-preferring matcher +
compacted-case fall-through), so no change is needed there now; tests updated to
assert the post-#5563 behavior (id-bearing = exact cut; id-less = errs toward
under-keeping, never the old raw-index mis-cut).
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Move the 1-5s commit_session_memory() extraction off the POST /api/session/new
request thread (was blocking + New Chat) into a fire-and-forget daemon thread,
guarded by the existing per-session in-flight serialization.
Data-loss prevention on managed stop:
- server.py installs a SIGTERM handler that requests an orderly stop so
serve_forever()'s finally block runs drain_all_on_shutdown() (the default
SIGTERM disposition terminated without unwinding, losing in-flight commits).
- session_lifecycle: bounded self-unregistering background-commit registry;
a _draining flag so no NEW worker starts once the drain begins (the inline
generation-drain commits any late arrival instead); a 30s overall drain
deadline so a stuck worker cannot hang the stop path forever.
Co-authored-by: luperrypf <luperrypf@users.noreply.github.com>
Fork ("Fork from here") on a large/compacted session could cut context off a
real turn boundary — slicing mid-turn or leaving a dangling tool call, feeding
a malformed context to the model. Align the fork cut to a resolved turn boundary
via a signature matcher handling both compacted and non-compacted display
coordinate spaces; errs toward under-keeping with send-time sanitization backstop.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Rebuilt the combined #5556+#5553 fix on v0.51.859 (now has the flake fix + the
#5542 RFC de-brittle, so no anchor-test collateral). Nathan's call: rodboev's
#5556 primary + fold self-built #5553 improvements, credit both.
Clear handler now:
- routes through shared truncate_session_at_keep(s,0) (single source of truth,
sets watermark=_truncation_watermark_for([])==0.0, the #2914 sentinel that
blocks state.db append-merge replay)
- detaches compression lineage ONLY when the parent is a pre_compression_snapshot
(Codex-caught: preserve genuine fork parent links for nesting + "Forked from")
- rodboev's persisted-clear read-back verification + stale .bak removal
RFC conflict resolved in favor of the de-brittled (symbol-anchor) master version.
Ships both PRs' test files (state_db_replay + clear_truncation_watermark incl the
fork-preservation regression). Opus (on the amended tree): both resurrection paths
closed, detach lineage-safe, verification sound — SHIP. 3 non-blocking follow-ups
filed (#5570 .bak crash-window, #5571 fork-stitch corner, #5572 messaging clear).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Restore the compact() source-order expectation used by the branch metadata contract test while keeping focused compression recovery markers in the compact payload.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Make focused compression-recovery continuation creation idempotent by marking child sessions and reusing the existing child on repeated start requests. Also surface malformed successful recovery responses in the UI instead of silently re-enabling the action.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
When the WebUI server is launched from a venv python whose PATH does not
include git (e.g. hermes-agent venv on Windows), shutil.which('git')
returns None and WEBUI_VERSION falls back to 'unknown'. That freezes the
?v=<stamp> static-asset cache key, so browsers keep serving stale cached
ui.js/messages.js even after the server is restarted with fixed code —
frontend fixes silently never reach clients (recurring scroll jump-back).
Add a Windows fallback to _resolve_git_executable(): read the
Git-for-Windows registry InstallPath (HKLM + WOW6432Node + HKCU), then
probe common install dirs. Mirrors the existing darwin /usr/bin/git
fallback. No behavior change on posix or when git is already on PATH.
Codex 5th CORE: the ownership set only covered static _PROVIDER_MODELS, not the
active provider's own config.yaml providers.<active>.models allowlist. So an
active provider defined purely via providers: (no static catalog) could be
hijacked by another slug's overlapping entry earlier in config order. Fix folds
the active provider's own providers: allowlist (list + dict shapes, copilot
excluded) into _provider_models_set — completing the ownership model so the
active provider owns its declared models from EITHER the static catalog OR its
own providers: entry. Regression test (fail-without-fix verified). 40 model_resolver
tests green.
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Codex 4th CORE (root latent bug in the shared canonicalizer): _canonicalise_provider_id
only accepted an alias resolution if the target was in _PROVIDER_DISPLAY. But some
canonical targets (e.g. 'gemini') are indexed in _PROVIDER_MODELS and NOT
_PROVIDER_DISPLAY, so 'google-gemini'->'gemini' was rejected and returned raw
'google-gemini' — leaving the ownership guard empty and letting providers.openai.models
hijack an active Gemini-owned model. Fix mirrors the direct-hit check (line 1445):
accept alias targets in EITHER _PROVIDER_DISPLAY OR _PROVIDER_MODELS. This is the
root fix for the whole alias-canonicalization class (gemini/google-gemini/
google-ai-studio etc.), not just the reported site. Blast-radius verified: 81
canonicalise/provider-alias tests across 5 files still green; x-ai still stays x-ai,
opencode_go still folds. Gemini regression test added (fail-without-fix verified).
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Codex 3rd CORE (pre-existing latent bug the providers: scan exposes): the
_provider_models_set ownership lookup used the RAW config_provider (e.g. 'z-ai')
but _PROVIDER_MODELS is keyed by canonical slug ('zai'), so an aliased active
provider got an empty ownership set -> _skip_custom_providers guard-2 silently
failed -> another providers.<slug>.models entry could hijack an active-owned
model (verified: active z-ai + glm-5 + providers.openai.models.glm-5 -> openai).
Fix: canonicalise config_provider before the _PROVIDER_MODELS lookup (and reuse
the canonical slug for the scan's ownership guard). Regression test with an
aliased active provider (verified fail-without-fix).
This is the root-cause fix for the alias class — the ownership guard now holds
for every aliased provider (zai/kilo/etc.), not just the reported site.
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Codex 2nd CORE: the new providers: scan didn't honor the _skip_custom_providers
active/default ownership guard (config.py:2535) that the custom_providers scan
uses. So with an active provider (e.g. ai-gateway) + model.default=gpt-5 AND a
providers.openai.models.gpt-5 entry, the scan hijacked gpt-5 to openai instead of
the active ai-gateway path. Fix: when _skip_custom_providers is true, the
providers: scan only considers the active provider's own canonical slug (skip all
other slugs). Regression tests: active-provider-owns-default not hijacked (fails
without fix); active provider's OWN providers: entry still matches.
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Gate-cert (nesquena-hermes) + Codex found 1 CORE: akay64's new providers: scan in
resolve_model_provider() iterated ALL providers.<slug>.models INCLUDING
providers.copilot.models — but that key is a documented per-model SETTINGS map
(reasoning_effort/limits), NOT a routing allowlist (config.py exception). So a
Copilot per-model settings entry hijacked that model's routing to Copilot
(Codex reproduced gpt-5 -> copilot). Fix (Codex-exact): skip copilot in the scan.
Added the regression tests the PR shipped 0 of: user-defined allowlist routes;
active-openai + providers.copilot.models (dict + list shapes) stays openai
(both verified to fail without the exclusion).
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Codex v4 SILENT finding: my audit Finding-2 fix made the orphan-.bak branch emit
a state_db_deleted_webui_tombstone item, but the state_db_missing_rows loop still
emitted the same sid/kind again when both a surviving .json.bak AND a state.db row
exist for one deleted session — double-counting unsafe_to_repair, violating the
no-double-classification invariant. Fix: track _bak_tombstoned_ids from the .bak
branch and skip them in the state.db loop. Regression test
test_audit_no_double_count_when_bak_and_state_db_row_both_survive (verified fails
without the dedup).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
@rodboev independently pushed fixes for messaging-tombstone (matches ours) and an
audit orphan-.bak skip — but the audit skip used _marks_deleted_webui_session
(index heuristic), which re-introduces the crash-suppression bug, and the other 4
resurrection/recovery paths were still open. Rebuilt the stage on rodboev's new
head (fc21c3f3, attribution preserved) and applied the complete superset:
- routes.py claim path: tombstone 404 gated on _state_db_session_source in
('','webui','fork') so foreign-source rows always materialize (finding B)
- models.py sidebar projection: skip tombstoned source='webui' rows w/ no live
sidecar so a deleted session can't resurface in /api/sessions (finding C)
- session_recovery.py orphan-.bak startup: skip restore when durable tombstone
present, durable-only (finding D)
- session_recovery.py audit orphan-.bak: durable-only classify as
deleted_session_skipped (corrects rodboev's index-heuristic version)
- session_recovery.py _read_state_db_missing_sidecar_rows: durable-tombstone-only
so a genuine crash (index intact, no durable tombstone) still repairs (finding F)
Regression tests (all verified fail-without-fix): sidebar-projection, orphan-.bak
startup skip, messaging-not-tombstoned, index-only-crash-repairable. Both advisors
(Codex SAFE TO SHIP + Opus ship-with-fixes) cleared the prior stage; this rebuild
carries the same fixes onto the contributor's latest work.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Mirrors the existing custom_providers: scan immediately above. User-defined
providers declared under config.yaml's providers: section with explicit
models: lists were invisible to model-to-provider resolution — bare model
IDs fell through to the stale config_provider, routing through the wrong
endpoint.
The insertion site sits after custom_providers: (unchanged) and before
all heuristics (@provider:model, slash, fallback — all unchanged).
- Add test_profile_model_selection_accepts_model_in_overflow_extra_models
(the extra_models overflow-bucket regression test the warm dossier flagged
as the #5453 ship-pass note).
- Drop the stray dead pseudo-code comment left in handle_post branch route
(#5449 cosmetic nit).
read_importable_agent_session_rows() is a pure read, but it opened a
read-write sqlite connection on the live multi-GB WAL state.db and re-ran a
defensive CREATE INDEX self-heal on every sidebar build. Holding a
write-capable handle while the agent streams into the same DB adds needless
checkpoint/lock surface.
Open the DB read-only (file:...?mode=ro) for the projection, and self-heal a
missing idx_messages_session through a separate short-lived writable
connection only. With the index present (the normal case) the read path
performs zero writes; when it is missing the self-heal still runs and rows
still come back.
Behavior-neutral: identical rows when the index exists; self-heal preserved
when missing; read-only-fs degradation to the pre-aggregated path unchanged.
Tests: tests/test_issue5455_listing_readonly_connection.py (read-only open,
no writable connection when index present, self-heal when missing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
