Codex found a verified lifecycle hole: the bare boolean model_explicitly_picked
was not cleared on /api/session/update (api/routes.py:14438-14460 changes+persists
the model/provider without touching it), so after an update+reload the flag stayed
True and a cold #433 stale leftover (openai/gpt-5.4) was wrongly preserved instead
of stripped to gpt-5.4.
Fix (Codex's prescription): replace the unscoped boolean with a SIGNATURE bound to
the routing context.
- api/models.py: new model_explicit_pick_signature(model, provider) ->
'<model>\x1f<provider>'; Session.model_explicit_pick_signature persisted field
(replaces the boolean; in METADATA_FIELDS, restored via kwargs, survives restart).
- api/routes.py /api/chat/start: on a fresh explicit_model_pick, stamp the signature
of the resolved model+provider.
- api/streaming.py: treat the send as a deliberate pick ONLY when the persisted
signature equals the signature recomputed from the session's CURRENT persisted
model+provider. Any model/provider change (chat-start, session-update,
normalization, provider repair) yields a different signature and auto-invalidates
the stale pick — no per-mutation-site clearing needed.
So: deliberate pick on unchanged context -> preserve (b3nw); any switch away ->
signature mismatch -> unmarked -> legacy strip (#433 stays correct even after a
session-update+cold-restart). Adds signature match/invalidate/persist tests.
75 focused tests pass (resolver + sprint40 + the 2 streaming-mock suites).
Nathan's call: adopt Codex's complete mechanism instead of the unconditional
cold-preserve. The cold-catalog custom-proxy decision is now gated on whether the
user DELIBERATELY selected the model this session, closing the residual where a
stale #433 leftover would preserve+400 on a cold+no-disk send.
Mechanism:
- resolve_model_provider(model_id, *, explicitly_picked=False): in the custom-arm
cold branch (no warm provenance, not config-declared), explicitly_picked=True
preserves the vendor namespace verbatim (deliberate pick, proxy routes on it);
otherwise the legacy redundant-prefix strip runs (stale cross-provider leftover,
#433). Warm endpoint-advertised provenance still wins over the flag.
- Session.model_explicitly_picked: new persisted field (default False, restored
from metadata via **kwargs, listed in METADATA_FIELDS so it survives a restart
— b3nw's exact cold scenario).
- /api/chat/start stamps it: True on a fresh explicit_model_pick; cleared only
when the resolved model actually CHANGED without a pick (real switch away);
preserved across plain follow-up sends of the same model (the onchange marker
is one-shot, so per-send explicit_model_pick alone would drop the intent).
- streaming.py reads s.model_explicitly_picked and passes explicitly_picked= into
resolve_model_provider (inside the profile scope).
Independent end-to-end matrix (all pass): ben default cold, ben picked non-default
cold+warm, ben stale cold (strips), #433 warm+cold, #3872 bedrock, #548 zai-org.
67 resolver+sprint40 tests pass incl. new explicit-pick + stale-companion +
persistence round-trip.
Codex gate found two real issues in the send-path warm:
1. Blocking lock wait (CORE): get_available_models(prefer_cache=True) still
acquires _available_models_cache_lock and can wait up to ~60s (unbounded in
synchronous rebuild mode) on an in-flight rebuild — unacceptable on the send
hot path. Rewrote warm_models_catalog_provenance_if_cold as a genuinely
non-blocking, disk-only publish: try the cache lock NON-BLOCKING (return
immediately if busy — a concurrent build will publish provenance itself),
read ONLY the on-disk cache via _load_models_cache_from_disk (no network, no
rebuild), publish snapshot+fingerprint + _sync_models_cache_provenance().
Verified: returns in ~3.5ms while the lock is held (was a 60s block).
2. Profile scope (CORE): the streaming worker is a separate thread that does NOT
inherit the HTTP handler's request-profile TLS. Both the warm and the resolve
read profile-keyed config (cache path + source fingerprint via
get_active_profile_name), so a cold send from a NAMED profile could resolve
against the DEFAULT profile's config and route to the wrong provider/base_url
(get_available_models reloads config on path mismatch). Wrapped warm + resolve
in profile_scope_for_detached_worker(_resolved_profile_name) so both see the
owning session's profile (no-op for default/root).
Tests: rewrote never-live-rebuilds (now asserts warm NEVER calls
get_available_models — reads disk directly) and added a non-blocking regression
(warm returns <2s while the cache lock is held). 65 resolver+sprint40 tests pass;
b3nw end-to-end + non-blocking + disk-warm all verified.
The earlier #5979 fix (PR #5980, exp-v0.52.53) resolved the WARM-catalog and
config-declared cases, but b3nw confirmed truncation persists on exp-v0.52.55/56.
Root cause (reproduced on master, confirmed by Codex + Fable):
b3nw's config is provider: custom:llm-proxy, model.default: x-ai/grok-4.5, no
models[] allowlist. His DEFAULT (x-ai/grok-4.5) preserves via config-declaration,
but a model he SELECTS in the picker that isn't the default (x-ai/grok-composer-2.5-fast)
truncates on every COLD send. The send path (api/streaming.py:7857) calls
resolve_model_provider without warming the catalog, and the #1855 chat/start fast
path deliberately skips the catalog when a session already has a persisted
model+provider -- so 'cold at send' is the designed state after any restart /
settings-save / cache-expiry, not a rare race. With provenance None and no config
declaration, the #5980 tri-state fell through to the LEGACY family strip
(if prefix in _PROVIDER_MODELS and _is_first_party_model: return bare), and
grok-composer-2.5-fast had graduated into the x-ai first-party catalog -> stripped.
Fix (two changes, both advisor-verified):
1. Flip the custom-arm cold default from family-strip to PRESERVE-verbatim
(api/config.py). For an explicit custom / custom:<slug> proxy the strip now
requires POSITIVE provenance (endpoint advertises only the bare id, #433). A
wrong strip destroys routing info the user can't recover; a wrong preserve
fails loudly and self-heals on warm. This also honours _endpoint_advertised_model_ids's
own documented 'None -> preserve verbatim' contract and removes the data-driven
flaw where a model graduating into the static catalog silently flips routing.
2. Network-free provenance warm on the send path
(warm_models_catalog_provenance_if_cold, called in streaming.py before the
resolve). prefer_cache=True republishes provenance from the durable disk cache
(no live probe, no _cfg_lock held on the worker thread -> no lock-order
deadlock), so #433's bare-only-strip stays exact whenever a disk cache exists.
Residual (accepted, strictly better than master): a #433-shaped wrong-preserve
now requires memory cold AND disk cache absent/invalid (first run, post-config-save
window, schema/version bump). It fails loudly and heals on the next /api/models or
session-visit rebuild. Under master b3nw's failure was the mirror image but silent,
recurring every send, and unfixable short of declaring every model in config.
Supersedes rodboev's #5996, which targeted the providers.<slug> arm and does not
cover b3nw's actual custom:llm-proxy config (confirmed by Codex).
Tests: rewrite the cold-fallback test (strip -> preserve), sharpen the
foreign-profile test so trusted-strip vs ignored-preserve genuinely diverge, add
b3nw's exact cold non-declared custom:slug case, and add three warm-helper tests
(publishes-from-disk, noop-when-warm, never-live-rebuilds). 64 resolver+sprint40
tests pass; b3nw end-to-end verified; #433 positive-provenance strip verified.
Thanks @b3nw for the CLI-vs-WebUI comparison that localized it.
* fix(#4685): meter post-compression model context
* Release exp-v0.52.48: meter post-compression model context (#5872, #4685)
---------
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
* fix(#5940): surface non-retryable provider error instead of no_response
The Agent emits a non-retryable terminal error (e.g. HTTP 400 invalid model / no
credentials) via its lifecycle status_callback, but the WebUI dropped it and the
run result/_last_error were empty for that abort path -> turn completion showed the
misleading no_response 'silent rate limit, try again' message. Capture the emitted
terminal error in _agent_status_callback and seed _last_err with it at completion so
_classify_provider_error reports the real cause (model_not_found / auth) with
actionable guidance. +regression test.
* Release exp-v0.52.47: surface non-retryable provider error instead of no_response (#5940)
---------
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Codex re-gate caught 2 defects in the prior UX commit:
1. streaming.py:8921 — the disclosure fired for ANY process_wakeup failure (e.g.
rate-limit) even when no pause was recorded -> falsely claimed 'retries paused'
while process_wakeup_pause stayed empty. Now gated on the record_* return value.
2. streaming.py:10130 — the raised-exception credential_pool_empty path (exercised
by the existing agent test) recorded the pause but omitted the disclosure. Added
the same conditional disclosure there.
Both paths now keep _error_payload['hint'] in sync with the persisted bubble, and
the disclosure appears only when a pause was actually recorded. 69 tests green.
Fable UX-gate follow-up on #5732: after the first credential-exhausted error,
automatic wakeups are silently suppressed. Disclose it in the wakeup-source
error card ('Automatic retries for this conversation are paused until you send
a message, switch the model/provider, or fix the credentials.') so the silence
reads as intentional, not a stuck agent — Gmail/Dropbox account-needs-attention
convention. Scoped to the process_wakeup + credential_pool_empty seam only.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
WebUI hard token-cap on protected-tail tool-result payloads in the MODEL CONTEXT
only, applied once after a confirmed compression event, to prevent post-compression
context blowup. Deep-copy content-replace only (no message removal/reorder → no
orphaned tool_result); visible transcript (s.messages) untouched.
Deep-review re-gate: Codex SAFE (pairing-safety + context-only + idempotent verified
by reproduction), 65 own + 992 regression tests green. Crown-jewel streaming path.
Co-authored-by: t <a@b>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Steering a live gateway-owned run had no process-local agent in SESSION_AGENT_CACHE,
so it errored with "No agent cached" and dropped the typed text. Now detects the live
gateway ACTIVE_RUNS stream and queues the steer as the next turn (model/provider/profile
snapshotted first), updates the queue badge, clears the draft, shows a toast — without
cancelling the stream. Gateway sibling of the shipped #5597 fallback.
Codex SAFE + Fable SHIP-UX; byte-identical to PR head; 66 steer/busy-input tests pass;
Nathan visual-approved (desktop + mobile screenshots).
Co-authored-by: t <a@b>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
The per-token hot path mirrors partial text, reasoning, and live tool
calls into STREAM_PARTIAL_TEXT / STREAM_REASONING_TEXT /
STREAM_LIVE_TOOL_CALLS without holding STREAMS_LOCK, while snapshot
readers (_snapshot_and_append_partial_on_error, cancel_stream) read them
under STREAMS_LOCK. This is safe under the CPython GIL for two reasons —
NOT because `+=` is one bytecode (it is not: `d[k] += s` compiles to
load / add / STORE_SUBSCR):
1. Single writer — only the streaming thread ever writes a given
stream_id's buffers, so there is no writer/writer race to tear.
2. Reader/writer atomicity — `+=` builds a complete new immutable str,
then the final STORE_SUBSCR binding it into the dict is a single
atomic bytecode, so a concurrent reader's dict.get returns either
the old or new *complete* string (strings are immutable), never a
half-built one. list.append / single-key dict writes are atomic too.
So a reader sees a complete-but-possibly-stale value, never a torn one,
and the snapshot is a best-effort partial reconciled by later journal/SSE
events.
Chose documentation over locking: pulling the writes under STREAMS_LOCK
would add per-token contention against readers copying large buffers and
entangle the documented LOCK -> STREAMS_LOCK ordering (deadlock risk) for
only a recoverable staleness window. Make the assumption explicit at the
write sites and the reader instead. No behaviour change.
Rollback: revert this commit (comments only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The streaming worker called meter().begin_session(stream_id) but never a
paired end_session. A turn that produced zero output tokens (pre-flight
cancel, or a setup exception before the first token) left a _SessionMeter
in GlobalMeter._sessions forever: get_stats() only prunes sessions with
first_token_ts > 0, so a zero-token session is never reclaimed, and each
leaked entry inflates the `active` count sent over the SSE metering event.
begin_session() and the metering ticker's .start() are now registered as
the first statements inside the worker's outer `try`, and paired with an
idempotent meter().end_session(stream_id, 0) plus a deterministic
_metering_stop.set() in that try's outer `finally` (the same block that
pops STREAMS/CANCEL_FLAGS) — so every exit path tears the session down.
Deferring .start() until after put() is defined also closes a latent
start-before-put ordering window in the ticker closure.
The metering payload is unchanged; end_session only pops the session.
Verification: tests/test_metering_session_lifecycle.py (leak reproduction,
end_session reclaim, idempotency, begin->cancel-before-token->empty) plus a
plain-assert run on python3.11 (conftest gates pytest to 3.11-3.13; system
python is 3.14). py_compile clean.
WARNING streaming-contract: adjusts the streaming worker's begin/end
teardown ordering. Needs RFC/contract review against docs/rfcs/
(session-sse-contract-v1.md) before merge — draft PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strict providers (DeepSeek v4, newer OpenAI) reject an assistant message
carrying tool_calls: [] with HTTP 400. _sanitize_messages_for_api already
strips orphaned tool results, but not an assistant message that literally
stores an empty tool_calls list.
Drop the empty key on ALL THREE sanitized-dict sites that must agree:
_sanitize_messages_for_api, _api_safe_message_positions, and the
_safe_projection inside _restore_reasoning_metadata (so a tool_calls: [] row
projects identically on both sides of the metadata-carry-forward alignment and
doesn't lose its reasoning/id/timestamp every turn — Fable gate follow-up,
folded in).
Added mutation-checked regression tests: empty tool_calls dropped + populated
chain preserved (both sanitizer paths), and metadata carry-forward survives a
tool_calls: [] row.
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
Per Greptile's review on #5717: the previous
`if result.get('messages') is not None` guard created an asymmetric
condition where the silent-failure classifier could miss the injected
fallback when the agent result had `messages: None` (which never
actually happens in practice since `finalize_turn` always returns a
list, but the asymmetry read as code smell). Drop the guard; the
writeback is now unconditional on `isinstance(result, dict)`.
Per the maintainer's review on #5717: there may be a
`finalize_turn` invariant on the agent branch this WebUI ships
against that already appends the fallback as an assistant row, in
which case the live bug is gone and this PR becomes a regression
guard. Reframe the test docstrings as dual-purpose pins of the user-
visible closure-text contract so they read correctly under both
interpretations.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
The WebUI already surfaces the model-generated summary when the agent's
tool-iteration budget is exhausted. The remaining gap: when the agent
hits the limit but its summary call yields no usable assistant content
(common with reasoning-only responses), the user sees a bare
`tool_limit_reached` error instead of any closure text.
`hermes-agent.handle_max_iterations()` always returns a non-empty
`final_response` (either the summary or a graceful fallback), and
`AIAgent` exposes that string on the result dict. The WebUI, by
contrast, only reads `result['messages']`, so an empty summary
left no final assistant answer and routed to the error path.
Inject `result['final_response']` as a final assistant turn when
`max_iterations_reached` fires and no usable answer exists in
`messages`. The existing `_mark_latest_assistant_tool_limit_status`
flow then attaches the status card and the user sees the closure
text instead of the bare error. Hermes-agent parity preserved (it's
literally the same string); the bare-error path stays for the case
where the agent died entirely (no fallback either).
Tests cover: fallback-injection surfaces closure text + status card,
no double-injection when a real summary already exists, the existing
'died entirely' apperror path is unchanged, and three unit cases for
the helper's skip conditions.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
The streaming settlement path could show a misleading "No response from provider"
error card AFTER a completed answer, because a stale terminal/partial result flag
still forced the generic no_response error path even when the merged current turn
already had a final assistant answer. Now the generic no_response terminal error
is suppressed ONLY when the already-merged current turn has a completed assistant
reply (scoped to soft `partial` results per the issue-thread recommendation),
letting normal `done` settlement continue. Real silent failures, partial
failures, auth/quota errors, cancellation, replayed rows, and terminal
tool/compression cases all keep their existing error behavior.
Self-built stage of contributor PR; gate-pass. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
'All 0 credential(s) exhausted for <provider>' did not match _is_quota_error_text
so it fell through to a generic 'Error' with no hint (the exception path's else
branch discarded the hint entirely). Adds a distinct credential_pool_empty
classification (ordered before the quota check) + an explicit exception-path
branch + an actionable pool-specific hint, so the user sees why the turn failed
and what to do. Does not over-match (doesn't swallow real quota errors) and
doesn't block partial-harvest; downstream tolerates the new type.
NOTE: this is the CLASSIFICATION half of #3929 only — the data-loss half
(reasoning/partial-work restore on reload-reconcile) is a scoped follow-up that
needs the reporter's server logs.
Self-built (nesquena-hermes); nesquena independent review APPROVED.
