Render structured background-process completions and watch matches as accessible collapsed cards while preserving raw fallback, verbatim output, and full watch patterns.
Co-authored-by: b3nw <b3nw@duck.com>
* fix(#6389): remove clear_offline_buffer() teardown blocks that cause data loss on late reconnects
Keep the real leak fixes:
- unregister_stream_owner() in both streaming.py and gateway_chat.py
- AGENT_INSTANCES.pop() in gateway_chat.py
The clear_offline_buffer() calls could clear a channel's offline buffer
after a late reconnect already fetched the channel but before it subscribed,
causing the reconnecting client to lose all buffered events (including
stream_end) and hang indefinitely on heartbeats.
STREAMS.pop() already releases registry ownership; GC reclaims the buffer
once the last outstanding channel reference is dropped, correctly preserving
replay for in-flight reconnects.
* docs(changelog): stamp #6389 stream-teardown leak fix
---------
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: nesquena-hermes <agent@webui>
Codex gate on #6312 found the extracted helper caught only (TypeError, ValueError)
while the inline code it replaced caught Exception; float(10**400) raises
OverflowError, escaping the helper and aborting terminal-error cleanup before the
error message is persisted / apperror is emitted. Widen _terminal_turn_duration's
float() guard to (TypeError, ValueError, OverflowError) — the exact exceptions
float() can raise. Adds 10**400 to the existing invalid-origin parametrize
(red-before-green verified).
Addresses the three gate-review blockers on #6113:
- Fallback attribution: read the served model from agent.model AFTER the
run (the agent mutates it when a fallback fires) instead of the pre-run
resolved_model, for both the persisted _usedModel and SSE used_model.
- Gateway double-label: suppress the additive chip whenever
_gatewayRouting is present, and pass msg._usedModel to
_formatGatewayModelLabel as its model fallback when routing omits
used_model.
- Transparent double-label: skip the generic .msg-used-model-inline chip
when the transparent turn footer owns the model label (.lf-model).
Also per Greptile: settled footer now orders the model chip after the
duration (elapsed - model - ...) to match the transparent footer, and the
test file gains its trailing newline.
Record resolved_model on assistant messages as _usedModel, expose used_model
in the usage SSE payload, and render model chips in transparent and settled
turn footers while suppressing duplicates when gateway routing already shows
used_model.
Route async-delegation completions by the immutable origin_ui_session_id (exact origin tab) and deliver them through a durable claim/complete/release lifecycle so they arrive exactly once, restart-safe, on both the background wakeup and next-turn drain paths. Combines #6185 (@carlotestor) + #6159 (@sysophelper-droid); supersedes #6002/#6225.
Co-authored-by: carlotestor <carlotestor@users.noreply.github.com>
Co-authored-by: sysophelper-droid <sysophelper-droid@users.noreply.github.com>
Codex gate on #6143 found a SILENT regression the reuse-path fix introduced:
_replace_session_db_in_kwargs early-returned the OLD handle when the rebuild
failed, but master assigned None. If the old handle was already CLOSED, both
credential self-heal callers (streaming.py:9032/10253) then built a fresh agent
around a closed SessionDB -> every persist/search fails with 'NoneType has no
attribute execute' while the chat keeps going. Guard on _session_db_is_open:
keep a live handle (subagents hold it by ref), else degrade to None (clean lazy
reinit, master behaviour). +2 regression tests.
Co-authored-by: carlotestor <carlotestor@users.noreply.github.com>
- Log failed closes in _adopt_session_db_for_cached_agent with
logger.debug(..., exc_info=True) instead of bare pass, matching
_replace_session_db_in_kwargs so a failed close (EMFILE risk) is
diagnosable.
- Drop the fragile exec/source-slice fallback in the test helper import;
real import or pytest.skip. Source-level pins still cover reverts
without importing streaming.
Background subagents (delegate_tool) capture parent_agent._session_db by
reference. The cached-agent reuse path created a fresh SessionDB every
stream request and unconditionally closed the previous handle (PR #1421
EMFILE fix). A server-side wakeup / new turn for the same parent session
could therefore close the shared handle while children were still
flushing messages, producing:
Session DB append_message failed: 'NoneType' object has no attribute 'execute'
Reproduce (observed in production):
1. Parent WebUI session runs delegate_task with N background subagents
2. Parent turn tears down / idle-hook redelivers a deferred wakeup
3. New stream for the parent adopts a fresh SessionDB and closes the old one
4. Live subagents' append_message hits conn=None
Fix: _adopt_session_db_for_cached_agent reuses a still-open handle (and
closes the unused new one — still EMFILE-safe) and only replaces when
the existing handle is missing or already closed. Same guard applied to
_replace_session_db_in_kwargs (credential self-heal). LRU eviction still
closes at true session boundaries.
Tests updated to pin the adopt helper and the subagent-safe semantics.
Codex found a verified lifecycle hole: the bare boolean model_explicitly_picked
was not cleared on /api/session/update (api/routes.py:14438-14460 changes+persists
the model/provider without touching it), so after an update+reload the flag stayed
True and a cold #433 stale leftover (openai/gpt-5.4) was wrongly preserved instead
of stripped to gpt-5.4.
Fix (Codex's prescription): replace the unscoped boolean with a SIGNATURE bound to
the routing context.
- api/models.py: new model_explicit_pick_signature(model, provider) ->
'<model>\x1f<provider>'; Session.model_explicit_pick_signature persisted field
(replaces the boolean; in METADATA_FIELDS, restored via kwargs, survives restart).
- api/routes.py /api/chat/start: on a fresh explicit_model_pick, stamp the signature
of the resolved model+provider.
- api/streaming.py: treat the send as a deliberate pick ONLY when the persisted
signature equals the signature recomputed from the session's CURRENT persisted
model+provider. Any model/provider change (chat-start, session-update,
normalization, provider repair) yields a different signature and auto-invalidates
the stale pick — no per-mutation-site clearing needed.
So: deliberate pick on unchanged context -> preserve (b3nw); any switch away ->
signature mismatch -> unmarked -> legacy strip (#433 stays correct even after a
session-update+cold-restart). Adds signature match/invalidate/persist tests.
75 focused tests pass (resolver + sprint40 + the 2 streaming-mock suites).
Nathan's call: adopt Codex's complete mechanism instead of the unconditional
cold-preserve. The cold-catalog custom-proxy decision is now gated on whether the
user DELIBERATELY selected the model this session, closing the residual where a
stale #433 leftover would preserve+400 on a cold+no-disk send.
Mechanism:
- resolve_model_provider(model_id, *, explicitly_picked=False): in the custom-arm
cold branch (no warm provenance, not config-declared), explicitly_picked=True
preserves the vendor namespace verbatim (deliberate pick, proxy routes on it);
otherwise the legacy redundant-prefix strip runs (stale cross-provider leftover,
#433). Warm endpoint-advertised provenance still wins over the flag.
- Session.model_explicitly_picked: new persisted field (default False, restored
from metadata via **kwargs, listed in METADATA_FIELDS so it survives a restart
— b3nw's exact cold scenario).
- /api/chat/start stamps it: True on a fresh explicit_model_pick; cleared only
when the resolved model actually CHANGED without a pick (real switch away);
preserved across plain follow-up sends of the same model (the onchange marker
is one-shot, so per-send explicit_model_pick alone would drop the intent).
- streaming.py reads s.model_explicitly_picked and passes explicitly_picked= into
resolve_model_provider (inside the profile scope).
Independent end-to-end matrix (all pass): ben default cold, ben picked non-default
cold+warm, ben stale cold (strips), #433 warm+cold, #3872 bedrock, #548 zai-org.
67 resolver+sprint40 tests pass incl. new explicit-pick + stale-companion +
persistence round-trip.
Codex gate found two real issues in the send-path warm:
1. Blocking lock wait (CORE): get_available_models(prefer_cache=True) still
acquires _available_models_cache_lock and can wait up to ~60s (unbounded in
synchronous rebuild mode) on an in-flight rebuild — unacceptable on the send
hot path. Rewrote warm_models_catalog_provenance_if_cold as a genuinely
non-blocking, disk-only publish: try the cache lock NON-BLOCKING (return
immediately if busy — a concurrent build will publish provenance itself),
read ONLY the on-disk cache via _load_models_cache_from_disk (no network, no
rebuild), publish snapshot+fingerprint + _sync_models_cache_provenance().
Verified: returns in ~3.5ms while the lock is held (was a 60s block).
2. Profile scope (CORE): the streaming worker is a separate thread that does NOT
inherit the HTTP handler's request-profile TLS. Both the warm and the resolve
read profile-keyed config (cache path + source fingerprint via
get_active_profile_name), so a cold send from a NAMED profile could resolve
against the DEFAULT profile's config and route to the wrong provider/base_url
(get_available_models reloads config on path mismatch). Wrapped warm + resolve
in profile_scope_for_detached_worker(_resolved_profile_name) so both see the
owning session's profile (no-op for default/root).
Tests: rewrote never-live-rebuilds (now asserts warm NEVER calls
get_available_models — reads disk directly) and added a non-blocking regression
(warm returns <2s while the cache lock is held). 65 resolver+sprint40 tests pass;
b3nw end-to-end + non-blocking + disk-warm all verified.
The earlier #5979 fix (PR #5980, exp-v0.52.53) resolved the WARM-catalog and
config-declared cases, but b3nw confirmed truncation persists on exp-v0.52.55/56.
Root cause (reproduced on master, confirmed by Codex + Fable):
b3nw's config is provider: custom:llm-proxy, model.default: x-ai/grok-4.5, no
models[] allowlist. His DEFAULT (x-ai/grok-4.5) preserves via config-declaration,
but a model he SELECTS in the picker that isn't the default (x-ai/grok-composer-2.5-fast)
truncates on every COLD send. The send path (api/streaming.py:7857) calls
resolve_model_provider without warming the catalog, and the #1855 chat/start fast
path deliberately skips the catalog when a session already has a persisted
model+provider -- so 'cold at send' is the designed state after any restart /
settings-save / cache-expiry, not a rare race. With provenance None and no config
declaration, the #5980 tri-state fell through to the LEGACY family strip
(if prefix in _PROVIDER_MODELS and _is_first_party_model: return bare), and
grok-composer-2.5-fast had graduated into the x-ai first-party catalog -> stripped.
Fix (two changes, both advisor-verified):
1. Flip the custom-arm cold default from family-strip to PRESERVE-verbatim
(api/config.py). For an explicit custom / custom:<slug> proxy the strip now
requires POSITIVE provenance (endpoint advertises only the bare id, #433). A
wrong strip destroys routing info the user can't recover; a wrong preserve
fails loudly and self-heals on warm. This also honours _endpoint_advertised_model_ids's
own documented 'None -> preserve verbatim' contract and removes the data-driven
flaw where a model graduating into the static catalog silently flips routing.
2. Network-free provenance warm on the send path
(warm_models_catalog_provenance_if_cold, called in streaming.py before the
resolve). prefer_cache=True republishes provenance from the durable disk cache
(no live probe, no _cfg_lock held on the worker thread -> no lock-order
deadlock), so #433's bare-only-strip stays exact whenever a disk cache exists.
Residual (accepted, strictly better than master): a #433-shaped wrong-preserve
now requires memory cold AND disk cache absent/invalid (first run, post-config-save
window, schema/version bump). It fails loudly and heals on the next /api/models or
session-visit rebuild. Under master b3nw's failure was the mirror image but silent,
recurring every send, and unfixable short of declaring every model in config.
Supersedes rodboev's #5996, which targeted the providers.<slug> arm and does not
cover b3nw's actual custom:llm-proxy config (confirmed by Codex).
Tests: rewrite the cold-fallback test (strip -> preserve), sharpen the
foreign-profile test so trusted-strip vs ignored-preserve genuinely diverge, add
b3nw's exact cold non-declared custom:slug case, and add three warm-helper tests
(publishes-from-disk, noop-when-warm, never-live-rebuilds). 64 resolver+sprint40
tests pass; b3nw end-to-end verified; #433 positive-provenance strip verified.
Thanks @b3nw for the CLI-vs-WebUI comparison that localized it.
* fix(#4685): meter post-compression model context
* Release exp-v0.52.48: meter post-compression model context (#5872, #4685)
---------
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
* fix(#5940): surface non-retryable provider error instead of no_response
The Agent emits a non-retryable terminal error (e.g. HTTP 400 invalid model / no
credentials) via its lifecycle status_callback, but the WebUI dropped it and the
run result/_last_error were empty for that abort path -> turn completion showed the
misleading no_response 'silent rate limit, try again' message. Capture the emitted
terminal error in _agent_status_callback and seed _last_err with it at completion so
_classify_provider_error reports the real cause (model_not_found / auth) with
actionable guidance. +regression test.
* Release exp-v0.52.47: surface non-retryable provider error instead of no_response (#5940)
---------
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Codex re-gate caught 2 defects in the prior UX commit:
1. streaming.py:8921 — the disclosure fired for ANY process_wakeup failure (e.g.
rate-limit) even when no pause was recorded -> falsely claimed 'retries paused'
while process_wakeup_pause stayed empty. Now gated on the record_* return value.
2. streaming.py:10130 — the raised-exception credential_pool_empty path (exercised
by the existing agent test) recorded the pause but omitted the disclosure. Added
the same conditional disclosure there.
Both paths now keep _error_payload['hint'] in sync with the persisted bubble, and
the disclosure appears only when a pause was actually recorded. 69 tests green.
Fable UX-gate follow-up on #5732: after the first credential-exhausted error,
automatic wakeups are silently suppressed. Disclose it in the wakeup-source
error card ('Automatic retries for this conversation are paused until you send
a message, switch the model/provider, or fix the credentials.') so the silence
reads as intentional, not a stuck agent — Gmail/Dropbox account-needs-attention
convention. Scoped to the process_wakeup + credential_pool_empty seam only.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
