Adversarial review of the previous commit found a leak it introduced.
`attach_terminal()` subscribes and only then are the response headers written.
A client that dropped in that instant makes `end_headers()` raise
BrokenPipeError/ConnectionResetError, and because the `try/finally` started
*after* those writes, the exception escaped `_handle_terminal_output` with the
queue still in `_subscribers`. That pins `unwatched_since` at None — only
`unsubscribe()` on an empty subscriber list resets it — so the terminal became
permanently unreapable: the exact fd/thread leak (#4633) this reaper exists to
prevent, and strictly worse than the TOCTOU the previous commit fixed, since
that at least still allowed cleanup.
The try now starts immediately after the attach, so every exit path
unsubscribes. Regression test drives a handler whose `end_headers()` raises and
asserts the subscriber list is empty, `unwatched_since` is restored, and the
terminal is reapable again once the grace elapses. It fails against the
previous ordering.
`_terminals_to_reap()` snapshots victims under `_LOCK` and then releases it,
but `subscribe()` clears `unwatched_since` and appends to `_subscribers` under
the terminal's own `_sub_lock`, which the selection pass never held. A viewer
reconnecting in that gap left object identity — all `close_terminal(expected=…)`
ever checked — still true, so the reaper killed a terminal that now had a live
viewer.
- `_claim_reap_victim()` re-establishes the *whole* selection predicate while
holding `_LOCK` and the terminal's `_sub_lock`, and removes the registry
entry in that same critical section: same object, no subscribers, and
`unwatched_since` still set and still expired. A dead process is still
reaped unconditionally — it cannot come back, and a viewer watching a corpse
must not pin the entry forever.
- `attach_terminal()` does the registry lookup and the subscribe atomically
under `_LOCK`, replacing `get_terminal()` + `term.subscribe()` in the SSE
route. It takes the same two locks in the same order as the claim, so
exactly one side wins: the viewer attaches (and the reap is refused), or the
entry is already gone (and the route answers "terminal not running" before
committing a 200, instead of streaming from a corpse). Registration is the
authority — a registered-but-`closed` terminal still attaches so the viewer
receives `terminal_closed`.
- Teardown moved to `_teardown_terminal()` and runs after both locks are
released; `killpg`+`wait` can block ~2.5s and must not pin the registry.
- Dropped the duplicated `self.last_activity = time.time()` the rebase left in
`put_output()`.
Tests: a barrier regression that pauses the reaper after selection, attaches a
viewer through the production path, resumes, and asserts the terminal stays
registered/live and still receives output; the inverse where the reap claims
first and the attach reports not-running; refreshed-grace and replaced-entry
claims; dead-with-viewer; and a probe that `_LOCK` is free during teardown.
Three of them fail against an identity-only claim.
The fd-leak fix retires a terminal whose shell EXITS, and _MAX_TERMINALS bounds
the worst case, but an interactive shell abandoned by its client (tab closed
without POSTing /api/terminal/close, browser crash, network drop) keeps running
forever — there is deliberately no PDEATHSIG. A background reaper now closes any
terminal that has had zero attached output streams for longer than a 15-minute
idle grace, plus any dead-process terminal as a belt-and-suspenders for the
reader-loop retire.
Attachment is tracked on the subscriber model from the output-broadcast change:
subscribe() clears unwatched_since, the last unsubscribe() sets it. A terminal
is born unwatched, so a spawn nobody ever attaches to is reaped too; the grace
spans a tab refresh / brief drop so a real reconnect keeps the session. Eviction
uses close_terminal(sid, expected=term) so a restart replacement is never torn
down. The reaper thread is a lazily-started daemon singleton like the spawn
supervisor.
Stacked on the terminal fd-leak + output-broadcast changes (same file).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two embedded-terminal resource leaks (#4633):
1. fd + dict entry on shell exit. When a shell exited on its own (user typed
`exit`, the process died), _reader_loop broke out of its loop but only set
`closed` and emitted terminal_closed — it never closed the pty master_fd nor
removed the _TERMINALS entry. Only an explicit close_terminal / restart /
atexit did that, so every self-exited shell leaked its master fd and dict
entry for the rest of the WebUI's uptime. The reader loop now retires the
session (close_terminal with an `expected=` identity guard so a restart's
replacement is never torn down).
2. Unbounded accumulation of abandoned terminals. A client that drops its output
stream without POSTing /api/terminal/close (tab close, crash, network drop)
leaves the shell running (no PDEATHSIG, by design). _MAX_TERMINALS (32) now
caps the live population, evicting a dead-proc terminal first, else the
least-recently-active one, to make room.
Also hardens an fd-reuse TOCTOU that #1 would otherwise make reachable on the
common `exit` path: write_terminal / resize now re-check `closed` and touch
master_fd under a per-terminal io_lock that close_terminal also holds around
os.close, so input can never land on a closed-and-recycled fd number.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Render structured background-process completions and watch matches as accessible collapsed cards while preserving raw fallback, verbatim output, and full watch patterns.
Co-authored-by: b3nw <b3nw@duck.com>
Embed local MEDIA: image refs as inline base64 at snapshot-build time,
gated by a workspace/attachment-root allow-list, 512KiB cap, image-MIME
allow-list, and magic-byte content validation. Everything else (traversal,
symlink-escape, file://, non-image, SVG, oversize) becomes a static
placeholder. Closes P0 #6174 (public shares no longer emit authenticated
/api/media requests for local paths).
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
* fix: resolve provider alias before matching _PROVIDER_MODELS (#6148)
* fix(backend): resolve model prefix through _PROVIDER_ALIASES before matching
- Fix IndentationError: dedent _canon_prefix assignment to 8 spaces
- Rebased onto upstream/master
- Add regression test: alias prefix (z-ai) resolves to canonical key (zai)
and routes cross-provider through OpenRouter (#6131)
* fix(#6131): canonicalize both sides via _canonicalise_provider_id (idempotent) + regression tests
Codex found _resolve_provider_alias breaks Qwen (qwen->alibaba, not in _PROVIDER_MODELS)
and cross-routes alias-of-active-provider (raw vs canon asymmetry). Use the idempotent
_canonicalise_provider_id for both membership and equality. Adds 2 regression tests.
* docs(changelog): stamp #6148 alias-prefix model routing fix
---------
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: nesquena-hermes <agent@webui>
* fix(#6389): remove clear_offline_buffer() teardown blocks that cause data loss on late reconnects
Keep the real leak fixes:
- unregister_stream_owner() in both streaming.py and gateway_chat.py
- AGENT_INSTANCES.pop() in gateway_chat.py
The clear_offline_buffer() calls could clear a channel's offline buffer
after a late reconnect already fetched the channel but before it subscribed,
causing the reconnecting client to lose all buffered events (including
stream_end) and hang indefinitely on heartbeats.
STREAMS.pop() already releases registry ownership; GC reclaims the buffer
once the last outstanding channel reference is dropped, correctly preserving
replay for in-flight reconnects.
* docs(changelog): stamp #6389 stream-teardown leak fix
---------
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: nesquena-hermes <agent@webui>
- Add only .ts→text/plain and .tsx→text/plain to MIME_MAP (exclude .js,
.jsx, .css and other unsafe mappings that would make workspace files
executable as same-origin scripts)
- Add test_ts_artifact_served_as_text_plain_with_attachment — asserts
Content-Type: text/plain, Content-Disposition: attachment, and
X-Content-Type-Options: nosniff for .ts files via /api/media
- Add test_tsx_artifact_served_as_text_plain_with_attachment — same for .tsx
- Add test_file_raw_js_not_served_as_inline_executable — proves that a
workspace .js file falls through to application/octet-stream instead of
text/javascript via /api/file/raw
Closes#6372
Codex gate on #6312 found the extracted helper caught only (TypeError, ValueError)
while the inline code it replaced caught Exception; float(10**400) raises
OverflowError, escaping the helper and aborting terminal-error cleanup before the
error message is persisted / apperror is emitted. Widen _terminal_turn_duration's
float() guard to (TypeError, ValueError, OverflowError) — the exact exceptions
float() can raise. Adds 10**400 to the existing invalid-origin parametrize
(red-before-green verified).
Addresses the three gate-review blockers on #6113:
- Fallback attribution: read the served model from agent.model AFTER the
run (the agent mutates it when a fallback fires) instead of the pre-run
resolved_model, for both the persisted _usedModel and SSE used_model.
- Gateway double-label: suppress the additive chip whenever
_gatewayRouting is present, and pass msg._usedModel to
_formatGatewayModelLabel as its model fallback when routing omits
used_model.
- Transparent double-label: skip the generic .msg-used-model-inline chip
when the transparent turn footer owns the model label (.lf-model).
Also per Greptile: settled footer now orders the model chip after the
duration (elapsed - model - ...) to match the transparent footer, and the
test file gains its trailing newline.
Record resolved_model on assistant messages as _usedModel, expose used_model
in the usage SSE payload, and render model chips in transparent and settled
turn footers while suppressing duplicates when gateway routing already shows
used_model.
* feat(extensions): token-v1 proxy->sidecar auth boundary
Mint a per-extension secret core injects (X-Hermes-Sidecar-Token) on every
proxied request; sidecars validate it. Closes the hole where a loopback sidecar
port is reachable by any local process and cannot distinguish a proxied request
from a direct one.
- api/extension_sidecar_auth.py: per-extension token lifecycle (atomic mint,
re-read-verified so an unpersisted token is never injected, per-request
mtime-cached read for live rotation, path-escape-safe)
- manifest proxy_auth negotiation: absent=legacy, token-v1=enforce, unknown=fail-closed
- proxy: inject token, strip inbound + response x-hermes-*, auth-off posture
(loopback-only local_unprotected, else 503), fail-closed when token unavailable
- consent-time auth_required in status payload; mint-on-consent
- docs/EXTENSIONS.md proxy_auth section
- 7 new tests (26/26 green)
* fix(extensions): address Codex+Fable gate on token-v1 (6 findings)
- align token-module extension-id grammar with core _EXTENSION_ID_RE (was
narrower -> legally-named ext consented then 503'd forever)
- resolve token dir dynamically (mirror _extension_state_dir) -> real test
isolation, no import-time STATE_DIR cache
- cross-process mint: O_CREAT|O_EXCL no-clobber claim (was os.replace clobber)
- rotation cache keyed on full fingerprint (ino/dev/mtime/ctime/size) +
re-fingerprint after read -> same-size/mtime replacement no longer stale
- validate token format on read (url-safe, 16-256) -> malformed file can't
leak via a ValueError echoed in a 502
- consent fails 503 when token can't be provisioned (was silent-swallow ->
persisted consent then 503 forever)
- rename status auth_required -> posture enum (protected|local_unprotected):
nothing is blocked for loopback, so 'required' was misleading
- panels.js: render local_unprotected warning on the consent row (+ CSS)
- docs: token-path resolution order, 401-vs-503, explicit 'legacy' acceptance
- tests: route-level token-injection+response-strip test; fix illusory isolation
in token-module test; 27/27 green
* fix(extensions): close 2 token-mint races (Codex re-gate round 2)
- mint via temp-file + atomic os.replace (not O_CREAT|O_EXCL) so the final
path is never observed empty/half-written — a concurrent loser can no longer
read an empty token file and 503
- single _stable_read helper (fingerprint-read-refingerprint, bounded retry on
mid-read change) used by BOTH ensure_token and current_token — a token that
changes during the read is never returned or cached stale
- stress-verified: 20 concurrent first-mints converge on 1 persisted token; 27/27
* fix(extensions): atomic no-clobber token publish via os.link (Codex re-gate round 3)
os.replace fixed empty-file exposure but still clobbered cross-process: two
processes could both write+replace and a reader between them got a token no
longer on disk -> 401. Switch to the repo's TOCTOU-safe os.link create-or-fail
idiom (session_recovery.py:627): write temp -> link into place (fails if a
winner already published) -> loser drops its temp and reads the winner via
_stable_read. Proven: 16 concurrent PROCESSES converge on 1 persisted token,
all matching disk. 27/27 tests green.
* fix(extensions): resolve Frank+Greptile #6331 review — token-v1 fail-closed when auth off (consent+resolution), token-v1-only proxy_auth/posture status fields, fullmatch ext-id validator
* fix(extensions): finish Frank and Greptile sidecar review
---------
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
* fix: preserve live anchor run identity
* Validate envelope run ids before snapshot cursor use
* docs(changelog): stamp #6201 stable Anchor run identity
---------
Co-authored-by: Frank Song <franksong2702@gmail.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
* fix(webui): keep the content search from evicting the user's working set
/api/sessions/search?content=1 walks EVERY session and pulls each one through
get_session(), which inserts it into the SESSIONS LRU and marks it
recently-used. On any install with more sessions than sessions_cache_max
(default 300), a single search therefore flushes the whole cache and refills it
with sessions the user is not looking at — the classic buffer-pool
scan-pollution problem. The sessions actually open in the UI are exactly the
ones evicted, and the search is keystroke-debounced, so it repeats while typing.
A scan reads each session exactly once, so nothing it touches has earned
"recently used". get_session_for_scan() reuses a resident session without
promoting it, and reads a cold one straight from disk without caching it. It
returns None rather than raising, since a scan skips what it cannot open.
This is a correctness fix for cache behaviour, not a latency fix. The
multi-second searches that led here were contention, not scan cost: a trivial
/api/profiles took 9.2s in the same window, and a full read+parse of ~1700 real
sessions measures ~4s total.
test_sessions_search_depth_validation patched api.routes.get_session. With the
search reading through the scan accessor it now patches get_session_for_scan —
left unfixed, two of its cases fail and the third passes vacuously against an
empty result set.
Validation:
pytest tests/test_issue4765_sessions_lru_eviction.py
tests/test_sessions_search_depth_validation.py -> 13 passed
Both added eviction tests fail on the pre-fix accessor (verified by revert):
the working set drops from 4/4 to 0/4 resident after one scan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(changelog): stamp #6084 content-search working-set preservation
---------
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
* feat(terminal): broadcast output to every viewer instead of one shared queue
TerminalSession.output was a single queue.Queue read destructively by the SSE
handler. Two tabs/windows viewing the SAME session each open their own
EventSource, so two _handle_terminal_output handlers competed on that one queue:
every PTY chunk was delivered to exactly one of them. Each tab saw a disjoint
half of the byte stream, and only one ever received terminal_closed.
Output now fans out, mirroring StreamChannel/SessionChannel: each SSE consumer
subscribe()s its own queue (seeded with a bounded backlog so a first/late attach
still replays the recent scrollback, preserving the old buffer-until-first-
consumer behaviour), and put_output broadcasts to all subscribers. A slow
viewer's queue drops its own oldest chunk (drop-oldest, isolated per subscriber)
so one lagging tab can't starve another. The handler unsubscribes in a finally
so the subscriber list can't grow.
Stacked on the terminal fd-leak fix (same file).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: cover terminal broadcast lifecycle
* fix: serialize terminal subscriber fanout
* test: cover terminal unsubscribe publication race
* restore timing-flaky pytest.skip on test_terminal_survives_short_lived_request_thread (keep API-updated body); changelog #5836
---------
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
* fix: catch unhandled exception in POST /api/updates/check (defensive hardening) (#6180)
* fix: generate non-empty approval_id when gateway approval.request omits it (#6008) (#6168)
* chore: mark update-check try/except as defensive-only guard, drop #6086 linkage
Per maintainer review, the try/except wrapper is defense-in-depth only —
it does NOT fix#6086 (root cause is signal/process-group reaping).
Updated log message and added inline comment to make this explicit.
Leave #6086 open.
* test: isolate state-dir probes from user state
* docs(changelog): stamp #6168 approval_id, #6180 update-check guard, #6305 test isolation
---------
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
Co-authored-by: pxxD1998 <214340659+pxxD1998@users.noreply.github.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
