Tests should not reach the public internet. Before this commit, an
accidentally-leaking outbound socket from the test_server fixture (real
TLS handshakes to Anthropic / Amazon / OpenRouter, sometimes triggered
by SDK-init paths that found a credential the credential-strip allowlist
missed) was adding 60+s of wall-time to a 100s test run and creating a
class of flaky failures.
This installs a default-deny socket-block at two layers:
1. Pytest process, via tests/conftest.py module-level monkey-patch on
socket.create_connection + socket.socket.connect. Loopback / RFC1918
private / link-local / RFC2606 reserved-TLD destinations pass through;
anything else raises OSError("hermes test network isolation: outbound
to ... blocked"). Tests that legitimately need real outbound opt back
in via the new `allow_outbound_network` fixture (no current callers).
2. Test_server subprocess (server.py), via a HERMES_WEBUI_TEST_NETWORK_BLOCK=1
environment-variable-gated guard at the top of server.py. tests/conftest.py
sets the env var on every test_server spawn. Without this, the subprocess
could make outbound that the pytest-side block can't see (which is exactly
what was happening — verified via `ss -tnp` showing the server.py child
with established ESTAB sockets to [2607:6bc0::10]:443).
In production the env var is unset, so the guard is a no-op.
Companion changes:
- test_dns_resolution_failure refactored to mock socket.getaddrinfo
raising gaierror, instead of relying on a real DNS lookup of a
*.invalid hostname. The test was the one outlier that genuinely
exercised real DNS; mocking matches what every other probe-error test
in the same file already does.
- New tests/test_conftest_network_isolation.py with 9 adversarial
tests proving the block fires for public IPs (including the exact
Anthropic IPv6 and Amazon IPv4 destinations we observed leaking),
the allow-list passes loopback / RFC1918 / link-local / reserved-TLDs,
and the opt-in fixture re-enables real outbound when needed.
Test suite: 5,120 → 5,192 (+72 net new from this commit + the regression
tests in the companion commits). Wall time: 161s → 95s on the same
hardware. No remaining outbound from any test path.
SSRF defense-in-depth: `urllib.request.urlopen` follows redirects by default,
so a probe at `http://example.com/v1/models` could be redirected to
`http://internal-service:8080/admin` — surfacing internal HTTP services to
the authenticated user. The probe is already gated behind WebUI auth and the
local-network check, so the practical attack surface is 'authenticated user
enumerating internal services' (same as `curl` from their browser DevTools).
Tightening the redirect default is cheap insurance.
Implementation:
- New module-level `_NoRedirectHandler` (subclasses `urllib.request.HTTPRedirectHandler`,
overrides `redirect_request` to return None — urllib then raises `HTTPError(3xx)`
rather than following).
- New module-level `_PROBE_OPENER = urllib.request.build_opener(_NoRedirectHandler())`.
- `probe_provider_endpoint` switches from `urlopen(req, …)` to `_PROBE_OPENER.open(req, …)`.
- The existing `HTTPError` handler now categorizes 3xx as `unreachable` with a
detail string mentioning 'redirect' so the user understands what happened.
3xx does NOT get its own error code in `PROBE_ERROR_CODES` — the error
taxonomy contract stays the same shape (frontend i18n unchanged).
Added regression test `test_probe_does_not_follow_redirects` in
`tests/test_issue1499_onboarding_probe.py`. Spins up a tiny HTTP server that
302-redirects `/v1/models` to `/different-endpoint` (which would return
`{'data': [{'id': 'should-not-see'}]}` if followed). Asserts the probe
returns `{ok: False, error: 'unreachable', status: 302, detail: …'redirect'…}`
and that the 'should-not-see' string never appears in the result.
Mutation-verified: reverting `_PROBE_OPENER.open` back to `urlopen` causes
the test to fail with "Probe followed a redirect — should have refused".
Suite delta: 3917 → 3918 passing (+1).
Reviewer-flagged in PR #1501. Per the
'reviewer-flagged-fix-in-release-not-followup' policy: <20 LOC defensive
fix, regression test path obvious, ship in this release rather than punting.
Addresses both #1499 (onboarding wizard never probes the configured base URL)
and #1500 (cross-tool env-var name divergence between webui and agent CLI).
Surfaced together because they're both LM-Studio onboarding bugs that pile
on top of each other — fixing only one leaves the broken UX.
#1499 — Onboarding wizard probes <base_url>/models before persisting
Pre-fix, `apply_onboarding_setup` accepted whatever `base_url` the user typed
without ever fetching `<base_url>/models`. @chwps's log timeline in #1420
showed the wizard finishing in 239ms with zero outbound HTTP — onboarding
silently persisted unreachable URLs and left users with empty model
dropdowns they had to populate by hand-editing config.yaml.
Backend:
* New `probe_provider_endpoint(provider, base_url, api_key, timeout=5.0)`
in `api/onboarding.py`. Stdlib-only (urllib + socket — no httpx dep).
Returns `{ok, models}` on success; `{ok: False, error: <code>, detail}`
on failure with stable error codes the frontend can switch on:
invalid_url, dns, connect_refused, timeout, http_4xx, http_5xx, parse,
unreachable. 256 KB response cap and 5s timeout keep a hostile or mis-
pointed endpoint from blocking the wizard.
* New `POST /api/onboarding/probe` route — thin JSON wrapper around the
function above. Same local-network gate as `/api/onboarding/setup`
because the body carries an `api_key` the user typed.
* The probe response is NEVER persisted. Only the user's typed selection
ends up in config.yaml; the probed model list just populates the
wizard's dropdown.
* SSRF: deliberately does NOT block private-IP ranges. The wizard is
gated behind WebUI auth and the legitimate target IS a local LM Studio
/ Ollama / vLLM server. A "block private IPs" SSRF defense would make
the feature useless for its primary use case.
Frontend:
* `static/onboarding.js`:
- New `ONBOARDING.probe` state ({status, error, detail, models, probedKey}).
- `_runOnboardingProbe()` — POSTs to /api/onboarding/probe, idempotent
& cached on (provider, baseUrl, apiKey).
- Debounced (400ms) on `oninput` of the base URL field.
- Explicit "Test connection" button.
- `nextOnboardingStep` blocks Continue at the setup step for any
provider with `requires_base_url=True` until the probe succeeds.
Same localized error renders inline.
* `static/i18n.js`: 13 new keys × 9 locales (canonical English in `en`,
English fallback with `// TODO: translate` markers in the other 8 —
same convention as v0.50.271 #1488 voice-buttons).
* `static/style.css`: probe banner + Test button styling (red-tinted
error variant, green-tinted success variant, neutral probing state).
Verified via manual repro on port 8789:
* connect_refused → red banner, helpful "from Docker, try the host IP"
hint, blocks Continue.
* DNS failure → red banner, "could not resolve host '...'", blocks Continue.
* Success against a mock /v1/models server → green banner, model dropdown
populates from the probed list, Continue advances normally.
#1500 — webui env var aligned with agent CLI (LM_API_KEY)
The webui has long used `LMSTUDIO_API_KEY` for LM Studio's API key in
both onboarding and Settings detection. The agent CLI runtime
(hermes_cli/auth.py:177-183) reads `LM_API_KEY`. So a user who configured
auth on their LM Studio instance got Settings → Providers reporting
has_key=True (because webui saw its own LMSTUDIO_API_KEY) but the agent
runtime ignored the key and fell back to LMSTUDIO_NOAUTH_PLACEHOLDER →
401 against the auth-enabled LM Studio server. Masked in practice for
the no-auth majority.
Picked Option B from the issue (defer to the agent — single source of
truth) but mitigated the migration cliff by reading the legacy name as
a fallback:
* `api/onboarding.py:_SUPPORTED_PROVIDER_SETUPS["lmstudio"]`:
- `env_var: "LM_API_KEY"` (canonical, what onboarding writes going forward).
- `env_var_aliases: ["LMSTUDIO_API_KEY"]` (read-only fallback for
pre-#1500 users so detection keeps working without forcing an
.env rewrite).
* `api/onboarding.py:_provider_api_key_present` reads aliases too.
* `api/providers.py:_PROVIDER_ENV_VAR["lmstudio"] = "LM_API_KEY"`.
* `api/providers.py:_PROVIDER_ENV_VAR_ALIASES["lmstudio"] = ("LMSTUDIO_API_KEY",)`
— new dict, used by `_provider_has_key` and `get_providers`'s
key_source resolution. Drops in cleanly when other providers later
rename their env vars too.
Verified:
```
before fix: webui writes LMSTUDIO_API_KEY → agent ignores it → 401 on chat
after fix: webui writes LM_API_KEY → agent picks it up → chat works
pre-#1500 .env with LMSTUDIO_API_KEY → still has_key=True in Settings
→ key_source='env_file'
```
Tests
* `tests/test_issue1499_onboarding_probe.py` — 17 tests:
3 invalid_url variants, dns, connect_refused, success (OpenAI shape),
success (bare-list shape), http_4xx, http_5xx, parse non-JSON, parse
wrong-shape, api_key authorization header passthrough, "probe must
not write to config.yaml or .env", PROBE_ERROR_CODES contract pin,
3 end-to-end route-level smoke tests against the live server fixture.
* `tests/test_issue1500_lmstudio_env_var_alignment.py` — 5 tests:
onboarding declares LM_API_KEY canonical with LMSTUDIO_API_KEY alias,
onboarding writes ONLY the canonical name, legacy env var still
detected post-migration, canonical takes precedence when both are
set, _provider_api_key_present reads aliases.
* `tests/test_issue1420_lmstudio_provider_env_var.py` — updated:
the original 5-test #1420 suite now pins LM_API_KEY as canonical
and LMSTUDIO_API_KEY as alias.
Full suite: 3879 → 3901 passing (+22), 0 failures.
Out of scope (explicitly NOT addressed here)
The third LM Studio onboarding sub-bug from #1420's thread — that
`apply_onboarding_setup` requires a non-empty api_key for lmstudio
even though most LM Studio installs run keyless — remains. The agent's
`LMSTUDIO_NOAUTH_PLACEHOLDER` substitution kicks in at runtime, but
the onboarding wizard rejects the empty-key case at submit. Fixing
this requires a UX decision (auto-write a sentinel? loosen the
required-key check for self-hosted providers?) and is left as a
separate follow-up.
Closes#1499Closes#1500
Co-authored-by: chwps <106549456+chwps@users.noreply.github.com>
Co-authored-by: AdoneyGalvan <25235323+AdoneyGalvan@users.noreply.github.com>
Frank Song