fix(tests): eliminate the two chronic full-suite flakes (nous_picker + service_tier)

Both fail ONLY in the full suite, not in isolation — they've been noise on every
release gate. Root-caused and fixed at the source (no retries/skips):

1. test_issue1567 nous_picker (test_hermes_cli_unavailable_falls_back_to_static_4):
   NOT actually a flake — a real assertion that silently broke as the curated list
   grew. It asserted every curated @nous: id (now 28) appears in the picker group's
   visible 'models', but when the list exceeds _NOUS_FEATURED_THRESHOLD (25) the
   dropdown intentionally shows only the featured subset and stashes the rest under
   'extra_models'. Fixed to assert the invariant that matters ('no curated entry is
   DROPPED') against the visible+extra_models union, so it tracks the real behavior
   instead of drifting to failure every time the catalog grows.

2. test_issue4536 service_tier (test_main_service_tier_roundtrip_via_auxiliary_endpoint):
   Classic test-isolation leak. get_auxiliary_models() reads the module-global
   config.cfg after reload_config(); if an EARLIER full-suite test rebinds
   config.cfg via monkeypatch and leaves 'cfg is not _cfg_cache',
   _cfg_has_in_memory_overrides() stays True and get_auxiliary_models() reads that
   stale dict instead of the freshly-reloaded temp config. Added an autouse fixture
   that re-aliases cfg to _cfg_cache and clears the override fingerprint before AND
   after each test in the class. PROVEN: reproduced the failure deterministically
   with a leak-probe test, confirmed the fixture neutralizes it (fixture off = FAIL,
   fixture on = PASS).
This commit is contained in:
nesquena-hermes 2026-07-04 18:32:43 +00:00
parent 980e98dc7e
commit f3f328df3d
2 changed files with 43 additions and 5 deletions

View file

@ -471,17 +471,27 @@ class TestNousLiveFetchEmpty:
"the curated static fallback so the picker isn't empty in "
"test envs that lack the agent package."
)
# The original curated 4 entries must all be present.
nous_ids = {m["id"] for m in nous_groups[0]["models"]}
# Every curated @nous: entry must survive the static fallback — but
# when the curated list exceeds _NOUS_FEATURED_THRESHOLD (>25) the
# picker dropdown intentionally shows only the featured subset and
# stashes the rest under "extra_models" (for /model autocomplete).
# The invariant is "no curated entry is DROPPED", so check the full
# returned set (visible models + extra_models), not just the visible
# dropdown slice (which drifts to failure as the curated list grows
# past the threshold — the root cause of the pre-fix flake).
_nous_group = nous_groups[0]
nous_ids = {m["id"] for m in _nous_group.get("models", [])}
nous_ids |= {m["id"] for m in _nous_group.get("extra_models", [])}
_curated_nous_ids = {
m["id"] for m in config._PROVIDER_MODELS.get("nous", [])
if m["id"].startswith("@nous:")
}
assert _curated_nous_ids.issubset(nous_ids), (
f"Static fallback is missing curated @nous: entries. "
f"Expected subset {_curated_nous_ids}, got {nous_ids}"
f"Static fallback dropped curated @nous: entries. "
f"Missing {_curated_nous_ids - nous_ids} "
f"(visible+extra_models total {len(nous_ids)})"
)
assert len(nous_groups[0]["models"]) >= 4, (
assert len(nous_ids) >= 4, (
"Static fallback should expose at least the curated 4-entry "
"list from _PROVIDER_MODELS['nous']."
)

View file

@ -1,7 +1,35 @@
"""Tests for issue #4536 — main-model service_tier persistence and guarded forwarding."""
import pytest
class TestIssue4536ServiceTier:
@pytest.fixture(autouse=True)
def _isolate_config_globals(self):
"""Make these tests hermetic against a leaked `config.cfg` rebind.
get_auxiliary_models() reads the module-global `config.cfg` after
reload_config() refreshes `_cfg_cache`. If an EARLIER test in the full
suite rebound `config.cfg` to its own dict via monkeypatch and its
teardown left `cfg is not _cfg_cache`, then `_cfg_has_in_memory_overrides()`
stays True and get_auxiliary_models() reads that stale dict instead of the
freshly-reloaded temp config — the root cause of this test's full-suite-only
flake (passes in isolation). Re-alias `cfg` to the live cache and clear the
override fingerprint before AND after, so neither an inherited leak nor our
own run can poison a neighbor.
"""
from api import config
def _reset():
with config._cfg_lock:
config.cfg = config._cfg_cache
config._cfg_fingerprint = None
config._cfg_mtime = 0.0
_reset()
yield
_reset()
def test_main_service_tier_roundtrip_via_auxiliary_endpoint(self, monkeypatch, tmp_path):
"""service_tier set on main model should persist in config and return via /api/model/auxiliary payload."""
from api import config