fix(tests): eliminate the two chronic full-suite flakes (nous_picker + service_tier)
Both fail ONLY in the full suite, not in isolation — they've been noise on every
release gate. Root-caused and fixed at the source (no retries/skips):
1. test_issue1567 nous_picker (test_hermes_cli_unavailable_falls_back_to_static_4):
NOT actually a flake — a real assertion that silently broke as the curated list
grew. It asserted every curated @nous: id (now 28) appears in the picker group's
visible 'models', but when the list exceeds _NOUS_FEATURED_THRESHOLD (25) the
dropdown intentionally shows only the featured subset and stashes the rest under
'extra_models'. Fixed to assert the invariant that matters ('no curated entry is
DROPPED') against the visible+extra_models union, so it tracks the real behavior
instead of drifting to failure every time the catalog grows.
2. test_issue4536 service_tier (test_main_service_tier_roundtrip_via_auxiliary_endpoint):
Classic test-isolation leak. get_auxiliary_models() reads the module-global
config.cfg after reload_config(); if an EARLIER full-suite test rebinds
config.cfg via monkeypatch and leaves 'cfg is not _cfg_cache',
_cfg_has_in_memory_overrides() stays True and get_auxiliary_models() reads that
stale dict instead of the freshly-reloaded temp config. Added an autouse fixture
that re-aliases cfg to _cfg_cache and clears the override fingerprint before AND
after each test in the class. PROVEN: reproduced the failure deterministically
with a leak-probe test, confirmed the fixture neutralizes it (fixture off = FAIL,
fixture on = PASS).
This commit is contained in:
parent
980e98dc7e
commit
f3f328df3d
2 changed files with 43 additions and 5 deletions
|
|
@ -471,17 +471,27 @@ class TestNousLiveFetchEmpty:
|
|||
"the curated static fallback so the picker isn't empty in "
|
||||
"test envs that lack the agent package."
|
||||
)
|
||||
# The original curated 4 entries must all be present.
|
||||
nous_ids = {m["id"] for m in nous_groups[0]["models"]}
|
||||
# Every curated @nous: entry must survive the static fallback — but
|
||||
# when the curated list exceeds _NOUS_FEATURED_THRESHOLD (>25) the
|
||||
# picker dropdown intentionally shows only the featured subset and
|
||||
# stashes the rest under "extra_models" (for /model autocomplete).
|
||||
# The invariant is "no curated entry is DROPPED", so check the full
|
||||
# returned set (visible models + extra_models), not just the visible
|
||||
# dropdown slice (which drifts to failure as the curated list grows
|
||||
# past the threshold — the root cause of the pre-fix flake).
|
||||
_nous_group = nous_groups[0]
|
||||
nous_ids = {m["id"] for m in _nous_group.get("models", [])}
|
||||
nous_ids |= {m["id"] for m in _nous_group.get("extra_models", [])}
|
||||
_curated_nous_ids = {
|
||||
m["id"] for m in config._PROVIDER_MODELS.get("nous", [])
|
||||
if m["id"].startswith("@nous:")
|
||||
}
|
||||
assert _curated_nous_ids.issubset(nous_ids), (
|
||||
f"Static fallback is missing curated @nous: entries. "
|
||||
f"Expected subset {_curated_nous_ids}, got {nous_ids}"
|
||||
f"Static fallback dropped curated @nous: entries. "
|
||||
f"Missing {_curated_nous_ids - nous_ids} "
|
||||
f"(visible+extra_models total {len(nous_ids)})"
|
||||
)
|
||||
assert len(nous_groups[0]["models"]) >= 4, (
|
||||
assert len(nous_ids) >= 4, (
|
||||
"Static fallback should expose at least the curated 4-entry "
|
||||
"list from _PROVIDER_MODELS['nous']."
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,7 +1,35 @@
|
|||
"""Tests for issue #4536 — main-model service_tier persistence and guarded forwarding."""
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestIssue4536ServiceTier:
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_config_globals(self):
|
||||
"""Make these tests hermetic against a leaked `config.cfg` rebind.
|
||||
|
||||
get_auxiliary_models() reads the module-global `config.cfg` after
|
||||
reload_config() refreshes `_cfg_cache`. If an EARLIER test in the full
|
||||
suite rebound `config.cfg` to its own dict via monkeypatch and its
|
||||
teardown left `cfg is not _cfg_cache`, then `_cfg_has_in_memory_overrides()`
|
||||
stays True and get_auxiliary_models() reads that stale dict instead of the
|
||||
freshly-reloaded temp config — the root cause of this test's full-suite-only
|
||||
flake (passes in isolation). Re-alias `cfg` to the live cache and clear the
|
||||
override fingerprint before AND after, so neither an inherited leak nor our
|
||||
own run can poison a neighbor.
|
||||
"""
|
||||
from api import config
|
||||
|
||||
def _reset():
|
||||
with config._cfg_lock:
|
||||
config.cfg = config._cfg_cache
|
||||
config._cfg_fingerprint = None
|
||||
config._cfg_mtime = 0.0
|
||||
|
||||
_reset()
|
||||
yield
|
||||
_reset()
|
||||
|
||||
def test_main_service_tier_roundtrip_via_auxiliary_endpoint(self, monkeypatch, tmp_path):
|
||||
"""service_tier set on main model should persist in config and return via /api/model/auxiliary payload."""
|
||||
from api import config
|
||||
|
|
|
|||
Loading…
Reference in a new issue