Commit graph

4 commits

Author SHA1 Message Date
nesquena-hermes
3ad07ba363 fix(agent_health): read cross-container gateway runtime status from gateway_state.json (#5065)
In cross-container Docker setups the agent-health check resolves the gateway's
runtime status from a shared gateway_state.json. After the keyword-form
read_runtime_status(pid_path=...) call fails against an older agent signature,
_read_gateway_runtime_status fell back to read_runtime_status(pid_path) passing
the sibling gateway.pid path positionally — so the reader parsed PID metadata as
runtime metadata and gateway_state/updated_at never resolved, breaking the
freshness check (issue #5030, #1879).

The positional fallback now passes the correct gateway_state.json sibling path
(pid_path.with_name(runtime_status_file), where runtime_status_file comes from
the agent's _RUNTIME_STATUS_FILE attr, defaulting to gateway_state.json). The
path computation is hoisted so both fallback branches use the correct file. The
keyword happy-path is unchanged for agents that accept pid_path=..., and normal
PID-based health is unaffected.

Verified: the new test test_cross_container_runtime_status_reads_sibling_runtime_file
fails on master without the fix and passes with it; #716 PID-health tests still pass.

Co-authored-by: rodboev <rodboev@users.noreply.github.com>
2026-06-27 19:54:50 +00:00
Lumen Yang
e37c69cf57 fix(agent-health): treat stale running gateway as unknown
(cherry picked from commit 4be346fece529118b652485d9045080f03e326cf)
2026-05-11 23:13:09 +00:00
Frank Song
b38cc2f1ea Mute stale stopped gateway heartbeat 2026-05-09 14:53:42 +08:00
Sanjay Santhanam
efcfff3d7f fix(#1879): cross-container gateway liveness via state-file freshness
The dashboard banner 'Hermes agent is not responding' fires on every
multi-container deployment that doesn't set 'pid: "service:hermes-agent"'
in compose, because get_running_pid() relies on fcntl.flock and
os.kill(pid, 0) — both PID-namespace-scoped and invisible across container
boundaries.

Fix: when get_running_pid() returns None, fall back to a freshness check on
gateway_state.json. The gateway already writes that file on every tick with
gateway_state == 'running' and an aware ISO-8601 updated_at timestamp, so a
recent (<= 120s) timestamp is an equivalent live-process signal that needs
only a shared volume — no PID namespace, no compose workaround, no extra
HTTP probe URL.

Behavior preserved:
- In-namespace deployments still hit the PID-based path first; payload shape
  unchanged (no 'reason' key) so #716 contract holds.
- Cross-container alive path adds reason='cross_container_freshness' so
  support diagnostics can tell which signal succeeded.
- Stale updated_at, non-running gateway_state, malformed/naive/missing
  timestamps, and timestamps far in the future all still report 'down' — the
  fallback never produces a false positive.
- Same redaction rules: argv/command/executable/env/raw pid never leak.

Tests: 15 new cases in test_issue1879_cross_container_gateway_liveness.py
covering the cross-container alive path, every refusal case, clock-skew
tolerance, and backward compat with the #716 PID path. Existing #716
heartbeat tests (8) continue to pass.
2026-05-08 15:15:50 +00:00