HERMES-GIELIE/api/helpers.py

176 lines
6.5 KiB
Python
Raw Normal View History

"""
2026-03-31 07:02:47 +00:00
Hermes Web UI -- HTTP helper functions.
"""
import json as _json
v0.46.0: security, Docker UID/GID, model discovery, i18n, cancel fix * fix: decode HTML entities before markdown processing + zh/zh-Hant translations (#239) Adds decode() helper in renderMd() to fix double-escaping of HTML entities from LLM output (e.g. &lt;code&gt; becoming &amp;lt;code&amp;gt; instead of rendering). XSS-safe: decode runs before esc(), only 5 entity patterns. Also adds 40+ missing zh (Simplified Chinese) translation keys and a new zh-Hant (Traditional Chinese) locale with 163 keys. Fix applied: removed duplicate settings_label_notifications key in both zh and zh-Hant locales. Fixes #240 * fix: restore custom model list discovery with config api key (#238) get_available_models() now reads api_key from config.yaml before env vars: 1. model.api_key 2. providers.<active>.api_key / providers.custom.api_key 3. env var fallbacks (HERMES_API_KEY, OPENAI_API_KEY, etc.) Also adds OpenAI/Python User-Agent header and a regression test covering authenticated /v1/models discovery. Fixes users with LM Studio / Ollama custom endpoints configured in config.yaml whose model picker silently collapsed to the default model. * feat: Docker UID/GID matching to avoid root-owned .hermes files (#237) Adds docker_init.bash with hermeswebuitoo/hermeswebui user pattern so container files match the host user UID/GID. Prevents .hermes volume mounts from being owned by root when using a non-root host user. Configure via WANTED_UID and WANTED_GID env vars (default 1000/1000). Readme updated with setup instructions. Fix applied: removed duplicate WANTED_GID=1000 line in docker-compose.yml that was overriding the ${GID:-1000} variable expansion. * security: redact credentials from API responses and fix credential file permissions (#243) Adds response-layer credential redaction to three endpoints: - GET /api/session — messages[], tool_calls[], and title - GET /api/session/export — download also redacted - SSE done event — session payload in stream - GET /api/memory — MEMORY.md and USER.md content Adds api/startup.py with fix_credential_permissions() at server startup. Adds 13 tests in tests/test_security_redaction.py. Merged with #237 container detection changes in server.py. * fix: cancel button now interrupts agent and cleans up UI state (#244) Wires agent.interrupt() into cancel_stream() so the backend actually stops tool execution when the user clicks Cancel, rather than only stopping the SSE stream while the agent keeps running. Changes: - api/config.py: adds AGENT_INSTANCES dict (stream_id -> AIAgent) - api/streaming.py: stores agent in AGENT_INSTANCES after creation, checks CANCEL_FLAGS immediately after store (race condition fix), calls agent.interrupt() in cancel_stream(), cleans up in finally block - static/boot.js: removes stale setStatus(cancelling) call - static/messages.js: setBusy(false)/setStatus('') unconditionally on cancel Race condition fix: after storing agent in AGENT_INSTANCES, immediately checks if CANCEL_FLAGS[stream_id] is already set (cancel arrived during agent init) and interrupts before starting. Check is inside the same STREAMS_LOCK acquisition, making it atomic. New test file: tests/test_cancel_interrupt.py with 6 unit tests. * docs: v0.46.0 release notes, bump version, update test counts --------- Co-authored-by: Nathan Esquenazi <nesquena@gmail.com>
2026-04-11 17:17:52 +00:00
import re as _re
from pathlib import Path
from api.config import IMAGE_EXTS, MD_EXTS
def require(body: dict, *fields) -> None:
"""Phase D: Validate required fields. Raises ValueError with clean message."""
missing = [f for f in fields if not body.get(f) and body.get(f) != 0]
if missing:
raise ValueError(f"Missing required field(s): {', '.join(missing)}")
def bad(handler, msg, status: int=400):
"""Return a clean JSON error response."""
return j(handler, {'error': msg}, status=status)
release: v0.39.0 — security hardening, 12 fixes (#171) * Security: harden auth, CSRF, SSRF, XSS, and env race conditions Twelve fixes from a full security audit: CRITICAL - Add CSRF Origin/Referer validation on all POST endpoints (prevents cross-origin abuse of self-update, settings, file ops) HIGH - Unify password hashing: config.py now uses PBKDF2 (600k iters) instead of single-iteration SHA-256 - Add per-IP rate limiting on login (5 attempts/60s, 429 on excess) MEDIUM - Validate session IDs as hex-only before filesystem operations (prevents path traversal via crafted session ID) - SSRF: resolve DNS before private-IP check in model fetching (prevents DNS rebinding to internal services) - Warn loudly when binding non-loopback without password set - SSE env var mutations: wrap sync chat + streaming restore in _ENV_LOCK - Force Content-Disposition:attachment for HTML/XHTML/SVG uploads (prevents stored XSS via uploaded files) LOW - Extend HMAC session signature from 64 to 128 bits - Add resolve()+relative_to() check on skills path construction - Set Secure flag on session cookie when connection is HTTPS - Sanitize exception messages to strip filesystem paths No breaking changes. All fixes are backward-compatible. * fix: use getattr for Secure cookie SSL detection handler.request.getpeercert raises AttributeError on plain sockets (non-SSL). Use getattr(..., None) to safely check for SSL. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * tests: add sprint 29 security hardening coverage (PR #171) 33 tests covering all 12 security fixes: - CSRF origin/referer validation - Login rate limiting (5 attempts/60s) - Session ID hex validation (path traversal prevention) - Error path sanitization (_sanitize_error) - Secure cookie getattr safety - HMAC signature length (64->128 bit) - Skills path traversal prevention - Content-Disposition for HTML/SVG/XHTML - PBKDF2 password hashing verification - Non-loopback startup warning - SSRF DNS guard code presence - _ENV_LOCK export from streaming module * release: v0.39.0 — security hardening, 12 fixes (#171) --------- Co-authored-by: betamod <matthew.sloly@gmail.com> Co-authored-by: Nathan Esquenazi <nesquena@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 05:26:03 +00:00
def _sanitize_error(e: Exception) -> str:
"""Strip filesystem paths from exception messages before returning to client."""
import re
msg = str(e)
# Remove absolute paths (Unix and Windows)
msg = re.sub(r'(?:(?:/[a-zA-Z0-9_.-]+)+|(?:[A-Z]:\\[^\s]+))', '<path>', msg)
return msg
def safe_resolve(root: Path, requested: str) -> Path:
"""Resolve a relative path inside root, raising ValueError on traversal."""
resolved = (root / requested).resolve()
resolved.relative_to(root.resolve()) # raises ValueError if outside root
return resolved
def _security_headers(handler):
"""Add security headers to every response."""
handler.send_header('X-Content-Type-Options', 'nosniff')
handler.send_header('X-Frame-Options', 'DENY')
handler.send_header('Referrer-Policy', 'same-origin')
handler.send_header(
'Content-Security-Policy',
"default-src 'self'; "
"script-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net; "
"style-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net; "
"img-src 'self' data:; font-src 'self' data:; connect-src 'self'; "
"base-uri 'self'; form-action 'self'"
)
handler.send_header(
'Permissions-Policy',
'camera=(), microphone=(), geolocation=()'
)
def j(handler, payload, status: int=200) -> None:
"""Send a JSON response."""
body = _json.dumps(payload, ensure_ascii=False, indent=2).encode('utf-8')
handler.send_response(status)
handler.send_header('Content-Type', 'application/json; charset=utf-8')
handler.send_header('Content-Length', str(len(body)))
handler.send_header('Cache-Control', 'no-store')
_security_headers(handler)
handler.end_headers()
handler.wfile.write(body)
def t(handler, payload, status: int=200, content_type: str='text/plain; charset=utf-8') -> None:
"""Send a plain text or HTML response."""
body = payload if isinstance(payload, bytes) else str(payload).encode('utf-8')
handler.send_response(status)
handler.send_header('Content-Type', content_type)
handler.send_header('Content-Length', str(len(body)))
handler.send_header('Cache-Control', 'no-store')
_security_headers(handler)
handler.end_headers()
handler.wfile.write(body)
MAX_BODY_BYTES = 20 * 1024 * 1024 # 20MB limit for non-upload POST bodies
v0.46.0: security, Docker UID/GID, model discovery, i18n, cancel fix * fix: decode HTML entities before markdown processing + zh/zh-Hant translations (#239) Adds decode() helper in renderMd() to fix double-escaping of HTML entities from LLM output (e.g. &lt;code&gt; becoming &amp;lt;code&amp;gt; instead of rendering). XSS-safe: decode runs before esc(), only 5 entity patterns. Also adds 40+ missing zh (Simplified Chinese) translation keys and a new zh-Hant (Traditional Chinese) locale with 163 keys. Fix applied: removed duplicate settings_label_notifications key in both zh and zh-Hant locales. Fixes #240 * fix: restore custom model list discovery with config api key (#238) get_available_models() now reads api_key from config.yaml before env vars: 1. model.api_key 2. providers.<active>.api_key / providers.custom.api_key 3. env var fallbacks (HERMES_API_KEY, OPENAI_API_KEY, etc.) Also adds OpenAI/Python User-Agent header and a regression test covering authenticated /v1/models discovery. Fixes users with LM Studio / Ollama custom endpoints configured in config.yaml whose model picker silently collapsed to the default model. * feat: Docker UID/GID matching to avoid root-owned .hermes files (#237) Adds docker_init.bash with hermeswebuitoo/hermeswebui user pattern so container files match the host user UID/GID. Prevents .hermes volume mounts from being owned by root when using a non-root host user. Configure via WANTED_UID and WANTED_GID env vars (default 1000/1000). Readme updated with setup instructions. Fix applied: removed duplicate WANTED_GID=1000 line in docker-compose.yml that was overriding the ${GID:-1000} variable expansion. * security: redact credentials from API responses and fix credential file permissions (#243) Adds response-layer credential redaction to three endpoints: - GET /api/session — messages[], tool_calls[], and title - GET /api/session/export — download also redacted - SSE done event — session payload in stream - GET /api/memory — MEMORY.md and USER.md content Adds api/startup.py with fix_credential_permissions() at server startup. Adds 13 tests in tests/test_security_redaction.py. Merged with #237 container detection changes in server.py. * fix: cancel button now interrupts agent and cleans up UI state (#244) Wires agent.interrupt() into cancel_stream() so the backend actually stops tool execution when the user clicks Cancel, rather than only stopping the SSE stream while the agent keeps running. Changes: - api/config.py: adds AGENT_INSTANCES dict (stream_id -> AIAgent) - api/streaming.py: stores agent in AGENT_INSTANCES after creation, checks CANCEL_FLAGS immediately after store (race condition fix), calls agent.interrupt() in cancel_stream(), cleans up in finally block - static/boot.js: removes stale setStatus(cancelling) call - static/messages.js: setBusy(false)/setStatus('') unconditionally on cancel Race condition fix: after storing agent in AGENT_INSTANCES, immediately checks if CANCEL_FLAGS[stream_id] is already set (cancel arrived during agent init) and interrupts before starting. Check is inside the same STREAMS_LOCK acquisition, making it atomic. New test file: tests/test_cancel_interrupt.py with 6 unit tests. * docs: v0.46.0 release notes, bump version, update test counts --------- Co-authored-by: Nathan Esquenazi <nesquena@gmail.com>
2026-04-11 17:17:52 +00:00
# ── Credential redaction ──────────────────────────────────────────────────────
def _build_redact_fn():
"""Return redact_sensitive_text from hermes-agent if available, else a fallback."""
try:
from agent.redact import redact_sensitive_text
return redact_sensitive_text
except ImportError:
pass
# Minimal fallback covering the most common credential prefixes
_CRED_RE = _re.compile(
r"(?<![A-Za-z0-9_-])("
r"sk-[A-Za-z0-9_-]{10,}" # OpenAI / Anthropic / OpenRouter
r"|ghp_[A-Za-z0-9]{10,}" # GitHub PAT (classic)
r"|github_pat_[A-Za-z0-9_]{10,}" # GitHub PAT (fine-grained)
r"|gho_[A-Za-z0-9]{10,}" # GitHub OAuth token
r"|ghu_[A-Za-z0-9]{10,}" # GitHub user-to-server token
r"|ghs_[A-Za-z0-9]{10,}" # GitHub server-to-server token
r"|ghr_[A-Za-z0-9]{10,}" # GitHub refresh token
r"|AKIA[A-Z0-9]{16}" # AWS Access Key ID
r"|xox[baprs]-[A-Za-z0-9-]{10,}" # Slack tokens
r"|hf_[A-Za-z0-9]{10,}" # HuggingFace token
r"|SG\.[A-Za-z0-9_-]{10,}" # SendGrid API key
r")(?![A-Za-z0-9_-])"
)
_AUTH_HDR_RE = _re.compile(r"(Authorization:\s*Bearer\s+)(\S+)", _re.IGNORECASE)
_ENV_RE = _re.compile(
r"([A-Z0-9_]{0,50}(?:API_?KEY|TOKEN|SECRET|PASSWORD|PASSWD|CREDENTIAL|AUTH)[A-Z0-9_]{0,50})"
r"\s*=\s*(['\"]?)(\S+)\2"
)
_PRIVKEY_RE = _re.compile(
r"-----BEGIN[A-Z ]*PRIVATE KEY-----[\s\S]*?-----END[A-Z ]*PRIVATE KEY-----"
)
def _mask(token: str) -> str:
return f"{token[:6]}...{token[-4:]}" if len(token) >= 18 else "***"
def _fallback_redact(text: str) -> str:
if not isinstance(text, str) or not text:
return text
text = _CRED_RE.sub(lambda m: _mask(m.group(1)), text)
text = _AUTH_HDR_RE.sub(lambda m: m.group(1) + _mask(m.group(2)), text)
text = _ENV_RE.sub(
lambda m: f"{m.group(1)}={m.group(2)}{_mask(m.group(3))}{m.group(2)}", text
)
text = _PRIVKEY_RE.sub("[REDACTED PRIVATE KEY]", text)
return text
return _fallback_redact
_redact_text = _build_redact_fn()
def _redact_value(v):
"""Recursively redact credentials from strings, dicts, and lists."""
if isinstance(v, str):
return _redact_text(v)
if isinstance(v, dict):
return {k: _redact_value(val) for k, val in v.items()}
if isinstance(v, list):
return [_redact_value(item) for item in v]
return v
def redact_session_data(session_dict: dict) -> dict:
"""Redact credentials from message content and tool_call data before API response.
Applies to: messages[], tool_calls[], and title.
The underlying session file is not modified; redaction is response-layer only.
"""
result = dict(session_dict)
if isinstance(result.get('title'), str):
result['title'] = _redact_text(result['title'])
if 'messages' in result:
result['messages'] = _redact_value(result['messages'])
if 'tool_calls' in result:
result['tool_calls'] = _redact_value(result['tool_calls'])
return result
def read_body(handler) -> dict:
"""Read and JSON-parse a POST request body (capped at 20MB)."""
length = int(handler.headers.get('Content-Length', 0))
if length > MAX_BODY_BYTES:
raise ValueError(f'Request body too large ({length} bytes, max {MAX_BODY_BYTES})')
raw = handler.rfile.read(length) if length else b'{}'
try:
return _json.loads(raw)
except Exception:
return {}