server.py exited SILENTLY after 9-16h: no traceback, no shutdown-audit line,
no core dump — the log just stopped mid-request. It runs a ThreadingHTTPServer
with daemon_threads=True, so an unhandled exception in a request/SSE/long-poll
handler thread could terminate work with nothing recorded, and faulthandler was
not enabled so a native crash left nothing at all. The WebUI also configures no
logging handlers, so INFO/ERROR records are dropped by logging's lastResort
filter (WARNING+ only) — meaning even the existing shutdown audit never reached
the log.
Add api/crash_visibility.py (stdlib-only, hooks never raise) and wire
install_crash_visibility() into server.main() before any heavy startup:
* faulthandler.enable(all_threads=True) — native crash dumps a C-level
traceback; SIGUSR1 registered for on-demand hang diagnosis.
* threading.excepthook — logs uncaught daemon/handler-thread exceptions
(thread name, ident, traceback) instead of losing them silently.
* sys.excepthook — logs uncaught main-thread exceptions (KeyboardInterrupt
preserved).
* atexit exit-audit breadcrumb — a clean/unwound exit is recorded; its
absence narrows a silent death to an un-unwound kill (OOM/SIGKILL/abort).
All diagnostics are written directly to the fault stream (stderr, which the
bootstrap redirects into the WebUI log) AND mirrored through logging, so the
line is guaranteed to land regardless of logging config.
No request-handling behavior change, no new deps. Paired memory root-cause: #4765.
Fixes#4633.
5bca5db900