Initial public release — Inspector v0.7.0
Docker-watchdog with per-container restart rules, Gotify/Pushover notifications, and an embedded dashboard + web settings page. See README.md for an overview and INSTALL.md for the Docker quickstart.
This commit is contained in:
Europium 63 committed
commit
cb67f70832
35 files changed
+5579
No files matched your search
Executable
+18
@@ -0,0 +1,18 @@
|
||||
.git
|
||||
.gitignore
|
||||
.dockerignore
|
||||
.aider.conf.yml
|
||||
__pycache__
|
||||
*.pyc
|
||||
.venv
|
||||
venv
|
||||
docs
|
||||
logo
|
||||
logs/*.log
|
||||
logs/*.log.*
|
||||
config.yml
|
||||
README.md
|
||||
AGENTS.md
|
||||
CLAUDE.md
|
||||
GEMINI.md
|
||||
CONVENTIONS.md
|
||||
Executable
+606
@@ -0,0 +1,606 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to Inspector are documented here.
|
||||
|
||||
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
---
|
||||
|
||||
## [0.7.0] — 2026-10-26
|
||||
|
||||
### Added
|
||||
|
||||
- **Web settings page** at `/settings` for editing `config.yml` without SSH:
|
||||
per-section forms for Inspector loop, Gotify, Pushover, Dashboard,
|
||||
Database, Health summary, Maintenance, Notification mute, and per-container
|
||||
tuning (thresholds, cooldowns, `restart_on`, color).
|
||||
- Saves round-trip through ruamel.yaml so comments and key order in
|
||||
`config.yml` are preserved. Each save keeps a `.bak` of the prior file.
|
||||
(In the Docker container `.bak` lives in the overlay fs because
|
||||
`config.yml` is a single-file bind mount; the live file is still updated
|
||||
cleanly.)
|
||||
- In-process hot reload via a new `ConfigRef`: edits to container-level
|
||||
settings and `inspector.dry_run`/`heartbeat`/`interval_seconds` take
|
||||
effect on the next loop iteration without a container restart. Fields
|
||||
requiring restart are explicitly labeled in the UI.
|
||||
- CSRF token on all cookie-authenticated POSTs (Settings forms,
|
||||
maintenance toggle, mute toggle). Basic Auth clients bypass the check
|
||||
because they prove identity on every request.
|
||||
- Secret fields (Gotify token, Pushover token/user_key, dashboard password)
|
||||
render masked and treat the `__UNCHANGED__` sentinel as "keep existing";
|
||||
real secret values are never echoed into HTML.
|
||||
|
||||
### Changed
|
||||
|
||||
- `main.run_loop()` now takes a `ConfigRef`, re-reads config each
|
||||
iteration, and rebuilds `enabled_configs`/`check_order` so edits apply
|
||||
without a restart.
|
||||
- `DashboardServer` constructor simplified: now takes `ConfigRef`,
|
||||
`states`, `maintenance`, `notification_mute` (per-section configs are
|
||||
resolved live via `ConfigRef.get()`).
|
||||
- Added `ruamel.yaml==0.18.6` to `requirements.txt`.
|
||||
|
||||
---
|
||||
|
||||
## [0.6.0] — 2026-10-26
|
||||
|
||||
### Added
|
||||
|
||||
- **Pushover notifications** as a second independent channel alongside Gotify.
|
||||
Enable either, both, or neither under `notifications.pushover`; alerts fan
|
||||
out to every enabled channel. Supports optional `device` and `sound`
|
||||
overrides.
|
||||
- Priority mapping from Inspector's Gotify-style integer scale to Pushover's
|
||||
`-2..2` range: `<=2 → -1`, `3..4 → 0`, `5..6 → 1`, `>=7 → 2`. Emergency
|
||||
tier (`2`) ships default `retry`/`expire` values and triggers the usual
|
||||
Pushover acknowledgement prompt.
|
||||
- `MultiNotifier` internally fans each `notify()` call out to every enabled
|
||||
channel; a crash in one channel never prevents the others from running.
|
||||
|
||||
---
|
||||
|
||||
## [0.5.5] — 2026-06-18
|
||||
|
||||
### Changed
|
||||
|
||||
- Timestamp display format shortened: year and timezone abbreviation (PST/PDT) removed.
|
||||
Timestamps now render as `MM-DD HH:MM AM/PM` (e.g. `06-18 09:30 AM`).
|
||||
|
||||
---
|
||||
|
||||
## [0.5.4] — 2026-06-17
|
||||
|
||||
### Added
|
||||
|
||||
- Dashboard cards now surface **scheduled maintenance** for containers opted into the recurring
|
||||
daily window (`maintenance_window: true`):
|
||||
- while idle, a meta-row `🔧 Next maintenance: in <countdown> (<start> – <end>)` shows when
|
||||
the next window begins, with the window times in the dashboard's local format;
|
||||
- while the window is active, the phase row shows `🔧 In maintenance — ends in <countdown>`
|
||||
so it's clear when normal monitoring resumes.
|
||||
- All countdowns reuse the existing 1-second client-side ticker (no page reload needed).
|
||||
- `/api/status` now returns `maintenance_scheduled` (bool) and `maintenance_window_start` /
|
||||
`maintenance_window_end` (ISO UTC or null) per container.
|
||||
|
||||
---
|
||||
|
||||
## [0.5.3] — 2026-06-15
|
||||
|
||||
### Added
|
||||
|
||||
- Dashboard status cards now show a **phase row** below the error bar, indicating what
|
||||
Inspector is currently doing for each container:
|
||||
- `⏱ Cooldown — Xs remaining` — post-restart cooldown is active (countdown to expiry)
|
||||
- `⏱ Grace period — Xs remaining` — container is booting post-restart; failures not counted
|
||||
- `⚠ Restart limit — resets in Xs` — hourly restart cap hit; countdown to when it clears
|
||||
- `⚠ Failing X/Y — next check in Zs` — error count is accumulating toward the restart threshold
|
||||
- `👁 Observe only` — `restart_on: []` is set; Inspector tracks status but never acts
|
||||
- `🔧 In maintenance` — container is in a maintenance window
|
||||
- `Next check in Zs` — container is healthy; soft reminder of the monitoring cadence
|
||||
- All countdown timers update **every second in the browser** without a page reload (JS
|
||||
`setInterval`); the existing 30 s auto-refresh keeps data fresh.
|
||||
- `/api/status` now returns `phase`, `phase_until` (ISO UTC or null), `last_check_at`
|
||||
(ISO UTC or null), and `interval_seconds` per container entry.
|
||||
|
||||
---
|
||||
|
||||
## [0.5.2] — 2026-06-15
|
||||
|
||||
### Added
|
||||
|
||||
- Portainer environments now support `api_token: "ptr_..."` to embed the API token
|
||||
directly in `config.yml`, as an alternative to `api_token_env` (env var lookup).
|
||||
`api_token` takes precedence when both are set. Useful when injecting env vars into
|
||||
the Inspector container is inconvenient.
|
||||
- Improved token-not-found error message: now mentions both `api_token` and
|
||||
`api_token_env` options so the fix is self-evident in the logs.
|
||||
- `_find_stack()` now logs a debug line with how many stacks were scanned and which
|
||||
`endpoint_id` was used, aiding multi-environment Portainer diagnosis.
|
||||
|
||||
---
|
||||
|
||||
## [0.5.1] — 2026-06-15
|
||||
|
||||
### Fixed
|
||||
|
||||
- `health_summary` digest crash: `KeyError: 'portainer_redeploy_failed'` on the
|
||||
first periodic digest when event history contained Portainer redeploy events.
|
||||
`health_summary._LABELS` was missing the three Portainer event types
|
||||
(`portainer_redeploy`, `portainer_redeploy_failed`, `portainer_redeploy_skipped`)
|
||||
that were added to `SUMMARY_EVENT_TYPES` in v0.4.0.
|
||||
|
||||
---
|
||||
|
||||
## [0.5.0] — 2026-06-16
|
||||
|
||||
### Added
|
||||
- New per-container `recovery_action: restart_then_redeploy` — tiered recovery
|
||||
for VPN-routed containers. Inspector first tries a plain `docker restart`
|
||||
(which fixes the common case where the VPN sidecar was merely *restarted*,
|
||||
keeping the same container id) and only escalates to a Portainer stack
|
||||
redeploy when the restart errors (e.g. the stale-namespace
|
||||
"No such container" case after the sidecar is *recreated*) or the container
|
||||
is still failing on the next recovery cycle. Escalation is non-blocking — it
|
||||
happens on the following cycle, not via an inline wait. Requires
|
||||
`portainer_stack_name` like `portainer_stack_redeploy`, and honors the same
|
||||
`depends_on` gating.
|
||||
|
||||
## [0.4.2] — 2026-06-15
|
||||
|
||||
### Added
|
||||
- Result notifications for recovery actions: a successful `docker restart` /
|
||||
`start` now sends a `✅ … completed` notification, and a successful Portainer
|
||||
stack redeploy sends `✅ … redeployed`. A failed post-redeploy HTTP recheck
|
||||
sends `⚠️ … recheck failed`.
|
||||
- "Testing the Portainer API" guide in the README (uses `python3` + `requests`,
|
||||
since the container image has no `curl`).
|
||||
|
||||
### Changed
|
||||
- Portainer API errors now include the HTTP status code and response body in the
|
||||
failure message/log (e.g. `HTTP 401: {"message":"Invalid JWT token"}`), making
|
||||
misconfigured tokens/endpoints far easier to diagnose. Added intermediate
|
||||
debug logging for stack lookup and a warning when a stack name isn't found.
|
||||
|
||||
---
|
||||
|
||||
## [0.4.1] — 2026-06-15
|
||||
|
||||
### Fixed
|
||||
- Per-container `restart_on: []` (empty list) now correctly disables all
|
||||
restart triggers for that container. Previously, an empty list was
|
||||
treated as "not set" and silently fell back to the default
|
||||
`[UNHEALTHY, NOT RUNNING]`, so a container couldn't be configured as
|
||||
observe-only (e.g. for `depends_on` status tracking without Inspector
|
||||
restarting it itself). Omitting `restart_on` entirely still defaults to
|
||||
`[UNHEALTHY, NOT RUNNING]`.
|
||||
|
||||
---
|
||||
|
||||
## [0.4.0] — 2026-06-15
|
||||
|
||||
### Added
|
||||
- New per-container `recovery_action`: `restart_container` (default,
|
||||
unchanged behavior), `start_container` (calls `docker start` instead of
|
||||
`docker restart`), and `portainer_stack_redeploy`.
|
||||
- `portainer_stack_redeploy` recovers containers whose
|
||||
`network_mode: container:<vpn>` namespace goes stale after the VPN sidecar
|
||||
(e.g. Gluetun) is recreated — the `"joining network namespace of container:
|
||||
No such container"` failure a plain restart can't fix. It redeploys the
|
||||
owning Portainer stack via the Portainer API. New `integrations.portainer`
|
||||
config: named environments (token loaded from `api_token_env`, never
|
||||
logged or stored), `default_environment`, `stack_redeploy_cooldown_seconds`,
|
||||
`post_redeploy_wait_seconds`. New per-container `portainer_environment` /
|
||||
`portainer_stack_name`.
|
||||
- New per-container `depends_on` / `vpn_routed`: containers are checked (and
|
||||
recovered) in dependency order each cycle, so a VPN sidecar is recovered
|
||||
before its dependents. Recovery is deferred — without consuming the
|
||||
container's cooldown — while its `depends_on` container isn't
|
||||
healthy/starting; `vpn_routed: true` logs why.
|
||||
- Dry-run extended to cover the new actions, including the Portainer
|
||||
redeploy flow (`DRY RUN: would resolve/redeploy/wait Portainer ...`).
|
||||
- New event types: `portainer_redeploy`, `portainer_redeploy_failed`,
|
||||
`portainer_redeploy_skipped`.
|
||||
- See [Portainer stack redeploy recovery](README.md#portainer-stack-redeploy-recovery)
|
||||
in the README.
|
||||
- New per-container notification mute: `notification_mute` config
|
||||
(`enabled`, `dir`, `default_hours`, default 12h auto-expiring), dashboard
|
||||
"Mute notifications"/"Unmute notifications" button + "Muted" badge,
|
||||
`POST /notifications/<name>/mute|unmute`. Muting silences outbound Gotify
|
||||
notifications only — monitoring, failure counting, restarts/recovery,
|
||||
cooldowns, event history, and the health-summary digest continue
|
||||
unaffected. New event types `notification_mute_enter`/`exit`. See
|
||||
[Muting notifications](README.md#muting-notifications) in the README.
|
||||
|
||||
All new fields are optional and backward-compatible; existing configs are
|
||||
unaffected. `recovery_action: recreate_container` is **not** implemented and
|
||||
is rejected at config load.
|
||||
|
||||
---
|
||||
|
||||
## [0.3.1] — 2026-06-14
|
||||
|
||||
### Changed
|
||||
- `http_check` now classifies outcomes into three explicit tiers: the
|
||||
configured `expected_status` = up, a connection refused / timeout / DNS
|
||||
failure = unreachable (reported as the synthetic status code `0`), and any
|
||||
other code = "responding but unexpected". Connection-level failures were
|
||||
already treated as failed checks (→ `UNREACHABLE`); this makes the
|
||||
distinction explicit in the result (`HttpCheckResult.status_code`) and in
|
||||
the log / notification messages. Matching remains exact — the README now
|
||||
documents how to find a container's actual status code.
|
||||
|
||||
---
|
||||
|
||||
## [0.3.0] — 2026-06-14
|
||||
|
||||
### Added
|
||||
- Opt-in "unreachable" detection per container: even when Docker reports a
|
||||
container as healthy/running, Inspector can independently verify it's
|
||||
actually reachable via an HTTP/HTML health check (`http_check`) and/or a
|
||||
shared [Uptime Kuma](https://github.com/louislam/uptime-kuma) status page
|
||||
(`integrations.uptime_kuma` + per-container `uptime_kuma.monitor_name`).
|
||||
A failing check on an otherwise-healthy container surfaces as a new
|
||||
`UNREACHABLE` status (red badge on the dashboard); opt it into a
|
||||
container's `restart_on` to have it count toward failures/restarts like
|
||||
any other status. Both checks default to disabled and never crash
|
||||
Inspector.
|
||||
- Events page (`/events`) redesigned as a daily view: shows one calendar
|
||||
day (12:00 AM-11:59 PM US Pacific) at a time, defaulting to today, with
|
||||
"Prev day" / "Next day" navigation (`?date=YYYY-MM-DD`, also supported by
|
||||
`/api/events`). The timeline now always renders all 24 hourly markers
|
||||
(`12 AM`..`11 PM`) so you can scroll directly to a time of day, plus a
|
||||
trailing "11 PM (yesterday)" section for the 1-hour lookback overlap from
|
||||
the previous day.
|
||||
|
||||
### Changed
|
||||
- All dashboard and Gotify notification timestamps now render in US
|
||||
Pacific time (`PDT`/`PST`, DST-aware) instead of UTC, e.g.
|
||||
`2026-06-14 03:45 PM PDT`. JSON API endpoints are unaffected and continue
|
||||
to return raw UTC ISO 8601 timestamps.
|
||||
|
||||
---
|
||||
|
||||
## [0.2.6] — 2026-06-14
|
||||
|
||||
### Changed
|
||||
- Gotify notifications reformatted as Markdown (`extras.client::display.contentType:
|
||||
text/markdown`) with richer context per message: error counts and
|
||||
thresholds, previous/current status, Docker error details for failed
|
||||
restarts, restart/cooldown info, and a formatted timestamp. The
|
||||
health-history digest is now a Markdown bulleted list with a header.
|
||||
- Dashboard timestamps (container "Last restart" / "Cooldown until", and
|
||||
the events timeline) now render as 12-hour time with AM/PM (e.g.
|
||||
`2026-06-14 03:45 PM UTC`) instead of raw ISO 8601.
|
||||
|
||||
### Fixed
|
||||
- "Last restart" no longer reverts to "Never" after Inspector itself
|
||||
restarts: on startup, with `database.enabled: true`, each container's
|
||||
most recent recorded restart (and any still-active cooldown) is restored
|
||||
from the event history database.
|
||||
|
||||
---
|
||||
|
||||
## [0.2.5] — 2026-06-14
|
||||
|
||||
### Added
|
||||
- Per-container identity colors on the dashboard: each container card gets
|
||||
a soft background tint and colored name from a 16-color auto-assigned
|
||||
palette, and the same color carries over to that container's chip on the
|
||||
events timeline. Override the auto-assigned color with an optional
|
||||
`color: "#rrggbb"` field per container in `config.yml`.
|
||||
|
||||
---
|
||||
|
||||
## [0.2.4] — 2026-06-14
|
||||
|
||||
### Fixed
|
||||
- Sign out: removed the redirect-if-authenticated on `GET /login` so sign
|
||||
out always reaches the login form, even with a cached browser Basic Auth
|
||||
session.
|
||||
- Mobile header: version/refresh text now hides on narrow screens, and
|
||||
sign-out renders as a small outlined button beside the brand.
|
||||
- Status page error bar: shows 100% green when a container has zero errors
|
||||
(previously rendered as an invisible 0-width bar) and a proportional red
|
||||
fill once errors are detected.
|
||||
- Container cards: hide the cooldown row when no cooldown is active, and
|
||||
show "Never" instead of "—" when a container has no restart history.
|
||||
- Events page: the "event history not enabled" message now includes the
|
||||
`database: enabled: true` config snippet needed to turn it on.
|
||||
|
||||
---
|
||||
|
||||
## [0.2.3] — 2026-06-14
|
||||
|
||||
### Changed
|
||||
- Replaced the browser-native Basic Auth prompt with a styled dark-theme
|
||||
login page backed by a session cookie (24h expiry); HTTP Basic Auth still
|
||||
works for API clients. Added a "Sign out" link, and made the dashboard
|
||||
header, stats grid, and card grid responsive on mobile.
|
||||
|
||||
---
|
||||
|
||||
## [0.2.2] — 2026-06-14
|
||||
|
||||
### Added
|
||||
- Dashboard pages now set a full favicon/app-icon set (favicon.ico,
|
||||
favicon.svg, favicon-96x96.png, apple-touch-icon.png, web app manifest
|
||||
icons, site.webmanifest) generated from the Inspector icon. Served from
|
||||
new `GET /static/<file>` endpoints alongside the existing icon route
|
||||
(Basic Auth required, same as every other dashboard route).
|
||||
|
||||
---
|
||||
|
||||
## [0.2.1] — 2026-06-14
|
||||
|
||||
### Changed
|
||||
- Web dashboard redesigned: dark "card grid" theme. Each container gets its
|
||||
own card (status badge, error-count progress bar, cooldown/last-restart
|
||||
details, maintenance badge and toggle button) instead of a plain table;
|
||||
the events page renders a colour-coded vertical timeline instead of a
|
||||
table. The header now shows the Inspector icon plus an "Inspector" /
|
||||
"Monitor. Restart. Report." wordmark set in
|
||||
[Google Sans Flex](https://fonts.google.com/specimen/Google+Sans+Flex)
|
||||
(loaded from Google Fonts; falls back to the system sans-serif if
|
||||
unavailable). New `GET /static/icon.png` endpoint (Basic Auth required,
|
||||
same as every other dashboard route) serves the icon from a new
|
||||
`src/inspector/assets/` package directory.
|
||||
|
||||
---
|
||||
|
||||
## [0.2.0] — 2026-06-14
|
||||
|
||||
### Added
|
||||
- Optional embedded web dashboard (`dashboard.enabled`, default `false`),
|
||||
protected by HTTP Basic Auth (`dashboard.username` / `dashboard.password`).
|
||||
If enabled with an empty password, Inspector logs an error and refuses to
|
||||
start the dashboard (fails closed). Runs in the same process as the
|
||||
monitoring loop — no extra container, zero new dependencies (stdlib
|
||||
`http.server`).
|
||||
- **Status page (`/`, JSON at `/api/status`)** — live per-container table:
|
||||
current status, error count vs. threshold, cooldown expiry, last restart
|
||||
time, and whether it's in a maintenance window. Auto-refreshes every
|
||||
30 seconds.
|
||||
- **Events page (`/events`, JSON at `/api/events`)** — last 100 rows from
|
||||
event history, if `database.enabled: true`.
|
||||
- **Maintenance toggle** — when `maintenance.enabled: true`, per-container
|
||||
"Enter/Exit maintenance" buttons `POST /maintenance/<name>/enter|exit`,
|
||||
touching/removing the same marker files described under "Ad-hoc marker
|
||||
files". Returns `409` if maintenance windows are disabled, `404` for an
|
||||
unknown container.
|
||||
New module `src/inspector/dashboard.py`, new config section `dashboard:`
|
||||
in `config.example.yml`.
|
||||
- `MaintenanceTracker.peek(name)` — read-only maintenance check (no
|
||||
transition logging, no eventstore writes) used by the dashboard.
|
||||
- `tests/test_dashboard.py` — unit tests covering the disabled no-op, the
|
||||
empty-password refusal, Basic Auth enforcement, the status page and API,
|
||||
the maintenance toggle (enter/exit, disabled-config 409, unknown-container
|
||||
404), and the events page/API with and without event history enabled.
|
||||
|
||||
---
|
||||
|
||||
## [0.1.11] — 2026-06-14
|
||||
|
||||
### Added
|
||||
- Optional maintenance windows (`maintenance.enabled`, default `false`).
|
||||
Two complementary mechanisms, both gated by this one flag:
|
||||
- **Recurring daily window** — containers with `maintenance_window: true`
|
||||
are "in maintenance" every day between `maintenance.start` and
|
||||
`maintenance.end` (default `00:00`–`06:00`, local time via `TZ`, may
|
||||
wrap past midnight).
|
||||
- **Ad-hoc marker files** — `touch`/`rm`
|
||||
`<maintenance.dir>/<container>` (default
|
||||
`/app/data/maintenance/<container>`) for unplanned maintenance. A
|
||||
marker older than `maintenance.max_hours` (default `8`) auto-expires.
|
||||
|
||||
While either mechanism is active for a container, Inspector is fully
|
||||
silent for it: no failure counting, restarts, notifications,
|
||||
status-change logging, or event recording. Only one-time
|
||||
"entering/exiting maintenance window" log lines and new
|
||||
`maintenance_enter`/`maintenance_exit` events (excluded from the
|
||||
health-history digest) mark the transitions. State (error counts,
|
||||
cooldowns, restart history) carries over unchanged across maintenance.
|
||||
New module `src/inspector/maintenance.py`, new config section
|
||||
`maintenance:` in `config.example.yml`, new per-container
|
||||
`maintenance_window` field.
|
||||
- `tests/test_maintenance.py` — unit tests covering the disabled no-op,
|
||||
marker-file activation and auto-expiry, the recurring window (including
|
||||
midnight wraparound), enter/exit transition logging and event recording,
|
||||
and safe handling of marker-check errors.
|
||||
|
||||
---
|
||||
|
||||
## [0.1.10] — 2026-06-14
|
||||
|
||||
### Added
|
||||
- Optional periodic health-history digest (`health_summary.enabled`,
|
||||
default `false`). When enabled, Inspector logs a per-container summary
|
||||
of notable activity (failures, restarts, restart-limit hits, recoveries,
|
||||
not-found events — status changes are excluded) since the last digest,
|
||||
every `health_summary.interval_hours` (default `24`). The first digest
|
||||
fires immediately on startup, covering the last `interval_hours`. The
|
||||
digest is always logged; it is pushed to Gotify (at
|
||||
`health_summary.priority`, default `3`) only if any container had
|
||||
activity in the window. Depends on `database.enabled: true` for real
|
||||
data — if `health_summary.enabled` is `true` but event history isn't
|
||||
enabled, Inspector logs a one-time startup warning and digests report
|
||||
"no issues" for every container. New module
|
||||
`src/inspector/health_summary.py`, new config section
|
||||
`health_summary:` in `config.example.yml`.
|
||||
- `EventStore.summarize(since)` — per-container counts of notable event
|
||||
types at/after a timestamp, and `EventStore.enabled` property.
|
||||
- `tests/test_health_summary.py` — unit tests covering the disabled
|
||||
no-op, the database-disabled warning, immediate first report with a
|
||||
capped window, interval gating, message formatting, and Gotify
|
||||
activity gating.
|
||||
|
||||
---
|
||||
|
||||
## [0.1.9] — 2026-06-13
|
||||
|
||||
### Added
|
||||
- Optional SQLite event history (`database.enabled`, default `false`).
|
||||
When enabled, Inspector persists notable per-container events (status
|
||||
changes, recoveries, failures, restarts, restart-limit hits,
|
||||
container-not-found) to a local SQLite database at `database.path`
|
||||
(default `/app/data/events.db`), surviving container restarts. Old
|
||||
events are pruned according to `database.retention_days` (default 30)
|
||||
on startup and once per 24h thereafter. The database is opened in WAL
|
||||
mode for future concurrent read access (e.g. a read-only dashboard).
|
||||
A database error is logged but never crashes Inspector. New module
|
||||
`src/inspector/eventstore.py`, new config section `database:` in
|
||||
`config.example.yml`, new `./data` volume mount in `compose.yml` /
|
||||
`compose.prod.yml`, new `/app/data` directory in `Dockerfile`.
|
||||
- `tests/test_eventstore.py` — unit tests covering the disabled no-op,
|
||||
row insertion, WAL mode, and retention pruning.
|
||||
|
||||
### Changed
|
||||
- `_check_one()` signature gains an `eventstore: EventStore` parameter
|
||||
(always passed; no-ops internally when `database.enabled` is `false`),
|
||||
mirroring the existing `notifier: GotifyNotifier` pattern.
|
||||
- README "What Inspector does not do" updated: live `ContainerState` is
|
||||
still in-memory and resets on restart, but event *history* can now
|
||||
persist via the new opt-in SQLite database.
|
||||
|
||||
---
|
||||
|
||||
## [0.1.8] — 2026-06-13
|
||||
|
||||
### Changed
|
||||
- Log format overhauled for readability: container names are now
|
||||
left-padded into a fixed-width column, the redundant logger-name field
|
||||
(`inspector:`) was dropped from both the file and console formatters, and
|
||||
a blank line now brackets each failure → restart → recovery "incident" so
|
||||
it stands out as a block. Message wording adjusted accordingly (e.g.
|
||||
`gotify Restarting due to 3 consecutive UNHEALTHY checks` instead of
|
||||
`Restarting gotify due to 3 consecutive UNHEALTHY checks`).
|
||||
|
||||
---
|
||||
|
||||
## [0.1.7] — 2026-06-13
|
||||
|
||||
### Fixed
|
||||
- Docker socket permission errors at startup no longer crash with a raw
|
||||
Python traceback. `run_loop()` now catches the connection failure and
|
||||
logs an actionable message pointing at `DOCKER_GID`
|
||||
(`stat -c '%g' /var/run/docker.sock`).
|
||||
|
||||
---
|
||||
|
||||
## [0.1.6] — 2026-06-13
|
||||
|
||||
### Added
|
||||
- `tests/test_config.py` — 4 unit tests covering config loading, defaults,
|
||||
and validation (non-mapping containers rejected, unknown `restart_on`
|
||||
statuses rejected).
|
||||
- `tests/test_health.py` — 8 unit tests covering all Docker status
|
||||
classifications (HEALTHY, UNHEALTHY, STARTING, NO HEALTH CHECK,
|
||||
NOT RUNNING, UNKNOWN, missing state, ERROR constant).
|
||||
- `tests/test_state.py` — 10 unit tests covering failure counting, restart
|
||||
decision logic, and the full per-container state machine (dry-run,
|
||||
grace period, cooldown, hourly restart limit, recovery).
|
||||
|
||||
### Changed
|
||||
- CI workflow now runs the full unit test suite
|
||||
(`python3 -m unittest discover -s tests -v`) before building the image.
|
||||
|
||||
---
|
||||
|
||||
## [0.1.5] — 2026-06-13
|
||||
|
||||
### Changed
|
||||
- CI and release workflows reverted to `runs-on: ubuntu-latest` (the
|
||||
self-hosted runner approach introduced in v0.1.3 was abandoned — the
|
||||
Gitea-hosted runner handles builds reliably).
|
||||
- Gitea runner setup guide (`docs/gitea_runner_setup.md`,
|
||||
`ops/gitea-runner/`) updated with label-mismatch troubleshooting and
|
||||
clearer registration instructions.
|
||||
|
||||
### Added
|
||||
- README documents the verified release baseline (v0.1.5) and the
|
||||
step-by-step release flow.
|
||||
|
||||
---
|
||||
|
||||
## [0.1.4] — 2026-06-13
|
||||
|
||||
### Added
|
||||
- Inspector logo and wordmark assets: `logo/inspector_icon.{png,svg}`,
|
||||
`logo/inspector_wordmark.{png,svg}`, `logo/inspector_icon_chip.{png,svg}`.
|
||||
- README header now displays the Inspector wordmark image.
|
||||
|
||||
### Changed
|
||||
- Previous logo files moved to `logo/archive/`.
|
||||
- `.gitignore` — added `act_data/` (local `act` runner data directory).
|
||||
|
||||
---
|
||||
|
||||
## [0.1.3] — 2026-06-13
|
||||
|
||||
### Changed
|
||||
- CI and release workflows switched to `runs-on: self-hosted`.
|
||||
- CI simplified to use `docker build` directly (removed
|
||||
`docker/setup-buildx-action` dependency).
|
||||
|
||||
---
|
||||
|
||||
## [0.1.2] — 2026-06-13
|
||||
|
||||
### Changed
|
||||
- Release workflow rewritten to use native `docker login` / `docker push`
|
||||
shell commands, replacing the `docker/build-push-action` third-party
|
||||
action.
|
||||
|
||||
### Added
|
||||
- `workflow_dispatch` trigger on the release workflow — enables manual
|
||||
image builds without a tag push (useful for testing the CI publish path).
|
||||
|
||||
---
|
||||
|
||||
## [0.1.1] — 2026-06-13
|
||||
|
||||
### Fixed
|
||||
- Release workflow secret names corrected:
|
||||
`GITEA_REGISTRY_USER` → `REGISTRY_USER`,
|
||||
`GITEA_REGISTRY_TOKEN` → `REGISTRY_TOKEN`.
|
||||
The v0.1.0 release workflow referenced the wrong secret names, causing
|
||||
the publish step to fail.
|
||||
|
||||
---
|
||||
|
||||
## [0.1.0] — 2026-06-13
|
||||
|
||||
### Added
|
||||
- Initial Inspector v1: self-hosted Python Docker container health watcher.
|
||||
- Multi-container support via a single `config.yml` — replaces the old
|
||||
pattern of one script + one systemd service per watched container.
|
||||
- Per-container state machine with independent failure counters.
|
||||
- Health status classification: HEALTHY, UNHEALTHY, STARTING,
|
||||
NO HEALTH CHECK, NOT RUNNING, CONTAINER NOT FOUND, ERROR, UNKNOWN.
|
||||
- Restart logic: fires after `max_error_count` consecutive failures.
|
||||
- Cooldown: post-restart window that blocks further restarts.
|
||||
- Grace period: post-restart window that suppresses failure counting while
|
||||
a container boots.
|
||||
- Hourly restart cap (`max_restarts_per_hour`) to break endless loops;
|
||||
fires a critical Gotify alert when hit.
|
||||
- Optional Gotify notifications on state transitions (not per-loop).
|
||||
- Dry-run mode: logs decisions without executing restarts.
|
||||
- Rotating log file (`TimedRotatingFileHandler`, midnight, 30-day retention)
|
||||
with colored console output and plain-text file output.
|
||||
- `Dockerfile` — `python:3.12-slim` base, non-root `inspector` user
|
||||
(UID/GID 1000).
|
||||
- `compose.yml` — dev compose: builds from source, mounts Docker socket.
|
||||
- `compose.prod.yml` — prod compose: pulls published image from Gitea
|
||||
registry, no local build.
|
||||
- `requirements.txt` — pinned: `docker==7.1.0`, `PyYAML==6.0.2`,
|
||||
`requests==2.32.3`, `colorlog==6.9.0`.
|
||||
- `Makefile` with `build`, `smoke`, `compose-config`, `tag`, `push-local`,
|
||||
`release`, `clean`, `version`, `help` targets.
|
||||
- Gitea Actions CI (`ci.yml`): byte-compile + Docker build on every push.
|
||||
- Gitea Actions release (`release.yml`): build and publish four image tags
|
||||
on `v*.*.*` tag push.
|
||||
- `scripts/bump_version.sh` — updates `__version__` in `src/inspector/__init__.py`.
|
||||
- `scripts/smoke_test.sh` — end-to-end dry-run test against the live
|
||||
Docker socket.
|
||||
- README with full feature documentation, configuration reference,
|
||||
Portainer deployment guide, and troubleshooting table.
|
||||
Executable
+27
@@ -0,0 +1,27 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt /app/requirements.txt
|
||||
RUN pip install --no-cache-dir -r /app/requirements.txt
|
||||
|
||||
COPY src/ /app/src/
|
||||
RUN chmod -R a+rX /app/src
|
||||
|
||||
ENV PYTHONPATH=/app/src
|
||||
|
||||
ARG INSPECTOR_UID=1000
|
||||
ARG INSPECTOR_GID=1000
|
||||
RUN groupadd --gid ${INSPECTOR_GID} inspector \
|
||||
&& useradd --uid ${INSPECTOR_UID} --gid inspector --home-dir /app --shell /usr/sbin/nologin inspector \
|
||||
&& mkdir -p /app/logs /app/data \
|
||||
&& chown -R inspector:inspector /app/logs /app/data
|
||||
|
||||
USER inspector
|
||||
|
||||
CMD ["python", "-m", "inspector.main", "--config", "/app/config.yml"]
|
||||
Executable
+70
@@ -0,0 +1,70 @@
|
||||
# Install
|
||||
|
||||
Inspector watches Docker containers, restarts the ones you tell it to,
|
||||
and notifies you via Gotify and/or Pushover. All configuration lives in
|
||||
a single `config.yml` that you can edit by hand or from the built-in web
|
||||
settings page.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Docker Engine 24+ and `docker compose` v2.
|
||||
- A machine that can reach `/var/run/docker.sock` (any Docker host).
|
||||
- Optional: a Gotify server and/or a Pushover account for notifications.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
# 1. Grab the compose file and example config
|
||||
mkdir -p ~/docker/inspector && cd ~/docker/inspector
|
||||
curl -L -o compose.yml https://gitea.euronvault.com/euronvault/inspector/raw/branch/main/compose.yml
|
||||
curl -L -o config.yml https://gitea.euronvault.com/euronvault/inspector/raw/branch/main/config.example.yml
|
||||
|
||||
# 2. Edit config.yml — at minimum, list the containers you want watched
|
||||
# under the `containers:` key.
|
||||
$EDITOR config.yml
|
||||
|
||||
# 3. Set your Docker group GID so the container can read the socket.
|
||||
echo "DOCKER_GID=$(getent group docker | cut -d: -f3)" > .env
|
||||
|
||||
# 4. Pull and run
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
docker compose logs -f inspector
|
||||
```
|
||||
|
||||
If `dashboard.enabled: true` in your config, visit
|
||||
`http://<host>:8080/` after startup to see the dashboard, and
|
||||
`http://<host>:8080/settings` to edit the config from the browser.
|
||||
|
||||
## Pinning a version
|
||||
|
||||
Prefer an immutable tag over `latest` in production:
|
||||
|
||||
```bash
|
||||
# .env
|
||||
INSPECTOR_TAG=0.7.0
|
||||
```
|
||||
|
||||
Images are published at `gitea.euronvault.com/euronvault/inspector`.
|
||||
|
||||
## Upgrading
|
||||
|
||||
```bash
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Your `config.yml` is persisted outside the container, so upgrades don't
|
||||
touch it. The settings page keeps a `.bak` of the previous file next to
|
||||
`config.yml` on every save (when the directory is bind-mounted; see the
|
||||
README for the Docker caveat).
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
rm -rf logs data config.yml compose.yml .env
|
||||
```
|
||||
|
||||
Inspector doesn't store anything outside the directory it was started
|
||||
from.
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Europium 63
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,139 @@
|
||||
# Inspector
|
||||
|
||||
Inspector is a small, opinionated Docker-watchdog. It polls the Docker
|
||||
daemon, applies per-container restart rules, pings Gotify and/or
|
||||
Pushover when things go wrong, and gives you a dashboard that shows
|
||||
what's happening right now and what happened earlier today.
|
||||
|
||||
Built for home-lab and small-fleet operators who want "it just works"
|
||||
container recovery without standing up a full observability stack.
|
||||
|
||||
## What Inspector does
|
||||
|
||||
- Polls each configured container on a fixed interval and classifies it:
|
||||
`HEALTHY`, `UNHEALTHY`, `STARTING`, `NOT RUNNING`, `ERROR`,
|
||||
`UNREACHABLE`, `CONTAINER NOT FOUND`, `NO HEALTH CHECK`, `UNKNOWN`.
|
||||
- Restarts a container when it crosses a configured error threshold —
|
||||
with cooldowns, grace periods, and an hourly restart cap so a flapping
|
||||
container doesn't DDoS your host.
|
||||
- Opt-in HTTP reachability checks per container (useful for VPN-routed
|
||||
containers whose tunnel drops while Docker still reports "healthy").
|
||||
- Opt-in Uptime Kuma integration as a second reachability source.
|
||||
- Advanced recovery via Portainer stack redeploy for containers whose
|
||||
network namespace is tied to a VPN sidecar.
|
||||
- Sends notifications on transitions, not on every poll — one line when
|
||||
something breaks, one line when it recovers.
|
||||
- Optional SQLite event history + a daily/weekly digest.
|
||||
- Scheduled maintenance windows (don't act during 2–6 AM).
|
||||
- Manual notification mute per container, from the dashboard.
|
||||
- Embedded web dashboard + events timeline + **web settings page** for
|
||||
editing `config.yml` without SSH.
|
||||
|
||||
## What Inspector is not
|
||||
|
||||
- Not a metrics system. It doesn't scrape Prometheus or push
|
||||
dashboards; use Grafana / Netdata / your real monitoring stack for
|
||||
that.
|
||||
- Not an orchestrator. It only acts on containers you've explicitly
|
||||
listed in `config.yml`.
|
||||
- Not multi-host. One Inspector per Docker socket.
|
||||
|
||||
## Install
|
||||
|
||||
See [`INSTALL.md`](./INSTALL.md). TL;DR:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/docker/inspector && cd ~/docker/inspector
|
||||
curl -L -o compose.yml https://gitea.euronvault.com/euronvault/inspector/raw/branch/main/compose.yml
|
||||
curl -L -o config.yml https://gitea.euronvault.com/euronvault/inspector/raw/branch/main/config.example.yml
|
||||
$EDITOR config.yml
|
||||
echo "DOCKER_GID=$(getent group docker | cut -d: -f3)" > .env
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Everything lives in `config.yml` next to the compose file. Start from
|
||||
[`config.example.yml`](./config.example.yml) — every key is commented.
|
||||
Highlights:
|
||||
|
||||
- `inspector.interval_seconds` — poll cadence.
|
||||
- `inspector.dry_run` — classify + log, never restart.
|
||||
- `notifications.gotify` / `notifications.pushover` — enable either,
|
||||
both, or neither.
|
||||
- `containers.<name>.max_error_count` / `cooldown_seconds` /
|
||||
`grace_period_seconds` / `max_restarts_per_hour` — the restart knobs.
|
||||
- `containers.<name>.restart_on` — which statuses trigger a restart
|
||||
(leave empty for observe-only).
|
||||
- `containers.<name>.http_check` — opt-in HTTP ping.
|
||||
- `containers.<name>.recovery_action` — `restart_container` (default)
|
||||
or the advanced Portainer-based options.
|
||||
|
||||
### Dashboard and settings page
|
||||
|
||||
If `dashboard.enabled: true`, Inspector exposes a small web UI on
|
||||
`:8080`:
|
||||
|
||||
- `/` — container cards with current status, error bar, phase
|
||||
(cooldown / grace / failing / observe only / maintenance), and
|
||||
maintenance / mute toggles.
|
||||
- `/events` — per-day timeline of status changes, restarts, recoveries,
|
||||
and failures (requires `database.enabled: true`).
|
||||
- `/settings` — forms for every top-level config section plus one card
|
||||
per container. Saves preserve comments and key order in `config.yml`
|
||||
and keep a `.bak`. Fields that take effect on the next loop iteration
|
||||
(per-container thresholds, `dry_run`, `heartbeat`, `interval_seconds`)
|
||||
hot-apply; fields bound at startup are labeled **requires restart**.
|
||||
|
||||
Note on `.bak` inside Docker: when `config.yml` is bind-mounted as a
|
||||
single file (the compose default), the `.bak` lives in the container's
|
||||
overlay fs — the live file still updates cleanly, but the backup is
|
||||
lost when the container is removed. To keep the `.bak` on the host,
|
||||
bind-mount the directory instead of the single file.
|
||||
|
||||
### Notifications
|
||||
|
||||
Both channels are independent. If both are enabled, every alert goes to
|
||||
both. Inspector sends on transitions:
|
||||
|
||||
```
|
||||
🚨 first failure threshold crossed
|
||||
🔁 container restart triggered
|
||||
✅ restart / start completed
|
||||
✅ container recovered
|
||||
🛑 restart failed / hourly limit exceeded
|
||||
❓ configured container not found
|
||||
```
|
||||
|
||||
Gotify messages are Markdown; Pushover doesn't render Markdown so the
|
||||
`**bold**` passthrough is cosmetic — bodies are identical.
|
||||
|
||||
Pushover priorities map from Inspector's Gotify-style scale (default 5)
|
||||
to Pushover's `-2..2` range: `<=2 → -1`, `3..4 → 0`, `5..6 → 1`,
|
||||
`>=7 → 2`. Priority 2 (emergency) ships default `retry` / `expire` and
|
||||
your Pushover client prompts for acknowledgement.
|
||||
|
||||
## Docker socket security
|
||||
|
||||
Inspector talks to `/var/run/docker.sock`. That socket is effectively
|
||||
root on your host: anything that can write to it can start, stop,
|
||||
inspect, and (through bind mounts) read any file as root. Inspector
|
||||
mounts it read-write because it needs to restart containers.
|
||||
|
||||
Follow the usual home-lab precautions:
|
||||
|
||||
- Don't publish the dashboard port on an untrusted network. Use a
|
||||
reverse proxy with auth, a tailnet, or SSH tunneling.
|
||||
- Set a strong `dashboard.password`.
|
||||
- Store Portainer / Gotify / Pushover tokens in the config file on a
|
||||
host-only path with permissions `600`.
|
||||
|
||||
## Versioning
|
||||
|
||||
Semantic versioning (`MAJOR.MINOR.PATCH`). Image tags are published for
|
||||
every release; `latest` tracks the newest tag. Pin a specific version
|
||||
in production — see `INSTALL.md`.
|
||||
|
||||
## License
|
||||
|
||||
[MIT](./LICENSE).
|
||||
Executable
+29
@@ -0,0 +1,29 @@
|
||||
# Inspector compose file for buddies / self-hosters.
|
||||
# Pulls the published image from gitea.euronvault.com.
|
||||
#
|
||||
# Before `docker compose up -d`:
|
||||
# 1. Copy config.example.yml -> config.yml and edit it.
|
||||
# 2. Set DOCKER_GID to your host's docker group GID:
|
||||
# getent group docker | cut -d: -f3
|
||||
# Export it in your shell or put it in a .env file next to this file.
|
||||
# 3. Optionally set INSPECTOR_TAG to pin a version
|
||||
# (default: latest). Prefer a pinned X.Y.Z for production.
|
||||
services:
|
||||
inspector:
|
||||
image: gitea.euronvault.com/euronvault/inspector:${INSPECTOR_TAG:-latest}
|
||||
container_name: inspector
|
||||
restart: unless-stopped
|
||||
# GID of the docker group on the host so the non-root user inside the
|
||||
# container can read /var/run/docker.sock. Required.
|
||||
group_add:
|
||||
- "${DOCKER_GID:-999}"
|
||||
ports:
|
||||
# Only published if dashboard.enabled is true in config.yml.
|
||||
- "${INSPECTOR_PORT:-8080}:8080"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- ./config.yml:/app/config.yml
|
||||
- ./logs:/app/logs
|
||||
- ./data:/app/data
|
||||
environment:
|
||||
- TZ=${TZ:-UTC}
|
||||
Executable
+137
@@ -0,0 +1,137 @@
|
||||
# Inspector example configuration.
|
||||
# Copy to config.yml, edit, and mount into the container at /app/config.yml.
|
||||
|
||||
inspector:
|
||||
interval_seconds: 60
|
||||
log_level: INFO
|
||||
dry_run: false
|
||||
log_dir: /app/logs
|
||||
log_filename: inspector.log
|
||||
log_retention_days: 30
|
||||
heartbeat: false
|
||||
|
||||
notifications:
|
||||
# Gotify: https://gotify.net
|
||||
gotify:
|
||||
enabled: false
|
||||
url: ""
|
||||
token: ""
|
||||
priority: 5
|
||||
|
||||
# Pushover: https://pushover.net
|
||||
# Independent of Gotify — enable either, both, or neither. If both are
|
||||
# enabled, every alert is sent to both. Inspector's priority is an
|
||||
# integer on a Gotify-style scale (default 5) and is mapped to
|
||||
# Pushover's -2..2 range at send time: <=2 -> -1, 3..4 -> 0,
|
||||
# 5..6 -> 1, >=7 -> 2.
|
||||
pushover:
|
||||
enabled: false
|
||||
token: "" # Pushover application API token
|
||||
user_key: "" # Pushover user or group key
|
||||
priority: 5
|
||||
device: "" # optional target device; blank = all devices
|
||||
sound: "" # optional sound name; blank = user default
|
||||
|
||||
# Persist a durable per-container event history to a local SQLite DB.
|
||||
# Enables the /events page and the Health summary digest.
|
||||
database:
|
||||
enabled: false
|
||||
path: /app/data/events.db
|
||||
retention_days: 30
|
||||
|
||||
# Periodic digest of per-container events, delivered through the enabled
|
||||
# notification channels. Requires database.enabled: true to have data.
|
||||
health_summary:
|
||||
enabled: false
|
||||
interval_hours: 24
|
||||
priority: 3
|
||||
|
||||
# Recurring daily maintenance window. Containers that opt in with
|
||||
# `maintenance_window: true` are not acted on during this window.
|
||||
maintenance:
|
||||
enabled: false
|
||||
start: "00:00"
|
||||
end: "06:00"
|
||||
dir: /app/data/maintenance
|
||||
max_hours: 8
|
||||
|
||||
# Manual notification muting per container (toggled from the dashboard).
|
||||
notification_mute:
|
||||
enabled: false
|
||||
dir: /app/data/muted
|
||||
default_hours: 12
|
||||
|
||||
# Embedded web dashboard. Enable with a non-empty password.
|
||||
# Visit http://<host>:<port>/ for status, /events for history,
|
||||
# /settings to edit this file from the browser.
|
||||
dashboard:
|
||||
enabled: false
|
||||
host: 0.0.0.0
|
||||
port: 8080
|
||||
username: admin
|
||||
password: ""
|
||||
|
||||
# Optional external integrations.
|
||||
integrations:
|
||||
# Uptime Kuma: optionally use a public status page as an extra
|
||||
# reachability check per container.
|
||||
uptime_kuma:
|
||||
enabled: false
|
||||
url: "https://status.example.com"
|
||||
status_page_slug: "default"
|
||||
|
||||
# Portainer API integration: used by the advanced recovery actions
|
||||
# `portainer_stack_redeploy` and `restart_then_redeploy`, which can
|
||||
# recover containers whose network namespace is tied to a VPN sidecar.
|
||||
# Leave disabled unless you need it.
|
||||
#
|
||||
# Token options (pick one per environment):
|
||||
# api_token_env: "ENV_VAR_NAME" — reads token from that env var at
|
||||
# runtime (preferred; not stored in config)
|
||||
# api_token: "ptr_..." — embed token directly in this file
|
||||
portainer:
|
||||
enabled: false
|
||||
default_environment: local_docker
|
||||
stack_redeploy_cooldown_seconds: 300
|
||||
post_redeploy_wait_seconds: 90
|
||||
environments:
|
||||
local_docker:
|
||||
base_url: "http://portainer.example.local:9000"
|
||||
api_token_env: "PORTAINER_API_TOKEN"
|
||||
endpoint_id: 1
|
||||
verify_ssl: false
|
||||
|
||||
containers:
|
||||
# Minimal entry: Inspector only acts on containers listed here.
|
||||
# Each key is the Docker container name on your host.
|
||||
example-web:
|
||||
enabled: true
|
||||
max_error_count: 5
|
||||
cooldown_seconds: 300
|
||||
grace_period_seconds: 120
|
||||
max_restarts_per_hour: 3
|
||||
restart_on:
|
||||
- UNHEALTHY
|
||||
- NOT RUNNING
|
||||
- ERROR
|
||||
|
||||
# Example with an opt-in HTTP reachability check. Useful when Docker
|
||||
# reports a container as healthy but it's not actually serving
|
||||
# requests (common with VPN-routed containers whose tunnel drops).
|
||||
example-api:
|
||||
enabled: true
|
||||
max_error_count: 5
|
||||
cooldown_seconds: 300
|
||||
grace_period_seconds: 120
|
||||
max_restarts_per_hour: 3
|
||||
color: "#60a5fa"
|
||||
restart_on:
|
||||
- UNHEALTHY
|
||||
- NOT RUNNING
|
||||
- ERROR
|
||||
- UNREACHABLE
|
||||
http_check:
|
||||
enabled: true
|
||||
url: "http://example-api:8080/health"
|
||||
timeout_seconds: 5
|
||||
expected_status: 200
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
docker==7.1.0
|
||||
PyYAML==6.0.2
|
||||
requests==2.32.3
|
||||
colorlog==6.9.0
|
||||
ruamel.yaml==0.18.6
|
||||
Executable
+1
@@ -0,0 +1 @@
|
||||
__version__ = "0.7.0"
|
||||
Executable
BIN
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Executable
BIN
Binary file not shown.
|
After Width: | Height: | Size: 7.6 KiB |
Executable
BIN
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
Executable
+1
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" xmlns:xlink="http://www.w3.org/1999/xlink" width="512" height="430" viewBox="0 0 512 430"><metadata><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"><rdf:Description><dc:creator>RealFaviconGenerator</dc:creator><dc:source>https://realfavicongenerator.net</dc:source></rdf:Description></rdf:RDF></metadata><image width="512" height="430" xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAgAAAAGuCAYAAADmhr75AAABgWlDQ1BzUkdCIElFQzYxOTY2LTIuMQAAKJF1kblLQ0EQh7/EI6LRCFpYWASJVh4xQtDGIuIFapFE8GqSl0vI8XgvQYKtYCsoiDZehf4F2grWgqAogliLpaKNhue8REgQM8vOfvvbmWF3FqzBpJLSa92QSmc1/6TPubC45LS9Uk8LViwMhBRdnQ1MBKlqn/cSJ3bbZ9aqHvevNUWiugKWBuFRRdWywlPCM2tZ1eQd4XYlEYoInwn3anJB4TtTD5f4xeR4ib9N1oL+MbC2CjvjFRyuYCWhpYTl5bhSyZzyex/zJfZoej4ga5fMTnT8TOLDyTTjjOFlkBHxXvrw0C87quS7i/lzZCRXEa+SR2OVOAmy9Iqak+pRWWOiR2UkyZv9/9tXPTbkKVW3+6Du2TDeu8G2DYUtw/g6MozCMdQ8wWW6nJ85hOEP0bfKmusAHBtwflXWwrtwsQkdj2pICxWlGpnWWAzeTqF5EdpuoHG51LPfc04eILguX3UNe/vQI/GOlR82u2fQqlFYOgAAAAlwSFlzAAALEwAACxMBAJqcGAAAIABJREFUeJzs3Xec1MX5wPHP7F6jHiCdo0vnaNJR7L1iEhN7icb8EpNfTGKixpgYNWrUGKMmxv5LjA1RFLAXRKUdVY7epfd2wJXdnd8fu8vdwfWb+bZ93q8XL8OVmclxu8/znZlnBoQQQgghhBBCCCGEEEIIIYQQQgghhBBCCOFLyu0BCCHM01pnA03KfOiAUmqfW+MRQniPJABC+JjWOgT0AkYBI4E+ib+3quDLtwMrgKXATGA6sEIpFXNmtEIIL5EEQAgf0Vo3BoYTD/ijE/9tXo8m9wAziCcD04E8pVRBfccphPA+SQCE8CittQK6UBrsRwMDgZDFbqPAN5QmBNOB9UopbbFPIYQLJAEQwiO01pnAEEqD/WigrauDittCPBFIzhTMU0oVuTskIUR9SQIghEu01u0o/3R/ApDh6qBqphiYQ+kMwQyl1FZ3hySEqC1JAIRwgNY6Dcil/NN9FzfHZNhayi8b5CulIu4OSQhRFUkAhLBAa92C+K78ZLAfDjRydVDOOgjMojQhmKmU2uPukIQQZUkCIEQ9lSnFK/t039vVQXnTEspXHEgJohAukgRAiFoqU4qXDPajgGauDsqfdhNPCJJJwWyl1EF3hyRE6pAEQIgqlCnFK/t0PwC7pXipKgospHzFgZQgCmGJJABClKG1zuLYUrw2rg4qtSVLEJN/5ksJohBmSAIgUprWuj3lS/GG4I9SvFRVBMxFShCFqDdJAETKSJTiDaD8031nVwclTJASRCHqQBIAEVha6+M4thSvoauDEk4oIF6CmNxHICWIQlRAEgARCIlSvN6Uf7rv5eqghJcsofzmwuWyuVCkOkkAhC9prZtQvhRvJFKKJ2ouWYJY9hZEKUEUKUUSAOF5iVK8rpSvu5dSPGFS2RLE5J9vZZZABJkkAMJzEqV4J1A+4EspnnCalCCKQJMEQLguUYpXNtifAKS7OighjlVE/BbE5NKBlCAKX5MEQDhKa51O+VK8UUgpnvCvNZTfXCgliMI3JAEQViVK8ZIH7YxCSvFEsCVLEMvegrjX3SEJUTFJAIQxiVK8PpR/updSPJHqkiWIZW9BlM2FwnWSAIg6S5TijaA02I8Csl0dlKAkCsu3xMjfGGPtjhhdW4XonxOiV7sQ6WG3RyeQEkThEZIAiBpJlOJ1o/zTfS5Siue6XQWa/I0xFm2IB/3lW2KURI/9uvQw9G4Xon/HELk58aSgRWN5C/CAKLCA8kmBlCAK6+TVLyqktW5AaSlecg2/tauDEkRjsHp7rFzA37K37nGiXTNFbpmEoFvrEGFJ6bxgM+U3F0oJojBOEgABgNa6A+WD/RCkFM91+w9rFm+Kkb8hxqKNMZZsilFYYq+/BhnQt0M8GcjNCdEvJ0STLHmb8IBkCeKRpEAptc3dIQm/k1d2CkqU4g2kfMDv5OqgBDEN3+7SiWAfJX9jjPU73Z8F7tJKkZsTpn9ilqBTSyVvHN6QLEEsewtiBYs/QlRMXscpQGvdktJNeslb8Rq4OijB4WJYsrn06X7xxhgHCt0P+NVp2kAdSQZyO4bo0z5ElswVeYGUIIpakQQgYBKleH0p/3Tf09VBCTSwda9mUSLY52+MsXpbjJj34321QgqObxtfMsjNiW8ybJMtswQeISWIolLyGvU5rXVT4qV4yWA/EinFc11xBJZvLX26z98QY/fB1Hnfbdmk/CxBz7ZSgugRu4jvIUhuLpQSxBQmCYCPJErxulMa7EcTL8WTf0eX7SrQR3blL9oQY8XWikvxUlV6GHq3L91cKCWInpEsQUzOEMxAShBThrwCPeyoUrzkn1auDkrES/G2xZ/sk2v39SnFS1Xtm6sjSwb9O4To3iZESN6RvCBZglj2FsRid4ckbJCXm4dorXMo/3Q/GCnFc92+w5rFiXX7fAdK8VJVsgQxN7Fs0K9DiMZSgugFZUsQk7cgSgliAMiryyWJUrxBlA/4HV0dlIiX4u3UR8rwvFKKl6rKliDmdgzR8TjZXOgRqyl/cqGUIPqQvJYcUqYULxnshyGleK47VAxLNpWu3S/eFKPAB6V4qUpKED2rAJhJaVIgJYg+IAmABUeV4iX/9HB1UAINbNmjj+zKz98UnFK8VBVS0KNtmc2FUoLoFZrSEsRkUiAliB4jrxMDypTiJYP9SKCpq4MS8VK8LaV196lWipeqWjYp3VyYmxOih5QgekWyBLHsLYiH3B1SapMEoJbKlOKVfbrvj/wsXbfzgD6yK3/RhviteJGY26MSbkuWIB7ZXJgTokUjebl6wNEliNOBDTJL4Bx5FVQjUYo3lPIBv6WrgxJEypTiJZ/ut+6T9w1RM2VLEHMTtyBKCaInSAmig+RX/iha6wzgNOAcSkvx0lwdlDhSipc8bGfpZinFE+ZICaJnFQF5xJOBD4AvlVIRd4cUHPIbzpGSvAuA7wHnI+v3roppWL+zdFf+og0xvt0lT/fCWV1blSYDUoLoGbuAd4HxwEdSelg/Kf37nDh456bEn3YuDydlHVOKtzFGQZEEfOEt2Q0U/RIzBP1zpATRA9YD/wKeV0ptd3swfpSSCYDWehRwG3ARIPuDHaSBzXv0kXX7RRtjrNkupXjCf5IliGX3ErSWEkQ3lABvAg8rpea7PRg/SanfVa31EOBe4Dy3x5IqihKlePll1u/3SCmeCCgpQXTdBOAPSqnFbg/ED1IiAUjU6T8FXOX2WIJuxwEdP2QncVHOCinFEyksPQx92peeXCgliI7QwLPArXLOQNUC/5uotR5EfMPI8W6PJWgiMVi1LVZuOn+blOIJUaUOzVX8BsQcKUG0bDFwmVJqidsD8apA/9pprW8A/gFkuj2WINh3SB/Zlb9oY4ylm2IUSUGOEPXSMFmCmEgKpATRqEPAjUqpV90eiBcF8rcscVrfPcDv3R6LX5UtxUtO52+QUjwhrFNAl1alCUFuTogcKUGsr98S3yQob2JlBO53KlHT/y/gerfH4icHi0pL8fI3SimeEF5StgQxNydEbylBrIsniO8LkLMDEgKVAGitGxIvBznX7bF4mQY27U6crJf4s2ZbDAn3QvhDOAQ92sRnCKQEsVYmAFcqpYrcHogXBOb3RWudSfyEqLPcHovXJEvxjkznb4ix95CEeyGCpFUTdSQZ6C8liFV5B/ieUirlDxMPRAKQmPYfD1zs9li8YPv+0qf7/A0xVmyVUjwhUk1GGvRuV7qXoH9OiOZSgpj0CnBNqi8HBOK3QWv9LHCj2+NwQ7IUr+zT/fb98nQvhDhWsgQxN7G5sGtqlyA+rpT6hduDcJPv/+m11pcAb7s9DqfsO6SP7MrP3xC/FU9K8YQQdSEliJytlPrI7UG4xdf/0lrrlsQPe2jt9lhsiGlYt6O0DG/Rhhgbd8vTvRDCDgV0bV26ZJACJYgbgVyl1F63B+IGX/+7aq1fBX7g9jhMSZbixZ/uoyzeFOOg7FUVQrgou0H5zYUBLEF8XimVkkvIvk0AtNYnAl+6PY66Spbilb0kZ812KcUTQnhbsgTxyEFFHUO0burbUJI0VCk11+1BOM2X/2pa6xAwExjm9lhqqigCyzaXvxVPSvGEEEFQrgSxY4gebXxXgvgFcGqqnRTo1wTgSuBlt8dRle37yz/dr9gaIyqleMIhyd3eXVuGWJvYR7JpT0q9twkX+bQEcZxSaqLbg3CS5/9Fjqa1zgJWAjlujyUpEoOVW8s/3UspnnBK8srZ6t5sdx9MnA+R+B1dujlGSUpXQQsn5bRQ5TYXerAEcRXQN5UOCPLWj78GtNY/In7Wv2v2Hir/dL9MSvGEg1o2UUfOhK/PdGtJtEzimvh93nlAElfhjIYZxO83yAnRL/GncabrIekqpdR/3R6EU1z/adeG1joMLAV6ONlvTMPMVVE+XxqVUjzhqJCCnm3LP93bOvLine truncated
|
||||
|
After Width: | Height: | Size: 34 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Executable
+21
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"name": "Inspector",
|
||||
"short_name": "Inspector",
|
||||
"icons": [
|
||||
{
|
||||
"src": "/static/web-app-manifest-192x192.png",
|
||||
"sizes": "192x192",
|
||||
"type": "image/png",
|
||||
"purpose": "maskable"
|
||||
},
|
||||
{
|
||||
"src": "/static/web-app-manifest-512x512.png",
|
||||
"sizes": "512x512",
|
||||
"type": "image/png",
|
||||
"purpose": "maskable"
|
||||
}
|
||||
],
|
||||
"theme_color": "#0b1220",
|
||||
"background_color": "#0b1220",
|
||||
"display": "standalone"
|
||||
}
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
Executable
+502
@@ -0,0 +1,502 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import os
|
||||
import shutil
|
||||
import threading
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
from ruamel.yaml import YAML
|
||||
|
||||
|
||||
VALID_STATUSES = {
|
||||
"HEALTHY",
|
||||
"UNHEALTHY",
|
||||
"STARTING",
|
||||
"NO HEALTH CHECK",
|
||||
"NOT RUNNING",
|
||||
"CONTAINER NOT FOUND",
|
||||
"ERROR",
|
||||
"UNKNOWN",
|
||||
"UNREACHABLE",
|
||||
}
|
||||
|
||||
VALID_RECOVERY_ACTIONS = {
|
||||
"restart_container",
|
||||
"start_container",
|
||||
"portainer_stack_redeploy",
|
||||
"restart_then_redeploy",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InspectorConfig:
|
||||
interval_seconds: int = 60
|
||||
log_level: str = "INFO"
|
||||
dry_run: bool = False
|
||||
log_dir: str = "/app/logs"
|
||||
log_filename: str = "inspector.log"
|
||||
log_retention_days: int = 30
|
||||
heartbeat: bool = False
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GotifyConfig:
|
||||
enabled: bool = False
|
||||
url: str = ""
|
||||
token: str = ""
|
||||
priority: int = 5
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PushoverConfig:
|
||||
enabled: bool = False
|
||||
token: str = "" # Pushover application API token
|
||||
user_key: str = "" # Pushover user or group key
|
||||
priority: int = 5 # Inspector-scale priority, mapped to Pushover's -2..2 at send time
|
||||
device: str = "" # optional target device name; blank = all devices
|
||||
sound: str = "" # optional sound name; blank = user default
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DatabaseConfig:
|
||||
enabled: bool = False
|
||||
path: str = "/app/data/events.db"
|
||||
retention_days: int = 30
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HealthSummaryConfig:
|
||||
enabled: bool = False
|
||||
interval_hours: int = 24
|
||||
priority: int = 3
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MaintenanceConfig:
|
||||
enabled: bool = False
|
||||
start: str = "00:00"
|
||||
end: str = "06:00"
|
||||
dir: str = "/app/data/maintenance"
|
||||
max_hours: int = 8
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NotificationMuteConfig:
|
||||
enabled: bool = False
|
||||
dir: str = "/app/data/muted"
|
||||
default_hours: int = 12
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DashboardConfig:
|
||||
enabled: bool = False
|
||||
host: str = "0.0.0.0"
|
||||
port: int = 8080
|
||||
username: str = "admin"
|
||||
password: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HttpCheckConfig:
|
||||
enabled: bool = False
|
||||
url: str = ""
|
||||
timeout_seconds: float = 5.0
|
||||
expected_status: int = 200
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UptimeKumaMonitorConfig:
|
||||
enabled: bool = False
|
||||
monitor_name: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UptimeKumaConfig:
|
||||
enabled: bool = False
|
||||
url: str = ""
|
||||
status_page_slug: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PortainerEnvironmentConfig:
|
||||
base_url: str = ""
|
||||
api_token_env: str = "" # name of env var holding the token (preferred; token never stored)
|
||||
api_token: str = "" # token embedded directly in config (use api_token_env for security)
|
||||
endpoint_id: int = 0
|
||||
verify_ssl: bool = True
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PortainerConfig:
|
||||
enabled: bool = False
|
||||
default_environment: str = ""
|
||||
stack_redeploy_cooldown_seconds: int = 300
|
||||
post_redeploy_wait_seconds: int = 90
|
||||
environments: dict[str, PortainerEnvironmentConfig] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IntegrationsConfig:
|
||||
uptime_kuma: UptimeKumaConfig
|
||||
portainer: PortainerConfig
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ContainerConfig:
|
||||
name: str
|
||||
enabled: bool = True
|
||||
max_error_count: int = 5
|
||||
cooldown_seconds: int = 300
|
||||
grace_period_seconds: int = 120
|
||||
max_restarts_per_hour: int = 3
|
||||
restart_on: frozenset[str] = field(default_factory=lambda: frozenset({"UNHEALTHY", "NOT RUNNING"}))
|
||||
maintenance_window: bool = False
|
||||
color: str = "" # optional dashboard accent color, e.g. "#60a5fa"
|
||||
http_check: HttpCheckConfig = field(default_factory=HttpCheckConfig)
|
||||
uptime_kuma: UptimeKumaMonitorConfig = field(default_factory=UptimeKumaMonitorConfig)
|
||||
depends_on: str = ""
|
||||
vpn_routed: bool = False
|
||||
recovery_action: str = "" # "" -> default restart_container behavior
|
||||
portainer_environment: str = ""
|
||||
portainer_stack_name: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AppConfig:
|
||||
inspector: InspectorConfig
|
||||
gotify: GotifyConfig
|
||||
pushover: PushoverConfig
|
||||
database: DatabaseConfig
|
||||
health_summary: HealthSummaryConfig
|
||||
maintenance: MaintenanceConfig
|
||||
notification_mute: NotificationMuteConfig
|
||||
dashboard: DashboardConfig
|
||||
integrations: IntegrationsConfig
|
||||
containers: tuple[ContainerConfig, ...]
|
||||
|
||||
|
||||
def load_config(path: str | Path) -> AppConfig:
|
||||
raw = yaml.safe_load(Path(path).read_text()) or {}
|
||||
return build_config_from_raw(raw)
|
||||
|
||||
|
||||
def build_config_from_raw(raw: dict) -> AppConfig:
|
||||
"""Validate and materialize an AppConfig from an already-parsed dict.
|
||||
|
||||
Used both by load_config() on startup and by the settings-page save path
|
||||
to validate an edited raw dict before writing it back to disk.
|
||||
"""
|
||||
insp_raw = raw.get("inspector", {}) or {}
|
||||
inspector = InspectorConfig(
|
||||
interval_seconds=int(insp_raw.get("interval_seconds", 60)),
|
||||
log_level=str(insp_raw.get("log_level", "INFO")).upper(),
|
||||
dry_run=bool(insp_raw.get("dry_run", False)),
|
||||
log_dir=str(insp_raw.get("log_dir", "/app/logs")),
|
||||
log_filename=str(insp_raw.get("log_filename", "inspector.log")),
|
||||
log_retention_days=int(insp_raw.get("log_retention_days", 30)),
|
||||
heartbeat=bool(insp_raw.get("heartbeat", False)),
|
||||
)
|
||||
|
||||
gotify_raw = ((raw.get("notifications") or {}).get("gotify")) or {}
|
||||
gotify = GotifyConfig(
|
||||
enabled=bool(gotify_raw.get("enabled", False)),
|
||||
url=str(gotify_raw.get("url", "") or "").rstrip("/"),
|
||||
token=str(gotify_raw.get("token", "") or ""),
|
||||
priority=int(gotify_raw.get("priority", 5)),
|
||||
)
|
||||
|
||||
pushover_raw = ((raw.get("notifications") or {}).get("pushover")) or {}
|
||||
pushover = PushoverConfig(
|
||||
enabled=bool(pushover_raw.get("enabled", False)),
|
||||
token=str(pushover_raw.get("token", "") or ""),
|
||||
user_key=str(pushover_raw.get("user_key", "") or ""),
|
||||
priority=int(pushover_raw.get("priority", 5)),
|
||||
device=str(pushover_raw.get("device", "") or ""),
|
||||
sound=str(pushover_raw.get("sound", "") or ""),
|
||||
)
|
||||
if pushover.enabled and (not pushover.token or not pushover.user_key):
|
||||
raise ValueError(
|
||||
"config: notifications.pushover.enabled is true but token and/or user_key is empty"
|
||||
)
|
||||
|
||||
db_raw = raw.get("database") or {}
|
||||
database = DatabaseConfig(
|
||||
enabled=bool(db_raw.get("enabled", False)),
|
||||
path=str(db_raw.get("path", "/app/data/events.db")),
|
||||
retention_days=int(db_raw.get("retention_days", 30)),
|
||||
)
|
||||
|
||||
hs_raw = raw.get("health_summary") or {}
|
||||
health_summary = HealthSummaryConfig(
|
||||
enabled=bool(hs_raw.get("enabled", False)),
|
||||
interval_hours=int(hs_raw.get("interval_hours", 24)),
|
||||
priority=int(hs_raw.get("priority", 3)),
|
||||
)
|
||||
|
||||
maint_raw = raw.get("maintenance") or {}
|
||||
maintenance = MaintenanceConfig(
|
||||
enabled=bool(maint_raw.get("enabled", False)),
|
||||
start=str(maint_raw.get("start", "00:00")),
|
||||
end=str(maint_raw.get("end", "06:00")),
|
||||
dir=str(maint_raw.get("dir", "/app/data/maintenance")),
|
||||
max_hours=int(maint_raw.get("max_hours", 8)),
|
||||
)
|
||||
|
||||
notif_mute_raw = raw.get("notification_mute") or {}
|
||||
notification_mute = NotificationMuteConfig(
|
||||
enabled=bool(notif_mute_raw.get("enabled", False)),
|
||||
dir=str(notif_mute_raw.get("dir", "/app/data/muted")),
|
||||
default_hours=int(notif_mute_raw.get("default_hours", 12)),
|
||||
)
|
||||
|
||||
dash_raw = raw.get("dashboard") or {}
|
||||
dashboard = DashboardConfig(
|
||||
enabled=bool(dash_raw.get("enabled", False)),
|
||||
host=str(dash_raw.get("host", "0.0.0.0")),
|
||||
port=int(dash_raw.get("port", 8080)),
|
||||
username=str(dash_raw.get("username", "admin")),
|
||||
password=str(dash_raw.get("password", "")),
|
||||
)
|
||||
|
||||
integrations_raw = raw.get("integrations") or {}
|
||||
|
||||
uk_raw = integrations_raw.get("uptime_kuma") or {}
|
||||
uptime_kuma = UptimeKumaConfig(
|
||||
enabled=bool(uk_raw.get("enabled", False)),
|
||||
url=str(uk_raw.get("url", "") or "").rstrip("/"),
|
||||
status_page_slug=str(uk_raw.get("status_page_slug", "") or ""),
|
||||
)
|
||||
|
||||
portainer_raw = integrations_raw.get("portainer") or {}
|
||||
portainer_envs_raw = portainer_raw.get("environments") or {}
|
||||
if not isinstance(portainer_envs_raw, dict):
|
||||
raise ValueError(
|
||||
"config: 'integrations.portainer.environments' must be a mapping of name -> settings"
|
||||
)
|
||||
portainer_environments: dict[str, PortainerEnvironmentConfig] = {}
|
||||
for env_name, env_settings in portainer_envs_raw.items():
|
||||
env_settings = env_settings or {}
|
||||
portainer_environments[str(env_name)] = PortainerEnvironmentConfig(
|
||||
base_url=str(env_settings.get("base_url", "") or "").rstrip("/"),
|
||||
api_token_env=str(env_settings.get("api_token_env", "") or ""),
|
||||
api_token=str(env_settings.get("api_token", "") or ""),
|
||||
endpoint_id=int(env_settings.get("endpoint_id", 0)),
|
||||
verify_ssl=bool(env_settings.get("verify_ssl", True)),
|
||||
)
|
||||
|
||||
portainer = PortainerConfig(
|
||||
enabled=bool(portainer_raw.get("enabled", False)),
|
||||
default_environment=str(portainer_raw.get("default_environment", "") or ""),
|
||||
stack_redeploy_cooldown_seconds=int(portainer_raw.get("stack_redeploy_cooldown_seconds", 300)),
|
||||
post_redeploy_wait_seconds=int(portainer_raw.get("post_redeploy_wait_seconds", 90)),
|
||||
environments=portainer_environments,
|
||||
)
|
||||
|
||||
integrations = IntegrationsConfig(uptime_kuma=uptime_kuma, portainer=portainer)
|
||||
|
||||
containers_raw = raw.get("containers") or {}
|
||||
if not isinstance(containers_raw, dict):
|
||||
raise ValueError("config: 'containers' must be a mapping of name -> settings")
|
||||
|
||||
containers: list[ContainerConfig] = []
|
||||
for name, settings in containers_raw.items():
|
||||
settings = settings or {}
|
||||
restart_on_raw = settings.get("restart_on")
|
||||
if restart_on_raw is None:
|
||||
restart_on_raw = ["UNHEALTHY", "NOT RUNNING"]
|
||||
restart_on = frozenset(str(s).upper() for s in restart_on_raw)
|
||||
unknown = restart_on - VALID_STATUSES
|
||||
if unknown:
|
||||
raise ValueError(
|
||||
f"config: container {name!r} has unknown restart_on values: {sorted(unknown)}"
|
||||
)
|
||||
http_check_raw = settings.get("http_check") or {}
|
||||
http_check = HttpCheckConfig(
|
||||
enabled=bool(http_check_raw.get("enabled", False)),
|
||||
url=str(http_check_raw.get("url", "") or ""),
|
||||
timeout_seconds=float(http_check_raw.get("timeout_seconds", 5.0)),
|
||||
expected_status=int(http_check_raw.get("expected_status", 200)),
|
||||
)
|
||||
|
||||
uptime_kuma_raw = settings.get("uptime_kuma") or {}
|
||||
uptime_kuma = UptimeKumaMonitorConfig(
|
||||
enabled=bool(uptime_kuma_raw.get("enabled", False)),
|
||||
monitor_name=str(uptime_kuma_raw.get("monitor_name", "") or ""),
|
||||
)
|
||||
|
||||
depends_on = str(settings.get("depends_on", "") or "").strip()
|
||||
vpn_routed = bool(settings.get("vpn_routed", False))
|
||||
recovery_action = str(settings.get("recovery_action", "") or "").strip()
|
||||
portainer_environment = str(settings.get("portainer_environment", "") or "").strip()
|
||||
portainer_stack_name = str(settings.get("portainer_stack_name", "") or "").strip()
|
||||
|
||||
if recovery_action and recovery_action not in VALID_RECOVERY_ACTIONS:
|
||||
raise ValueError(
|
||||
f"config: container {name!r} has unknown recovery_action "
|
||||
f"{recovery_action!r}; expected one of {sorted(VALID_RECOVERY_ACTIONS)}"
|
||||
)
|
||||
|
||||
if (
|
||||
recovery_action in ("portainer_stack_redeploy", "restart_then_redeploy")
|
||||
and not portainer_stack_name
|
||||
):
|
||||
raise ValueError(
|
||||
f"config: container {name!r} has recovery_action "
|
||||
f"{recovery_action!r} but no portainer_stack_name"
|
||||
)
|
||||
|
||||
containers.append(
|
||||
ContainerConfig(
|
||||
name=str(name),
|
||||
enabled=bool(settings.get("enabled", True)),
|
||||
max_error_count=int(settings.get("max_error_count", 5)),
|
||||
cooldown_seconds=int(settings.get("cooldown_seconds", 300)),
|
||||
grace_period_seconds=int(settings.get("grace_period_seconds", 120)),
|
||||
max_restarts_per_hour=int(settings.get("max_restarts_per_hour", 3)),
|
||||
restart_on=restart_on,
|
||||
maintenance_window=bool(settings.get("maintenance_window", False)),
|
||||
color=str(settings.get("color", "") or "").strip(),
|
||||
http_check=http_check,
|
||||
uptime_kuma=uptime_kuma,
|
||||
depends_on=depends_on,
|
||||
vpn_routed=vpn_routed,
|
||||
recovery_action=recovery_action,
|
||||
portainer_environment=portainer_environment,
|
||||
portainer_stack_name=portainer_stack_name,
|
||||
)
|
||||
)
|
||||
|
||||
container_names = {c.name for c in containers}
|
||||
for c in containers:
|
||||
if c.depends_on and c.depends_on not in container_names:
|
||||
raise ValueError(
|
||||
f"config: container {c.name!r} has depends_on={c.depends_on!r}, "
|
||||
f"which is not a configured container"
|
||||
)
|
||||
|
||||
return AppConfig(
|
||||
inspector=inspector,
|
||||
gotify=gotify,
|
||||
pushover=pushover,
|
||||
database=database,
|
||||
health_summary=health_summary,
|
||||
maintenance=maintenance,
|
||||
notification_mute=notification_mute,
|
||||
dashboard=dashboard,
|
||||
integrations=integrations,
|
||||
containers=tuple(containers),
|
||||
)
|
||||
|
||||
|
||||
# --- Roundtrip load/write (preserves comments and key order) ----------------
|
||||
|
||||
_yaml_rt = YAML(typ="rt")
|
||||
_yaml_rt.preserve_quotes = True
|
||||
_yaml_rt.indent(mapping=2, sequence=4, offset=2)
|
||||
_yaml_rt.width = 4096 # don't line-wrap scalar values
|
||||
|
||||
|
||||
def load_raw(path: str | Path):
|
||||
"""Load config.yml as a ruamel CommentedMap preserving comments."""
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
data = _yaml_rt.load(fh)
|
||||
return data if data is not None else {}
|
||||
|
||||
|
||||
def dump_raw(raw) -> str:
|
||||
"""Serialize a raw (ruamel) config dict back to YAML text."""
|
||||
buf = io.StringIO()
|
||||
_yaml_rt.dump(raw, buf)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def write_config_raw(path: str | Path, raw) -> None:
|
||||
"""Write `raw` back to `path`, keeping a `.bak` of the prior file.
|
||||
|
||||
Prefers an atomic tempfile + os.replace() swap. Falls back to an
|
||||
in-place write when the target is a single-file bind mount (as in
|
||||
the Inspector Docker container, where /app/config.yml is a mount
|
||||
point and can't be replaced from inside the container). The
|
||||
fallback still writes the full text at once (not incrementally),
|
||||
so a reader can see the old-or-new content, never partial lines
|
||||
from this writer.
|
||||
"""
|
||||
path = Path(path)
|
||||
text = dump_raw(raw)
|
||||
tmp_path = path.with_suffix(path.suffix + ".tmp")
|
||||
bak_path = path.with_suffix(path.suffix + ".bak")
|
||||
|
||||
if path.exists():
|
||||
try:
|
||||
shutil.copy2(path, bak_path)
|
||||
except OSError:
|
||||
pass # non-fatal; we still want to try the write
|
||||
|
||||
def _fsync(fh) -> None:
|
||||
try:
|
||||
os.fsync(fh.fileno())
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
try:
|
||||
with open(tmp_path, "w", encoding="utf-8") as fh:
|
||||
fh.write(text)
|
||||
fh.flush()
|
||||
_fsync(fh)
|
||||
os.replace(tmp_path, path)
|
||||
except OSError:
|
||||
# Cross-device link (bind-mounted single file) or similar.
|
||||
if tmp_path.exists():
|
||||
try:
|
||||
tmp_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
with open(path, "w", encoding="utf-8") as fh:
|
||||
fh.write(text)
|
||||
fh.flush()
|
||||
_fsync(fh)
|
||||
|
||||
|
||||
class ConfigRef:
|
||||
"""Thread-safe holder for the live AppConfig and its raw ruamel dict.
|
||||
|
||||
Writers call `reload_from_disk()` or `swap(new_raw, new_app)` under the
|
||||
lock. Readers call `.get()` to grab the current AppConfig atomically.
|
||||
"""
|
||||
|
||||
def __init__(self, path: str | Path, app: AppConfig, raw) -> None:
|
||||
self._path = Path(path)
|
||||
self._app = app
|
||||
self._raw = raw
|
||||
self._lock = threading.Lock()
|
||||
|
||||
@property
|
||||
def path(self) -> Path:
|
||||
return self._path
|
||||
|
||||
def get(self) -> AppConfig:
|
||||
with self._lock:
|
||||
return self._app
|
||||
|
||||
def get_raw(self):
|
||||
"""Return a deep copy of the current raw dict for safe mutation."""
|
||||
import copy
|
||||
with self._lock:
|
||||
return copy.deepcopy(self._raw)
|
||||
|
||||
def swap(self, new_raw, new_app: AppConfig) -> None:
|
||||
with self._lock:
|
||||
self._raw = new_raw
|
||||
self._app = new_app
|
||||
|
||||
def reload_from_disk(self) -> AppConfig:
|
||||
"""Re-read config.yml from disk and swap in the new AppConfig."""
|
||||
with self._lock:
|
||||
raw = load_raw(self._path)
|
||||
app = build_config_from_raw(raw)
|
||||
self._raw = raw
|
||||
self._app = app
|
||||
return app
|
||||
Executable
+1601
File diff suppressed because it is too large.
Load diff
Executable
+67
@@ -0,0 +1,67 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
import docker
|
||||
from docker.errors import APIError, NotFound
|
||||
|
||||
from .health import CONTAINER_NOT_FOUND, ERROR, HealthResult, classify
|
||||
|
||||
|
||||
log = logging.getLogger("inspector.docker")
|
||||
|
||||
|
||||
def build_client() -> docker.DockerClient:
|
||||
return docker.from_env()
|
||||
|
||||
|
||||
def get_container_status(client: docker.DockerClient, name: str) -> HealthResult:
|
||||
try:
|
||||
container = client.containers.get(name)
|
||||
return classify(name, container.attrs)
|
||||
except NotFound:
|
||||
return HealthResult(
|
||||
container_name=name,
|
||||
status=CONTAINER_NOT_FOUND,
|
||||
message=f"container {name!r} not found",
|
||||
)
|
||||
except APIError as exc:
|
||||
log.error("Docker API error for %s: %s", name, exc)
|
||||
return HealthResult(
|
||||
container_name=name,
|
||||
status=ERROR,
|
||||
message=f"docker api error: {exc}",
|
||||
)
|
||||
except Exception as exc: # defensive: never let the loop die
|
||||
log.exception("Unexpected error inspecting %s: %s", name, exc)
|
||||
return HealthResult(
|
||||
container_name=name,
|
||||
status=ERROR,
|
||||
message=f"unexpected: {exc}",
|
||||
)
|
||||
|
||||
|
||||
def restart_container(client: docker.DockerClient, name: str) -> tuple[bool, str | None]:
|
||||
try:
|
||||
container = client.containers.get(name)
|
||||
container.restart()
|
||||
return True, None
|
||||
except NotFound:
|
||||
return False, "container not found"
|
||||
except APIError as exc:
|
||||
return False, f"docker api error: {exc}"
|
||||
except Exception as exc:
|
||||
return False, f"unexpected: {exc}"
|
||||
|
||||
|
||||
def start_container(client: docker.DockerClient, name: str) -> tuple[bool, str | None]:
|
||||
try:
|
||||
container = client.containers.get(name)
|
||||
container.start()
|
||||
return True, None
|
||||
except NotFound:
|
||||
return False, "container not found"
|
||||
except APIError as exc:
|
||||
return False, f"docker api error: {exc}"
|
||||
except Exception as exc:
|
||||
return False, f"unexpected: {exc}"
|
||||
Executable
+219
@@ -0,0 +1,219 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
from .config import ContainerConfig, DatabaseConfig
|
||||
from .state import ContainerState, utcnow
|
||||
|
||||
|
||||
log = logging.getLogger("inspector.eventstore")
|
||||
|
||||
|
||||
EVENT_STATUS_CHANGE = "status_change"
|
||||
EVENT_RECOVERY = "recovery"
|
||||
EVENT_NOT_FOUND = "not_found"
|
||||
EVENT_FAILURE = "failure"
|
||||
EVENT_RESTART_LIMIT = "restart_limit"
|
||||
EVENT_RESTART_DRYRUN = "restart_dry_run"
|
||||
EVENT_RESTART = "restart"
|
||||
EVENT_RESTART_FAILED = "restart_failed"
|
||||
EVENT_PORTAINER_REDEPLOY = "portainer_redeploy"
|
||||
EVENT_PORTAINER_REDEPLOY_FAILED = "portainer_redeploy_failed"
|
||||
EVENT_PORTAINER_REDEPLOY_SKIPPED = "portainer_redeploy_skipped"
|
||||
EVENT_MAINTENANCE_ENTER = "maintenance_enter"
|
||||
EVENT_MAINTENANCE_EXIT = "maintenance_exit"
|
||||
EVENT_NOTIFICATION_MUTE_ENTER = "notification_mute_enter"
|
||||
EVENT_NOTIFICATION_MUTE_EXIT = "notification_mute_exit"
|
||||
|
||||
# Event types considered "notable" for the health-history summary digest.
|
||||
# EVENT_STATUS_CHANGE, EVENT_MAINTENANCE_ENTER, EVENT_MAINTENANCE_EXIT,
|
||||
# EVENT_NOTIFICATION_MUTE_ENTER, and EVENT_NOTIFICATION_MUTE_EXIT are
|
||||
# excluded: they're bookkeeping/transition markers, not failure activity,
|
||||
# and would dominate the digest with noise.
|
||||
SUMMARY_EVENT_TYPES = (
|
||||
EVENT_FAILURE,
|
||||
EVENT_RESTART,
|
||||
EVENT_RESTART_DRYRUN,
|
||||
EVENT_RESTART_FAILED,
|
||||
EVENT_RESTART_LIMIT,
|
||||
EVENT_RECOVERY,
|
||||
EVENT_NOT_FOUND,
|
||||
EVENT_PORTAINER_REDEPLOY,
|
||||
EVENT_PORTAINER_REDEPLOY_FAILED,
|
||||
EVENT_PORTAINER_REDEPLOY_SKIPPED,
|
||||
)
|
||||
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS events (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ts TEXT NOT NULL,
|
||||
container TEXT NOT NULL,
|
||||
event_type TEXT NOT NULL,
|
||||
old_status TEXT,
|
||||
new_status TEXT,
|
||||
error_count INTEGER,
|
||||
message TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_container_ts ON events (container, ts);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_ts ON events (ts);
|
||||
"""
|
||||
|
||||
|
||||
class EventStore:
|
||||
"""Durable per-container event history, backed by SQLite.
|
||||
|
||||
Always safe to construct and call: when disabled, or if the database
|
||||
file can't be opened, every method is a no-op. A database error is
|
||||
logged but never crashes Inspector, same philosophy as GotifyNotifier.
|
||||
"""
|
||||
|
||||
def __init__(self, config: DatabaseConfig):
|
||||
self._config = config
|
||||
self._conn: sqlite3.Connection | None = None
|
||||
self._last_prune: datetime | None = None
|
||||
|
||||
if not config.enabled:
|
||||
return
|
||||
|
||||
try:
|
||||
path = Path(config.path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(str(path))
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.executescript(_SCHEMA)
|
||||
self._conn = conn
|
||||
log.info("Event history database opened: %s", path)
|
||||
self.prune()
|
||||
except Exception as exc:
|
||||
log.error("Event history database unavailable (%s); continuing without it", exc)
|
||||
self._conn = None
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self._conn is not None
|
||||
|
||||
def record(
|
||||
self,
|
||||
container: str,
|
||||
event_type: str,
|
||||
old_status: str | None = None,
|
||||
new_status: str | None = None,
|
||||
error_count: int | None = None,
|
||||
message: str | None = None,
|
||||
) -> None:
|
||||
if self._conn is None:
|
||||
return
|
||||
try:
|
||||
self._conn.execute(
|
||||
"INSERT INTO events "
|
||||
"(ts, container, event_type, old_status, new_status, error_count, message) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
utcnow().isoformat(),
|
||||
container,
|
||||
event_type,
|
||||
old_status,
|
||||
new_status,
|
||||
error_count,
|
||||
message,
|
||||
),
|
||||
)
|
||||
self._conn.commit()
|
||||
except Exception as exc:
|
||||
log.error("Failed to record event (%s/%s): %s", container, event_type, exc)
|
||||
|
||||
def prune(self) -> None:
|
||||
if self._conn is None:
|
||||
return
|
||||
try:
|
||||
cutoff = (utcnow() - timedelta(days=self._config.retention_days)).isoformat()
|
||||
cur = self._conn.execute("DELETE FROM events WHERE ts < ?", (cutoff,))
|
||||
self._conn.commit()
|
||||
if cur.rowcount:
|
||||
log.info(
|
||||
"Event history: pruned %d row(s) older than %d days",
|
||||
cur.rowcount,
|
||||
self._config.retention_days,
|
||||
)
|
||||
self._last_prune = utcnow()
|
||||
except Exception as exc:
|
||||
log.error("Event history prune failed: %s", exc)
|
||||
|
||||
def maybe_prune(self) -> None:
|
||||
if self._conn is None:
|
||||
return
|
||||
now = utcnow()
|
||||
if self._last_prune is None or (now - self._last_prune) >= timedelta(hours=24):
|
||||
self.prune()
|
||||
|
||||
def summarize(self, since: datetime) -> dict[str, dict[str, int]]:
|
||||
"""Per-container counts of notable events at/after `since`.
|
||||
|
||||
Returns {container: {event_type: count}}. Containers with zero
|
||||
matching events are absent from the result. Returns {} if disabled
|
||||
or on any error.
|
||||
"""
|
||||
if self._conn is None:
|
||||
return {}
|
||||
try:
|
||||
placeholders = ",".join("?" for _ in SUMMARY_EVENT_TYPES)
|
||||
cur = self._conn.execute(
|
||||
f"SELECT container, event_type, COUNT(*) FROM events "
|
||||
f"WHERE ts >= ? AND event_type IN ({placeholders}) "
|
||||
f"GROUP BY container, event_type",
|
||||
(since.isoformat(), *SUMMARY_EVENT_TYPES),
|
||||
)
|
||||
result: dict[str, dict[str, int]] = {}
|
||||
for container, event_type, count in cur.fetchall():
|
||||
result.setdefault(container, {})[event_type] = count
|
||||
return result
|
||||
except Exception as exc:
|
||||
log.error("Event history summarize failed: %s", exc)
|
||||
return {}
|
||||
|
||||
def restore_last_restarts(
|
||||
self,
|
||||
states: dict[str, ContainerState],
|
||||
configs: dict[str, ContainerConfig],
|
||||
) -> None:
|
||||
"""Backfill last_restart_at (and any still-active cooldown) from the
|
||||
most recent recorded restart per container.
|
||||
|
||||
Called once on startup so the dashboard doesn't show "Never" just
|
||||
because Inspector itself restarted, even though the container's
|
||||
last actual restart is recorded in the event history. No-op when
|
||||
disabled or on any error.
|
||||
"""
|
||||
if self._conn is None:
|
||||
return
|
||||
now = utcnow()
|
||||
for name, state in states.items():
|
||||
try:
|
||||
cur = self._conn.execute(
|
||||
"SELECT ts FROM events WHERE container = ? AND event_type = ? "
|
||||
"ORDER BY ts DESC LIMIT 1",
|
||||
(name, EVENT_RESTART),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
except Exception as exc:
|
||||
log.error("Event history: failed to read last restart for %s: %s", name, exc)
|
||||
continue
|
||||
if row is None:
|
||||
continue
|
||||
last_restart = datetime.fromisoformat(row[0])
|
||||
state.last_restart_at = last_restart
|
||||
state.restart_history.append(last_restart)
|
||||
cooldown_until = last_restart + timedelta(seconds=configs[name].cooldown_seconds)
|
||||
if cooldown_until > now:
|
||||
state.cooldown_until = cooldown_until
|
||||
|
||||
def close(self) -> None:
|
||||
if self._conn is not None:
|
||||
try:
|
||||
self._conn.close()
|
||||
finally:
|
||||
self._conn = None
|
||||
Executable
+27
@@ -0,0 +1,27 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
|
||||
PACIFIC = ZoneInfo("America/Los_Angeles")
|
||||
|
||||
|
||||
def format_timestamp(value: datetime | str | None) -> str:
|
||||
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'.
|
||||
|
||||
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
|
||||
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
|
||||
assumed to be UTC (matching utcnow()/EventStore storage). Unparseable
|
||||
strings are returned unchanged.
|
||||
"""
|
||||
if not value:
|
||||
return ""
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
value = datetime.fromisoformat(value)
|
||||
except ValueError:
|
||||
return value
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p")
|
||||
Executable
+71
@@ -0,0 +1,71 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
HEALTHY = "HEALTHY"
|
||||
UNHEALTHY = "UNHEALTHY"
|
||||
STARTING = "STARTING"
|
||||
NO_HEALTH_CHECK = "NO HEALTH CHECK"
|
||||
NOT_RUNNING = "NOT RUNNING"
|
||||
CONTAINER_NOT_FOUND = "CONTAINER NOT FOUND"
|
||||
ERROR = "ERROR"
|
||||
UNKNOWN = "UNKNOWN"
|
||||
UNREACHABLE = "UNREACHABLE"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HealthResult:
|
||||
container_name: str
|
||||
status: str
|
||||
docker_status: str | None = None
|
||||
health_status: str | None = None
|
||||
message: str | None = None
|
||||
|
||||
|
||||
def classify(container_name: str, inspect_data: dict[str, Any]) -> HealthResult:
|
||||
state = (inspect_data or {}).get("State") or {}
|
||||
docker_status = state.get("Status")
|
||||
health = (state.get("Health") or {}).get("Status")
|
||||
|
||||
if docker_status is None:
|
||||
return HealthResult(
|
||||
container_name=container_name,
|
||||
status=UNKNOWN,
|
||||
docker_status=docker_status,
|
||||
health_status=health,
|
||||
message="missing State.Status",
|
||||
)
|
||||
|
||||
if docker_status != "running":
|
||||
return HealthResult(
|
||||
container_name=container_name,
|
||||
status=NOT_RUNNING,
|
||||
docker_status=docker_status,
|
||||
health_status=health,
|
||||
message=f"docker status: {docker_status}",
|
||||
)
|
||||
|
||||
if health is None:
|
||||
return HealthResult(
|
||||
container_name=container_name,
|
||||
status=NO_HEALTH_CHECK,
|
||||
docker_status=docker_status,
|
||||
health_status=None,
|
||||
)
|
||||
|
||||
if health == "healthy":
|
||||
return HealthResult(container_name, HEALTHY, docker_status, health)
|
||||
if health == "unhealthy":
|
||||
return HealthResult(container_name, UNHEALTHY, docker_status, health)
|
||||
if health == "starting":
|
||||
return HealthResult(container_name, STARTING, docker_status, health)
|
||||
|
||||
return HealthResult(
|
||||
container_name=container_name,
|
||||
status=UNKNOWN,
|
||||
docker_status=docker_status,
|
||||
health_status=health,
|
||||
message=f"unknown health status: {health}",
|
||||
)
|
||||
Executable
+94
@@ -0,0 +1,94 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from .config import HealthSummaryConfig
|
||||
from .eventstore import SUMMARY_EVENT_TYPES, EventStore
|
||||
from .notifier import GotifyNotifier
|
||||
from .state import utcnow
|
||||
|
||||
|
||||
log = logging.getLogger("inspector.health_summary")
|
||||
|
||||
|
||||
_LABELS = {
|
||||
"failure": "failures",
|
||||
"restart": "restarts",
|
||||
"restart_dry_run": "dry-run restarts",
|
||||
"restart_failed": "restart failures",
|
||||
"restart_limit": "restart-limit hits",
|
||||
"recovery": "recoveries",
|
||||
"not_found": "not-found events",
|
||||
"portainer_redeploy": "Portainer redeploys",
|
||||
"portainer_redeploy_failed": "Portainer redeploy failures",
|
||||
"portainer_redeploy_skipped": "Portainer redeploys skipped",
|
||||
}
|
||||
|
||||
|
||||
class HealthSummaryReporter:
|
||||
"""Periodic digest of per-container health activity from EventStore.
|
||||
|
||||
Always safe to construct and call: no-ops when health_summary.enabled
|
||||
is false. If enabled but EventStore isn't recording (database.enabled
|
||||
is false, or the database failed to open), logs a one-time startup
|
||||
warning and digests will always show "no issues".
|
||||
"""
|
||||
|
||||
def __init__(self, config: HealthSummaryConfig, eventstore: EventStore, notifier: GotifyNotifier):
|
||||
self._config = config
|
||||
self._eventstore = eventstore
|
||||
self._notifier = notifier
|
||||
self._last_report: datetime | None = None
|
||||
|
||||
if config.enabled and not eventstore.enabled:
|
||||
log.warning(
|
||||
"health_summary.enabled is true but database.enabled is false "
|
||||
"(or the event database failed to open); digests will report "
|
||||
"no data until event history is enabled"
|
||||
)
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self._config.enabled
|
||||
|
||||
def maybe_report(self, container_names: list[str]) -> None:
|
||||
if not self.enabled:
|
||||
return
|
||||
now = utcnow()
|
||||
if self._last_report is not None and (now - self._last_report) < timedelta(hours=self._config.interval_hours):
|
||||
return
|
||||
|
||||
since = self._last_report or (now - timedelta(hours=self._config.interval_hours))
|
||||
self._report(container_names, since=since, now=now)
|
||||
self._last_report = now
|
||||
|
||||
def _report(self, container_names: list[str], since: datetime, now: datetime) -> None:
|
||||
counts = self._eventstore.summarize(since)
|
||||
|
||||
lines: list[str] = []
|
||||
md_lines: list[str] = []
|
||||
any_activity = False
|
||||
for name in sorted(container_names):
|
||||
c = counts.get(name, {})
|
||||
parts = [f"{_LABELS[t]}={c[t]}" for t in SUMMARY_EVENT_TYPES if c.get(t)]
|
||||
if parts:
|
||||
any_activity = True
|
||||
detail = ", ".join(parts)
|
||||
else:
|
||||
detail = "no issues"
|
||||
lines.append(f"{name}: {detail}")
|
||||
md_lines.append(f"- **{name}**: {detail}")
|
||||
|
||||
hours = max(1, round((now - since).total_seconds() / 3600))
|
||||
log.info("Health summary (last %dh):", hours)
|
||||
for line in lines:
|
||||
log.info(" %s", line)
|
||||
|
||||
if any_activity:
|
||||
body = f"### 📋 Health summary — last {hours}h\n\n" + "\n".join(md_lines)
|
||||
self._notifier.notify(
|
||||
title="📋 Inspector: health summary",
|
||||
message=body,
|
||||
priority=self._config.priority,
|
||||
)
|
||||
Executable
+65
@@ -0,0 +1,65 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
|
||||
import requests
|
||||
|
||||
from .config import HttpCheckConfig
|
||||
|
||||
log = logging.getLogger("inspector.http_check")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HttpCheckResult:
|
||||
ok: bool
|
||||
status_code: int = 0 # synthetic 0 == connection-level failure (refused/timeout/DNS)
|
||||
message: str | None = None
|
||||
|
||||
|
||||
def check_http(config: HttpCheckConfig) -> HttpCheckResult:
|
||||
"""GET config.url and classify the outcome into three tiers:
|
||||
|
||||
- status_code == expected_status -> ok=True (up)
|
||||
- status_code == 0 -> ok=False (unreachable: connection
|
||||
refused / timeout / DNS failure)
|
||||
- any other code -> ok=False (responding, but the
|
||||
status code is unexpected)
|
||||
|
||||
Never raises. Connection-level failures return before any comparison to
|
||||
expected_status, so they can never be mistaken for "up".
|
||||
"""
|
||||
url = config.url
|
||||
try:
|
||||
resp = requests.get(url, timeout=config.timeout_seconds)
|
||||
except requests.Timeout:
|
||||
return HttpCheckResult(
|
||||
ok=False, status_code=0, message=f"HTTP check unreachable (timed out): {url}"
|
||||
)
|
||||
except requests.ConnectionError as exc:
|
||||
return HttpCheckResult(
|
||||
ok=False,
|
||||
status_code=0,
|
||||
message=f"HTTP check unreachable (connection failed): {url} ({exc})",
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
return HttpCheckResult(
|
||||
ok=False, status_code=0, message=f"HTTP check failed: {url} ({exc})"
|
||||
)
|
||||
except Exception as exc:
|
||||
log.exception("Unexpected error in http_check for %s: %s", url, exc)
|
||||
return HttpCheckResult(
|
||||
ok=False, status_code=0, message=f"HTTP check error: {url} ({exc})"
|
||||
)
|
||||
|
||||
code = resp.status_code
|
||||
if code == config.expected_status:
|
||||
return HttpCheckResult(ok=True, status_code=code)
|
||||
return HttpCheckResult(
|
||||
ok=False,
|
||||
status_code=code,
|
||||
message=(
|
||||
f"HTTP check unexpected status: {url} returned {code}, "
|
||||
f"expected {config.expected_status} (responding but unexpected)"
|
||||
),
|
||||
)
|
||||
Executable
+74
@@ -0,0 +1,74 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from logging.handlers import TimedRotatingFileHandler
|
||||
from pathlib import Path
|
||||
|
||||
import colorlog
|
||||
|
||||
from .config import InspectorConfig
|
||||
|
||||
|
||||
class _BlankLineFormatter(logging.Formatter):
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
if record.getMessage() == "":
|
||||
return ""
|
||||
return super().format(record)
|
||||
|
||||
|
||||
def setup_logging(config: InspectorConfig) -> logging.Logger:
|
||||
class _BlankLineColoredFormatter(colorlog.ColoredFormatter):
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
if record.getMessage() == "":
|
||||
return ""
|
||||
return super().format(record)
|
||||
|
||||
log_dir = Path(config.log_dir)
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_path = log_dir / config.log_filename
|
||||
|
||||
root = logging.getLogger()
|
||||
root.setLevel(config.log_level)
|
||||
|
||||
for handler in list(root.handlers):
|
||||
root.removeHandler(handler)
|
||||
|
||||
file_handler = TimedRotatingFileHandler(
|
||||
filename=str(log_path),
|
||||
when="midnight",
|
||||
interval=1,
|
||||
backupCount=config.log_retention_days,
|
||||
encoding="utf-8",
|
||||
utc=False,
|
||||
)
|
||||
file_handler.setFormatter(
|
||||
_BlankLineFormatter(
|
||||
fmt="%(asctime)s %(levelname)-8s %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
)
|
||||
root.addHandler(file_handler)
|
||||
|
||||
console_handler = colorlog.StreamHandler()
|
||||
console_handler.setFormatter(
|
||||
_BlankLineColoredFormatter(
|
||||
fmt="%(log_color)s%(asctime)s %(levelname)-8s%(reset)s %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
log_colors={
|
||||
"DEBUG": "cyan",
|
||||
"INFO": "green",
|
||||
"WARNING": "yellow",
|
||||
"ERROR": "red",
|
||||
"CRITICAL": "bold_red",
|
||||
},
|
||||
)
|
||||
)
|
||||
root.addHandler(console_handler)
|
||||
|
||||
logging.getLogger("urllib3").setLevel(logging.WARNING)
|
||||
logging.getLogger("docker").setLevel(logging.WARNING)
|
||||
|
||||
logger = logging.getLogger("inspector")
|
||||
logger.info("Logging initialized: file=%s level=%s", log_path, config.log_level)
|
||||
return logger
|
||||
Executable
+519
@@ -0,0 +1,519 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from . import __version__
|
||||
from .config import AppConfig, ConfigRef, ContainerConfig, load_config, load_raw
|
||||
from .dashboard import DashboardServer
|
||||
from .docker_client import build_client, get_container_status
|
||||
from .eventstore import (
|
||||
EVENT_FAILURE,
|
||||
EVENT_NOT_FOUND,
|
||||
EVENT_RECOVERY,
|
||||
EVENT_RESTART_LIMIT,
|
||||
EVENT_STATUS_CHANGE,
|
||||
EventStore,
|
||||
)
|
||||
from .formatting import format_timestamp
|
||||
from .health import (
|
||||
CONTAINER_NOT_FOUND,
|
||||
HEALTHY,
|
||||
NO_HEALTH_CHECK,
|
||||
NOT_RUNNING,
|
||||
STARTING,
|
||||
UNHEALTHY,
|
||||
UNKNOWN,
|
||||
UNREACHABLE,
|
||||
HealthResult,
|
||||
)
|
||||
from .health_summary import HealthSummaryReporter
|
||||
from .http_check import check_http
|
||||
from .logging_setup import setup_logging
|
||||
from .maintenance import MaintenanceTracker
|
||||
from .notification_mute import NotificationMuteTracker
|
||||
from .notifier import GotifyNotifier, MultiNotifier, NullNotifier, PushoverNotifier
|
||||
from .portainer_client import PortainerClient
|
||||
from .recovery import perform_recovery
|
||||
from .uptime_kuma import HeartbeatSnapshot, UptimeKumaClient
|
||||
from .state import (
|
||||
ContainerState,
|
||||
should_count_failure,
|
||||
should_restart,
|
||||
utcnow,
|
||||
)
|
||||
|
||||
|
||||
log = logging.getLogger("inspector")
|
||||
|
||||
|
||||
_stop = False
|
||||
|
||||
|
||||
def _request_stop(signum, _frame):
|
||||
global _stop
|
||||
log.info("Received signal %s; shutting down after current loop", signum)
|
||||
_stop = True
|
||||
|
||||
|
||||
def _apply_reachability_override(
|
||||
container_config: ContainerConfig,
|
||||
result: HealthResult,
|
||||
uptime_kuma_snapshot: HeartbeatSnapshot | None,
|
||||
) -> HealthResult:
|
||||
"""Override an otherwise-OK Docker status with UNREACHABLE if an
|
||||
enabled per-container reachability check (HTTP or Uptime Kuma) says the
|
||||
container can't actually be reached.
|
||||
|
||||
Docker failure statuses (UNHEALTHY, NOT_RUNNING, ERROR,
|
||||
CONTAINER_NOT_FOUND, UNKNOWN) are never masked. HTTP check takes
|
||||
precedence over Uptime Kuma if both are enabled and failing.
|
||||
"""
|
||||
if result.status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
|
||||
return result
|
||||
|
||||
name = container_config.name
|
||||
|
||||
http_cfg = container_config.http_check
|
||||
if http_cfg.enabled:
|
||||
check = check_http(http_cfg)
|
||||
if not check.ok:
|
||||
return HealthResult(
|
||||
container_name=name,
|
||||
status=UNREACHABLE,
|
||||
docker_status=result.docker_status,
|
||||
health_status=result.health_status,
|
||||
message=check.message,
|
||||
)
|
||||
|
||||
uk_cfg = container_config.uptime_kuma
|
||||
if uk_cfg.enabled and uptime_kuma_snapshot is not None:
|
||||
if uptime_kuma_snapshot.is_up(uk_cfg.monitor_name) is False:
|
||||
return HealthResult(
|
||||
container_name=name,
|
||||
status=UNREACHABLE,
|
||||
docker_status=result.docker_status,
|
||||
health_status=result.health_status,
|
||||
message=f"Uptime Kuma monitor '{uk_cfg.monitor_name}' is down",
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _check_one(
|
||||
container_config: ContainerConfig,
|
||||
state: ContainerState,
|
||||
result: HealthResult,
|
||||
notifier: GotifyNotifier,
|
||||
eventstore: EventStore,
|
||||
maintenance: MaintenanceTracker,
|
||||
notification_mute: NotificationMuteTracker,
|
||||
client,
|
||||
dry_run: bool,
|
||||
width: int,
|
||||
uptime_kuma_snapshot: HeartbeatSnapshot | None,
|
||||
app: AppConfig,
|
||||
states: dict[str, ContainerState],
|
||||
portainer_client: PortainerClient,
|
||||
stack_cooldowns: dict[str, datetime],
|
||||
) -> None:
|
||||
name = container_config.name
|
||||
now = utcnow()
|
||||
state.last_check_at = now
|
||||
|
||||
state.previous_status = state.current_status
|
||||
state.current_status = result.status
|
||||
|
||||
result = _apply_reachability_override(container_config, result, uptime_kuma_snapshot)
|
||||
state.current_status = result.status # may now be UNREACHABLE
|
||||
|
||||
if maintenance.is_active(name):
|
||||
return
|
||||
|
||||
if notification_mute.is_muted(name):
|
||||
notifier = NullNotifier()
|
||||
|
||||
if state.previous_status != state.current_status:
|
||||
log.info(
|
||||
"%-*s status: %s -> %s%s",
|
||||
width,
|
||||
name,
|
||||
state.previous_status or "INIT",
|
||||
state.current_status,
|
||||
f" ({result.message})" if result.message else "",
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_STATUS_CHANGE,
|
||||
old_status=state.previous_status,
|
||||
new_status=state.current_status,
|
||||
error_count=state.error_count,
|
||||
message=result.message,
|
||||
)
|
||||
|
||||
if (
|
||||
state.previous_status is not None
|
||||
and state.previous_status != HEALTHY
|
||||
and state.current_status == HEALTHY
|
||||
and state.last_notification_state != HEALTHY
|
||||
):
|
||||
log.info("%-*s recovered: %s -> HEALTHY", width, name, state.previous_status)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} recovered",
|
||||
message=(
|
||||
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
|
||||
f"- Errors before recovery: **{state.error_count}**\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RECOVERY,
|
||||
old_status=state.previous_status,
|
||||
new_status=HEALTHY,
|
||||
error_count=state.error_count,
|
||||
)
|
||||
state.last_notification_state = HEALTHY
|
||||
state.error_count = 0
|
||||
state.restart_limit_notified = False
|
||||
state.restarted_this_episode = False
|
||||
log.info("")
|
||||
|
||||
if result.status == CONTAINER_NOT_FOUND:
|
||||
if state.last_notification_state != CONTAINER_NOT_FOUND:
|
||||
log.error("%-*s container not found", width, name)
|
||||
notifier.notify(
|
||||
title=f"❓ Inspector: {name} not found",
|
||||
message=(
|
||||
f"**{name}** is missing from Docker.\n\n"
|
||||
+ (f"- Detail: {result.message}\n" if result.message else "")
|
||||
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=7,
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_NOT_FOUND,
|
||||
old_status=state.previous_status,
|
||||
new_status=CONTAINER_NOT_FOUND,
|
||||
message=result.message,
|
||||
)
|
||||
state.last_notification_state = CONTAINER_NOT_FOUND
|
||||
return
|
||||
|
||||
in_grace = state.in_grace(now, container_config.grace_period_seconds)
|
||||
if in_grace and result.status != HEALTHY:
|
||||
log.debug("%-*s in grace period; not counting %s", width, name, result.status)
|
||||
|
||||
if state.in_cooldown(now):
|
||||
remaining = int((state.cooldown_until - now).total_seconds())
|
||||
log.debug("%-*s in cooldown for %ss", width, name, remaining)
|
||||
return
|
||||
|
||||
if should_count_failure(result.status, container_config, in_grace):
|
||||
state.error_count += 1
|
||||
if state.error_count == 1:
|
||||
log.info("")
|
||||
log.warning(
|
||||
"%-*s failure: status=%s count=%d/%d",
|
||||
width,
|
||||
name,
|
||||
result.status,
|
||||
state.error_count,
|
||||
container_config.max_error_count,
|
||||
)
|
||||
if state.error_count == 1 or state.last_notification_state != result.status:
|
||||
notifier.notify(
|
||||
title=f"🚨 Inspector: {name} is {result.status}",
|
||||
message=(
|
||||
f"**{name}** is **{result.status}**"
|
||||
+ (f" — {result.message}" if result.message else "")
|
||||
+ ".\n\n"
|
||||
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
|
||||
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=5,
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_FAILURE,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=result.message,
|
||||
)
|
||||
state.last_notification_state = result.status
|
||||
elif result.status == HEALTHY:
|
||||
if state.error_count != 0:
|
||||
log.info("%-*s healthy again; resetting error count", width, name)
|
||||
state.error_count = 0
|
||||
state.restarted_this_episode = False
|
||||
|
||||
if state.error_count >= container_config.max_error_count:
|
||||
ok, reason = should_restart(state, container_config, now)
|
||||
if not ok and reason == "hourly restart limit reached":
|
||||
if not state.restart_limit_notified:
|
||||
log.critical(
|
||||
"%-*s exceeded max_restarts_per_hour (%d); pausing restarts",
|
||||
width,
|
||||
name,
|
||||
container_config.max_restarts_per_hour,
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"🛑 Inspector: {name} restart limit reached",
|
||||
message=(
|
||||
f"**{name}** hit its restart limit: "
|
||||
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
|
||||
f"- Current status: **{result.status}**\n"
|
||||
f"- Restarts paused until the hourly window clears\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=9,
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART_LIMIT,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"max_restarts_per_hour={container_config.max_restarts_per_hour}",
|
||||
)
|
||||
state.restart_limit_notified = True
|
||||
return
|
||||
|
||||
if ok:
|
||||
dep = container_config.depends_on
|
||||
if dep:
|
||||
dep_status = states[dep].current_status if dep in states else None
|
||||
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
|
||||
if container_config.vpn_routed:
|
||||
log.info(
|
||||
"%-*s is vpn_routed and depends on %s; deferring recovery "
|
||||
"because %s is %s",
|
||||
width, name, dep, dep, dep_status,
|
||||
)
|
||||
return
|
||||
|
||||
perform_recovery(
|
||||
app=app,
|
||||
container_config=container_config,
|
||||
state=state,
|
||||
result=result,
|
||||
notifier=notifier,
|
||||
eventstore=eventstore,
|
||||
portainer_client=portainer_client,
|
||||
stack_cooldowns=stack_cooldowns,
|
||||
client=client,
|
||||
dry_run=dry_run,
|
||||
width=width,
|
||||
now=now,
|
||||
)
|
||||
|
||||
|
||||
def _topological_order(enabled_configs: dict[str, ContainerConfig]) -> list[str]:
|
||||
"""Order enabled container names so a container with `depends_on`
|
||||
referencing another *enabled* container is checked after its dependency,
|
||||
preserving relative config-file order otherwise.
|
||||
|
||||
Falls back to config order (with a warning) on a dependency cycle, or
|
||||
per-container (with a warning) if `depends_on` points to a disabled or
|
||||
not-enabled container.
|
||||
"""
|
||||
names = list(enabled_configs.keys())
|
||||
deps: dict[str, str | None] = {}
|
||||
for name in names:
|
||||
dep = enabled_configs[name].depends_on
|
||||
if dep and dep in enabled_configs:
|
||||
deps[name] = dep
|
||||
else:
|
||||
if dep and dep not in enabled_configs:
|
||||
log.warning(
|
||||
"%s depends_on %r, which is disabled or not found among enabled "
|
||||
"containers; ignoring dependency ordering for %s",
|
||||
name, dep, name,
|
||||
)
|
||||
deps[name] = None
|
||||
|
||||
order: list[str] = []
|
||||
visiting: set[str] = set()
|
||||
visited: set[str] = set()
|
||||
cycle_detected = False
|
||||
|
||||
def visit(n: str) -> None:
|
||||
nonlocal cycle_detected
|
||||
if n in visited:
|
||||
return
|
||||
if n in visiting:
|
||||
cycle_detected = True
|
||||
return
|
||||
visiting.add(n)
|
||||
dep = deps.get(n)
|
||||
if dep is not None:
|
||||
visit(dep)
|
||||
visiting.discard(n)
|
||||
visited.add(n)
|
||||
order.append(n)
|
||||
|
||||
for name in names:
|
||||
visit(name)
|
||||
if cycle_detected:
|
||||
log.warning(
|
||||
"Dependency cycle detected in container 'depends_on' configuration; "
|
||||
"falling back to config-file order"
|
||||
)
|
||||
return names
|
||||
|
||||
return order
|
||||
|
||||
|
||||
def _build_notifier(app: AppConfig) -> MultiNotifier:
|
||||
return MultiNotifier([GotifyNotifier(app.gotify), PushoverNotifier(app.pushover)])
|
||||
|
||||
|
||||
def run_loop(config_ref: ConfigRef) -> int:
|
||||
app = config_ref.get()
|
||||
notifier = _build_notifier(app)
|
||||
uptime_kuma_client = UptimeKumaClient(app.integrations.uptime_kuma)
|
||||
eventstore = EventStore(app.database)
|
||||
health_summary = HealthSummaryReporter(app.health_summary, eventstore, notifier)
|
||||
maintenance = MaintenanceTracker(
|
||||
app.maintenance,
|
||||
eventstore,
|
||||
scheduled_containers={c.name for c in app.containers if c.maintenance_window},
|
||||
)
|
||||
notification_mute = NotificationMuteTracker(app.notification_mute, eventstore)
|
||||
|
||||
try:
|
||||
client = build_client()
|
||||
client.ping()
|
||||
log.info("Docker connection successful")
|
||||
except Exception as exc:
|
||||
if "Permission denied" in str(exc):
|
||||
log.critical(
|
||||
"Permission denied on /var/run/docker.sock. Set DOCKER_GID to "
|
||||
"the host docker group GID: stat -c '%g' /var/run/docker.sock"
|
||||
)
|
||||
else:
|
||||
log.critical("Cannot reach Docker daemon: %s", exc)
|
||||
return 2
|
||||
|
||||
states: dict[str, ContainerState] = {
|
||||
c.name: ContainerState(name=c.name) for c in app.containers if c.enabled
|
||||
}
|
||||
enabled_configs = {c.name: c for c in app.containers if c.enabled}
|
||||
name_width = max((len(name) for name in enabled_configs), default=0)
|
||||
eventstore.restore_last_restarts(states, enabled_configs)
|
||||
|
||||
check_order = _topological_order(enabled_configs)
|
||||
portainer_client = PortainerClient(app.integrations.portainer)
|
||||
stack_cooldowns: dict[str, datetime] = {}
|
||||
|
||||
dashboard = DashboardServer(
|
||||
config_ref,
|
||||
states,
|
||||
maintenance,
|
||||
notification_mute,
|
||||
)
|
||||
dashboard.start()
|
||||
|
||||
log.info(
|
||||
"Inspector %s started; watching %d container(s); interval=%ss dry_run=%s",
|
||||
__version__,
|
||||
len(states),
|
||||
app.inspector.interval_seconds,
|
||||
app.inspector.dry_run,
|
||||
)
|
||||
|
||||
while not _stop:
|
||||
app = config_ref.get() # re-read each iteration so settings-page edits take effect
|
||||
notifier = _build_notifier(app)
|
||||
|
||||
# Rebuild enabled_configs / check_order from current config so edits
|
||||
# to per-container settings apply next cycle. Containers added via
|
||||
# the settings page are logged as 'requires restart' because the
|
||||
# states dict is built once at startup.
|
||||
current_enabled = {c.name: c for c in app.containers if c.enabled}
|
||||
new_names = current_enabled.keys() - states.keys()
|
||||
if new_names:
|
||||
log.warning(
|
||||
"Containers added in config require Inspector restart to begin monitoring: %s",
|
||||
sorted(new_names),
|
||||
)
|
||||
enabled_configs = {n: c for n, c in current_enabled.items() if n in states}
|
||||
check_order = _topological_order(enabled_configs)
|
||||
|
||||
uptime_kuma_snapshot = uptime_kuma_client.fetch()
|
||||
for name in check_order:
|
||||
cfg = enabled_configs[name]
|
||||
result = get_container_status(client, name)
|
||||
try:
|
||||
_check_one(
|
||||
container_config=cfg,
|
||||
state=states[name],
|
||||
result=result,
|
||||
notifier=notifier,
|
||||
eventstore=eventstore,
|
||||
maintenance=maintenance,
|
||||
notification_mute=notification_mute,
|
||||
client=client,
|
||||
dry_run=app.inspector.dry_run,
|
||||
width=name_width,
|
||||
uptime_kuma_snapshot=uptime_kuma_snapshot,
|
||||
app=app,
|
||||
states=states,
|
||||
portainer_client=portainer_client,
|
||||
stack_cooldowns=stack_cooldowns,
|
||||
)
|
||||
except Exception as exc:
|
||||
log.exception("%-*s error processing check: %s", name_width, name, exc)
|
||||
|
||||
if app.inspector.heartbeat:
|
||||
log.info("heartbeat: %d containers checked", len(states))
|
||||
|
||||
eventstore.maybe_prune()
|
||||
health_summary.maybe_report(list(enabled_configs.keys()))
|
||||
|
||||
# interruptible sleep
|
||||
slept = 0
|
||||
while slept < app.inspector.interval_seconds and not _stop:
|
||||
time.sleep(min(1, app.inspector.interval_seconds - slept))
|
||||
slept += 1
|
||||
|
||||
dashboard.stop()
|
||||
eventstore.close()
|
||||
log.info("Inspector stopped")
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(prog="inspector", description="Docker container health watcher")
|
||||
parser.add_argument("--config", "-c", default="/app/config.yml", help="Path to config.yml")
|
||||
parser.add_argument("--version", action="version", version=f"inspector {__version__}")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
config_path = Path(args.config)
|
||||
if not config_path.exists():
|
||||
print(f"config file not found: {config_path}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
app = load_config(config_path)
|
||||
raw = load_raw(config_path)
|
||||
config_ref = ConfigRef(config_path, app, raw)
|
||||
setup_logging(app.inspector)
|
||||
|
||||
signal.signal(signal.SIGTERM, _request_stop)
|
||||
signal.signal(signal.SIGINT, _request_stop)
|
||||
|
||||
return run_loop(config_ref)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+142
@@ -0,0 +1,142 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, time, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
from .config import MaintenanceConfig
|
||||
from .eventstore import EVENT_MAINTENANCE_ENTER, EVENT_MAINTENANCE_EXIT, EventStore
|
||||
from .state import localnow, utcnow
|
||||
|
||||
|
||||
log = logging.getLogger("inspector.maintenance")
|
||||
|
||||
|
||||
def _parse_hhmm(value: str) -> time:
|
||||
hour, minute = value.split(":")
|
||||
return time(int(hour), int(minute))
|
||||
|
||||
|
||||
def _in_window(now_time: time, start: time, end: time) -> bool:
|
||||
if start <= end:
|
||||
return start <= now_time < end
|
||||
return now_time >= start or now_time < end # wraps past midnight
|
||||
|
||||
|
||||
def _next_window(now: datetime, start: time, end: time) -> tuple[datetime, datetime]:
|
||||
"""Return the recurring daily maintenance window as (start, end) datetimes,
|
||||
in the same (naive) timezone as ``now``. If ``now`` is inside a window, that
|
||||
active window is returned; otherwise the next upcoming one. Handles both
|
||||
same-day and past-midnight (start > end) windows.
|
||||
|
||||
Caller must ensure start != end (a zero/full-day window is meaningless here).
|
||||
"""
|
||||
start_secs = start.hour * 3600 + start.minute * 60 + start.second
|
||||
end_secs = end.hour * 3600 + end.minute * 60 + end.second
|
||||
duration = timedelta(seconds=(end_secs - start_secs) % 86400)
|
||||
|
||||
today = now.date()
|
||||
candidates = [
|
||||
datetime.combine(today + timedelta(days=offset), start)
|
||||
for offset in (-1, 0, 1)
|
||||
]
|
||||
upcoming: list[datetime] = []
|
||||
for s in candidates:
|
||||
if s <= now < s + duration:
|
||||
return s, s + duration
|
||||
if s > now:
|
||||
upcoming.append(s)
|
||||
s = min(upcoming)
|
||||
return s, s + duration
|
||||
|
||||
|
||||
class MaintenanceTracker:
|
||||
"""Per-container maintenance windows: recurring daily schedule OR
|
||||
ad-hoc marker files. Always safe to construct and call: a complete
|
||||
no-op when maintenance.enabled is false.
|
||||
|
||||
- Recurring window: containers with maintenance_window: true in their
|
||||
config are "in maintenance" every day between maintenance.start and
|
||||
maintenance.end (local time, HH:MM, may wrap past midnight).
|
||||
- Marker files: touch <maintenance.dir>/<container> to start, rm to
|
||||
end. A marker older than maintenance.max_hours auto-expires.
|
||||
|
||||
Logs one INFO line on enter/exit and records
|
||||
EVENT_MAINTENANCE_ENTER/EVENT_MAINTENANCE_EXIT (if event history is
|
||||
enabled) on each transition.
|
||||
"""
|
||||
|
||||
def __init__(self, config: MaintenanceConfig, eventstore: EventStore, scheduled_containers: set[str]):
|
||||
self._config = config
|
||||
self._eventstore = eventstore
|
||||
self._scheduled = scheduled_containers
|
||||
self._start = _parse_hhmm(config.start)
|
||||
self._end = _parse_hhmm(config.end)
|
||||
self._active: dict[str, bool] = {}
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self._config.enabled
|
||||
|
||||
def is_active(self, name: str) -> bool:
|
||||
if not self.enabled:
|
||||
return False
|
||||
|
||||
active = self._marker_active(name) or self._scheduled_active(name)
|
||||
|
||||
was_active = self._active.get(name, False)
|
||||
if active != was_active:
|
||||
self._log_transition(name, active)
|
||||
self._active[name] = active
|
||||
return active
|
||||
|
||||
def peek(self, name: str) -> bool:
|
||||
"""Read-only maintenance check for display purposes (e.g. the
|
||||
dashboard). No transition logging, no eventstore writes, and no
|
||||
update to internal tracking state. Safe to call from another
|
||||
thread."""
|
||||
if not self.enabled:
|
||||
return False
|
||||
return self._marker_active(name) or self._scheduled_active(name)
|
||||
|
||||
def next_scheduled_window(self, name: str) -> tuple[datetime, datetime] | None:
|
||||
"""Return the current or next recurring maintenance window for a
|
||||
scheduled container as (start, end) naive-local datetimes, or None if
|
||||
maintenance is disabled, the container has no recurring window, or the
|
||||
window is empty (start == end). Read-only; safe to call from the
|
||||
dashboard thread."""
|
||||
if not self.enabled or name not in self._scheduled or self._start == self._end:
|
||||
return None
|
||||
return _next_window(localnow(), self._start, self._end)
|
||||
|
||||
def _scheduled_active(self, name: str) -> bool:
|
||||
if name not in self._scheduled:
|
||||
return False
|
||||
return _in_window(localnow().time(), self._start, self._end)
|
||||
|
||||
def _marker_active(self, name: str) -> bool:
|
||||
marker = Path(self._config.dir) / name
|
||||
try:
|
||||
if not marker.is_file():
|
||||
return False
|
||||
age_hours = (utcnow().timestamp() - marker.stat().st_mtime) / 3600
|
||||
if age_hours > self._config.max_hours:
|
||||
if self._active.get(name, False):
|
||||
log.warning(
|
||||
"%s maintenance marker expired after %.1fh (max_hours=%d); "
|
||||
"resuming normal monitoring",
|
||||
name, age_hours, self._config.max_hours,
|
||||
)
|
||||
return False
|
||||
return True
|
||||
except Exception as exc:
|
||||
log.error("%s maintenance marker check failed: %s", name, exc)
|
||||
return False
|
||||
|
||||
def _log_transition(self, name: str, active: bool) -> None:
|
||||
if active:
|
||||
log.info("%s entering maintenance window", name)
|
||||
self._eventstore.record(container=name, event_type=EVENT_MAINTENANCE_ENTER)
|
||||
else:
|
||||
log.info("%s exiting maintenance window; resuming normal monitoring", name)
|
||||
self._eventstore.record(container=name, event_type=EVENT_MAINTENANCE_EXIT)
|
||||
Executable
+93
@@ -0,0 +1,93 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from .config import NotificationMuteConfig
|
||||
from .eventstore import EVENT_NOTIFICATION_MUTE_ENTER, EVENT_NOTIFICATION_MUTE_EXIT, EventStore
|
||||
from .state import utcnow
|
||||
|
||||
|
||||
log = logging.getLogger("inspector.notification_mute")
|
||||
|
||||
|
||||
class NotificationMuteTracker:
|
||||
"""Per-container ad-hoc notification mute via marker files under
|
||||
notification_mute.dir. Always safe to construct and call: a complete
|
||||
no-op when notification_mute.enabled is false.
|
||||
|
||||
Muting silences outbound Gotify notifications only -- monitoring,
|
||||
failure counting, cooldowns, restarts/recovery, and event-history
|
||||
recording continue unchanged. A marker older than
|
||||
notification_mute.default_hours auto-expires.
|
||||
|
||||
Logs one INFO line on mute/unmute and records
|
||||
EVENT_NOTIFICATION_MUTE_ENTER/EVENT_NOTIFICATION_MUTE_EXIT (if event
|
||||
history is enabled) on each transition.
|
||||
"""
|
||||
|
||||
def __init__(self, config: NotificationMuteConfig, eventstore: EventStore):
|
||||
self._config = config
|
||||
self._eventstore = eventstore
|
||||
self._active: dict[str, bool] = {}
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self._config.enabled
|
||||
|
||||
def is_muted(self, name: str) -> bool:
|
||||
if not self.enabled:
|
||||
return False
|
||||
|
||||
active = self._marker_expiry(name) is not None
|
||||
|
||||
was_active = self._active.get(name, False)
|
||||
if active != was_active:
|
||||
self._log_transition(name, active)
|
||||
self._active[name] = active
|
||||
return active
|
||||
|
||||
def peek(self, name: str) -> bool:
|
||||
"""Read-only mute check for display (e.g. the dashboard). No
|
||||
transition logging, no eventstore writes, no state update. Safe to
|
||||
call from another thread."""
|
||||
if not self.enabled:
|
||||
return False
|
||||
return self._marker_expiry(name) is not None
|
||||
|
||||
def muted_until(self, name: str) -> datetime | None:
|
||||
"""Expiry time for an active mute, or None if not muted/disabled."""
|
||||
if not self.enabled:
|
||||
return None
|
||||
return self._marker_expiry(name)
|
||||
|
||||
def _marker_expiry(self, name: str) -> datetime | None:
|
||||
marker = Path(self._config.dir) / name
|
||||
try:
|
||||
if not marker.is_file():
|
||||
return None
|
||||
mtime = marker.stat().st_mtime
|
||||
age_hours = (utcnow().timestamp() - mtime) / 3600
|
||||
if age_hours > self._config.default_hours:
|
||||
if self._active.get(name, False):
|
||||
log.warning(
|
||||
"%s notification mute expired after %.1fh (default_hours=%d); "
|
||||
"alerts resumed",
|
||||
name, age_hours, self._config.default_hours,
|
||||
)
|
||||
return None
|
||||
return datetime.fromtimestamp(mtime, tz=timezone.utc) + timedelta(
|
||||
hours=self._config.default_hours
|
||||
)
|
||||
except Exception as exc:
|
||||
log.error("%s notification mute marker check failed: %s", name, exc)
|
||||
return None
|
||||
|
||||
def _log_transition(self, name: str, active: bool) -> None:
|
||||
if active:
|
||||
log.info("%s notifications muted", name)
|
||||
self._eventstore.record(container=name, event_type=EVENT_NOTIFICATION_MUTE_ENTER)
|
||||
else:
|
||||
log.info("%s notifications unmuted; alerts resumed", name)
|
||||
self._eventstore.record(container=name, event_type=EVENT_NOTIFICATION_MUTE_EXIT)
|
||||
Executable
+149
@@ -0,0 +1,149 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Iterable
|
||||
|
||||
import requests
|
||||
|
||||
from .config import GotifyConfig, PushoverConfig
|
||||
|
||||
|
||||
log = logging.getLogger("inspector.notify")
|
||||
|
||||
|
||||
class GotifyNotifier:
|
||||
def __init__(self, config: GotifyConfig, timeout: float = 5.0):
|
||||
self._config = config
|
||||
self._timeout = timeout
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self._config.enabled and bool(self._config.url) and bool(self._config.token)
|
||||
|
||||
def notify(self, title: str, message: str, priority: int | None = None) -> None:
|
||||
if not self.enabled:
|
||||
return
|
||||
url = f"{self._config.url}/message"
|
||||
payload = {
|
||||
"title": title,
|
||||
"message": message,
|
||||
"priority": self._config.priority if priority is None else priority,
|
||||
"extras": {
|
||||
"client::display": {"contentType": "text/markdown"},
|
||||
},
|
||||
}
|
||||
try:
|
||||
resp = requests.post(
|
||||
url,
|
||||
params={"token": self._config.token},
|
||||
json=payload,
|
||||
timeout=self._timeout,
|
||||
)
|
||||
if resp.status_code >= 400:
|
||||
log.error(
|
||||
"Gotify notification failed: status=%s body=%s",
|
||||
resp.status_code,
|
||||
resp.text[:200],
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
log.error("Gotify notification failed: %s", exc)
|
||||
except Exception as exc:
|
||||
log.exception("Gotify notifier crashed: %s", exc)
|
||||
|
||||
|
||||
# Pushover's priority range is -2..2. Inspector uses a Gotify-style scale
|
||||
# (roughly 1..10). Mapping chosen 2026-10-26 (see plan file).
|
||||
# priority 2 ("emergency") requires retry+expire; defaults below.
|
||||
_PUSHOVER_EMERGENCY_RETRY_SECONDS = 60
|
||||
_PUSHOVER_EMERGENCY_EXPIRE_SECONDS = 3600
|
||||
|
||||
|
||||
def _map_priority_to_pushover(inspector_priority: int) -> int:
|
||||
if inspector_priority <= 2:
|
||||
return -1
|
||||
if inspector_priority <= 4:
|
||||
return 0
|
||||
if inspector_priority <= 6:
|
||||
return 1
|
||||
return 2
|
||||
|
||||
|
||||
class PushoverNotifier:
|
||||
API_URL = "https://api.pushover.net/1/messages.json"
|
||||
|
||||
def __init__(self, config: PushoverConfig, timeout: float = 5.0):
|
||||
self._config = config
|
||||
self._timeout = timeout
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return (
|
||||
self._config.enabled
|
||||
and bool(self._config.token)
|
||||
and bool(self._config.user_key)
|
||||
)
|
||||
|
||||
def notify(self, title: str, message: str, priority: int | None = None) -> None:
|
||||
if not self.enabled:
|
||||
return
|
||||
|
||||
inspector_priority = self._config.priority if priority is None else priority
|
||||
pushover_priority = _map_priority_to_pushover(inspector_priority)
|
||||
|
||||
data = {
|
||||
"token": self._config.token,
|
||||
"user": self._config.user_key,
|
||||
"title": title,
|
||||
"message": message,
|
||||
"priority": pushover_priority,
|
||||
}
|
||||
if self._config.device:
|
||||
data["device"] = self._config.device
|
||||
if self._config.sound:
|
||||
data["sound"] = self._config.sound
|
||||
if pushover_priority == 2:
|
||||
data["retry"] = _PUSHOVER_EMERGENCY_RETRY_SECONDS
|
||||
data["expire"] = _PUSHOVER_EMERGENCY_EXPIRE_SECONDS
|
||||
|
||||
try:
|
||||
resp = requests.post(self.API_URL, data=data, timeout=self._timeout)
|
||||
if resp.status_code >= 400:
|
||||
log.error(
|
||||
"Pushover notification failed: status=%s body=%s",
|
||||
resp.status_code,
|
||||
resp.text[:200],
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
log.error("Pushover notification failed: %s", exc)
|
||||
except Exception as exc:
|
||||
log.exception("Pushover notifier crashed: %s", exc)
|
||||
|
||||
|
||||
class MultiNotifier:
|
||||
"""Fan out notify() to every enabled child notifier.
|
||||
|
||||
Enabled is true if any child is enabled. A failure in one child never
|
||||
prevents the others from running.
|
||||
"""
|
||||
|
||||
def __init__(self, notifiers: Iterable[object]):
|
||||
self._notifiers = list(notifiers)
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return any(getattr(n, "enabled", False) for n in self._notifiers)
|
||||
|
||||
def notify(self, title: str, message: str, priority: int | None = None) -> None:
|
||||
for n in self._notifiers:
|
||||
try:
|
||||
n.notify(title=title, message=message, priority=priority)
|
||||
except Exception as exc:
|
||||
log.exception("notifier %r crashed: %s", type(n).__name__, exc)
|
||||
|
||||
|
||||
class NullNotifier:
|
||||
"""No-op notifier: used to silence a muted container's alerts for one
|
||||
_check_one call. Mirrors the notify() signature of the real notifiers."""
|
||||
|
||||
def notify(self, title: str, message: str, priority: int | None = None) -> None:
|
||||
pass
|
||||
Executable
+182
@@ -0,0 +1,182 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
|
||||
import requests
|
||||
|
||||
from .config import PortainerConfig, PortainerEnvironmentConfig
|
||||
|
||||
|
||||
log = logging.getLogger("inspector.portainer")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PortainerRedeployResult:
|
||||
ok: bool
|
||||
message: str = ""
|
||||
|
||||
|
||||
class PortainerClient:
|
||||
"""Talks to one or more named Portainer environments to redeploy stacks.
|
||||
|
||||
Always safe to construct and call: never raises. Any error (missing
|
||||
environment, missing token env var, network failure, stack not found,
|
||||
unexpected response) yields PortainerRedeployResult(ok=False, message=...).
|
||||
|
||||
Security: the API token value is read from the OS environment per-call
|
||||
and is never logged, returned, or included in any message. Only the
|
||||
*name* of the environment variable and the fact that "a Portainer API
|
||||
call was attempted" are logged.
|
||||
"""
|
||||
|
||||
def __init__(self, config: PortainerConfig, timeout: float = 15.0):
|
||||
self._config = config
|
||||
self._timeout = timeout
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self._config.enabled
|
||||
|
||||
def _resolve_environment(self, environment_name: str) -> PortainerEnvironmentConfig | None:
|
||||
return self._config.environments.get(environment_name)
|
||||
|
||||
def _load_token(self, env_cfg: PortainerEnvironmentConfig) -> str | None:
|
||||
if env_cfg.api_token:
|
||||
return env_cfg.api_token
|
||||
if env_cfg.api_token_env:
|
||||
return os.environ.get(env_cfg.api_token_env)
|
||||
return None
|
||||
|
||||
def redeploy_stack(self, environment_name: str, stack_name: str) -> PortainerRedeployResult:
|
||||
env_cfg = self._resolve_environment(environment_name)
|
||||
if env_cfg is None:
|
||||
return PortainerRedeployResult(
|
||||
ok=False,
|
||||
message=(
|
||||
f"Portainer environment {environment_name!r} is not configured "
|
||||
f"under integrations.portainer.environments"
|
||||
),
|
||||
)
|
||||
|
||||
if not env_cfg.base_url:
|
||||
return PortainerRedeployResult(
|
||||
ok=False,
|
||||
message=f"Portainer environment {environment_name!r} has no base_url configured",
|
||||
)
|
||||
|
||||
token = self._load_token(env_cfg)
|
||||
if not token:
|
||||
if env_cfg.api_token_env:
|
||||
msg = (
|
||||
f"Portainer API token not found in environment variable "
|
||||
f"{env_cfg.api_token_env!r} for environment {environment_name!r}. "
|
||||
f"Ensure the env var is set in the Inspector container, or use "
|
||||
f"'api_token' to embed the token directly in config.yml."
|
||||
)
|
||||
else:
|
||||
msg = (
|
||||
f"Portainer environment {environment_name!r} has no API token configured. "
|
||||
f"Set 'api_token_env' (name of an env var) or 'api_token' (direct value) "
|
||||
f"under integrations.portainer.environments.{environment_name}."
|
||||
)
|
||||
return PortainerRedeployResult(ok=False, message=msg)
|
||||
|
||||
headers = {"X-API-Key": token}
|
||||
verify = env_cfg.verify_ssl
|
||||
|
||||
log.info(
|
||||
"Portainer API call attempted: environment=%s endpoint_id=%s stack=%s",
|
||||
environment_name, env_cfg.endpoint_id, stack_name,
|
||||
)
|
||||
|
||||
try:
|
||||
stack = self._find_stack(env_cfg, headers, verify, stack_name)
|
||||
if stack is None:
|
||||
log.warning(
|
||||
"stack %r not found in Portainer environment %r (endpoint_id=%s)",
|
||||
stack_name, environment_name, env_cfg.endpoint_id,
|
||||
)
|
||||
return PortainerRedeployResult(
|
||||
ok=False,
|
||||
message=(
|
||||
f"Stack {stack_name!r} not found in Portainer environment "
|
||||
f"{environment_name!r} (endpoint_id={env_cfg.endpoint_id})"
|
||||
),
|
||||
)
|
||||
|
||||
stack_id = stack["Id"]
|
||||
log.debug("found stack %r with Id=%s", stack_name, stack_id)
|
||||
file_content = self._fetch_stack_file(env_cfg, headers, verify, stack_id)
|
||||
env_vars = stack.get("Env") or []
|
||||
|
||||
log.debug("sending PUT /api/stacks/%s for stack %r", stack_id, stack_name)
|
||||
resp = requests.put(
|
||||
f"{env_cfg.base_url}/api/stacks/{stack_id}",
|
||||
params={"endpointId": env_cfg.endpoint_id},
|
||||
headers=headers,
|
||||
json={
|
||||
"stackFileContent": file_content,
|
||||
"env": env_vars,
|
||||
"prune": False,
|
||||
"pullImage": True,
|
||||
},
|
||||
timeout=self._timeout,
|
||||
verify=verify,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
except requests.HTTPError as exc:
|
||||
body = ""
|
||||
try:
|
||||
body = exc.response.text[:500]
|
||||
except Exception:
|
||||
pass
|
||||
detail = f"HTTP {exc.response.status_code}: {body}" if body else str(exc)
|
||||
log.error(
|
||||
"Portainer stack redeploy failed: environment=%s stack=%s error=%s",
|
||||
environment_name, stack_name, detail,
|
||||
)
|
||||
return PortainerRedeployResult(ok=False, message=f"Portainer API error: {detail}")
|
||||
except requests.RequestException as exc:
|
||||
log.error(
|
||||
"Portainer stack redeploy failed: environment=%s stack=%s error=%s",
|
||||
environment_name, stack_name, exc,
|
||||
)
|
||||
return PortainerRedeployResult(ok=False, message=f"Portainer API request failed: {exc}")
|
||||
except Exception as exc:
|
||||
log.exception(
|
||||
"Portainer stack redeploy crashed: environment=%s stack=%s",
|
||||
environment_name, stack_name,
|
||||
)
|
||||
return PortainerRedeployResult(ok=False, message=f"unexpected error: {exc}")
|
||||
|
||||
log.info(
|
||||
"Portainer stack redeploy succeeded: environment=%s stack=%s stack_id=%s",
|
||||
environment_name, stack_name, stack_id,
|
||||
)
|
||||
return PortainerRedeployResult(ok=True, message=f"redeployed stack {stack_name!r}")
|
||||
|
||||
def _find_stack(self, env_cfg, headers, verify, stack_name):
|
||||
resp = requests.get(
|
||||
f"{env_cfg.base_url}/api/stacks", headers=headers, timeout=self._timeout, verify=verify,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
stacks = resp.json() or []
|
||||
log.debug(
|
||||
"_find_stack: %d stacks total; searching name=%r endpoint_id=%s",
|
||||
len(stacks), stack_name, env_cfg.endpoint_id,
|
||||
)
|
||||
for stack in stacks:
|
||||
if stack.get("Name") == stack_name and stack.get("EndpointId") == env_cfg.endpoint_id:
|
||||
return stack
|
||||
return None
|
||||
|
||||
def _fetch_stack_file(self, env_cfg, headers, verify, stack_id) -> str:
|
||||
log.debug("fetching compose file for stack_id=%s", stack_id)
|
||||
resp = requests.get(
|
||||
f"{env_cfg.base_url}/api/stacks/{stack_id}/file",
|
||||
headers=headers, timeout=self._timeout, verify=verify,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return (resp.json() or {}).get("StackFileContent", "")
|
||||
Executable
+544
@@ -0,0 +1,544 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from .config import AppConfig, ContainerConfig, PortainerConfig
|
||||
from .docker_client import restart_container, start_container
|
||||
from .eventstore import (
|
||||
EVENT_PORTAINER_REDEPLOY,
|
||||
EVENT_PORTAINER_REDEPLOY_FAILED,
|
||||
EVENT_PORTAINER_REDEPLOY_SKIPPED,
|
||||
EVENT_RESTART,
|
||||
EVENT_RESTART_DRYRUN,
|
||||
EVENT_RESTART_FAILED,
|
||||
EventStore,
|
||||
)
|
||||
from .formatting import format_timestamp
|
||||
from .health import HealthResult
|
||||
from .http_check import check_http
|
||||
from .notifier import GotifyNotifier
|
||||
from .portainer_client import PortainerClient
|
||||
from .state import ContainerState
|
||||
|
||||
|
||||
log = logging.getLogger("inspector.recovery")
|
||||
|
||||
_STALE_NAMESPACE_MARKERS = (
|
||||
"joining network namespace of container",
|
||||
"no such container",
|
||||
)
|
||||
|
||||
|
||||
def is_stale_vpn_namespace_error(message: str | None) -> bool:
|
||||
"""Case-insensitive substring match for the Docker error text emitted
|
||||
when a VPN-routed container's `network_mode: container:<vpn>` target no
|
||||
longer exists (the VPN container was recreated)."""
|
||||
if not message:
|
||||
return False
|
||||
lowered = message.lower()
|
||||
return any(marker in lowered for marker in _STALE_NAMESPACE_MARKERS)
|
||||
|
||||
|
||||
def resolve_portainer_environment(
|
||||
container_config: ContainerConfig,
|
||||
portainer_config: PortainerConfig,
|
||||
) -> tuple[str | None, str | None]:
|
||||
"""Resolve the Portainer environment for this container's redeploy.
|
||||
|
||||
Returns (environment_name, error); exactly one is non-None.
|
||||
"""
|
||||
name = container_config.name
|
||||
|
||||
if container_config.portainer_environment:
|
||||
return container_config.portainer_environment, None
|
||||
|
||||
if portainer_config.default_environment:
|
||||
return portainer_config.default_environment, None
|
||||
|
||||
configured = list(portainer_config.environments.keys())
|
||||
if len(configured) == 1:
|
||||
only_env = configured[0]
|
||||
log.info(
|
||||
"resolved Portainer environment %s for %s (only environment configured)",
|
||||
only_env, name,
|
||||
)
|
||||
return only_env, None
|
||||
|
||||
return None, (
|
||||
f"{name} requests recovery_action 'portainer_stack_redeploy' but "
|
||||
f"integrations.portainer.default_environment is not set and "
|
||||
f"{len(configured)} environments are configured "
|
||||
f"({sorted(configured)}); cannot resolve which Portainer environment to use"
|
||||
)
|
||||
|
||||
|
||||
def stack_cooldown_key(environment: str, stack_name: str) -> str:
|
||||
return f"{environment}:{stack_name}"
|
||||
|
||||
|
||||
def in_stack_cooldown(stack_cooldowns: dict[str, datetime], key: str, now: datetime) -> bool:
|
||||
until = stack_cooldowns.get(key)
|
||||
return until is not None and now < until
|
||||
|
||||
|
||||
def perform_recovery(
|
||||
*,
|
||||
app: AppConfig,
|
||||
container_config: ContainerConfig,
|
||||
state: ContainerState,
|
||||
result: HealthResult,
|
||||
notifier: GotifyNotifier,
|
||||
eventstore: EventStore,
|
||||
portainer_client: PortainerClient,
|
||||
stack_cooldowns: dict[str, datetime],
|
||||
client,
|
||||
dry_run: bool,
|
||||
width: int,
|
||||
now: datetime,
|
||||
) -> None:
|
||||
action = container_config.recovery_action or "restart_container"
|
||||
|
||||
if action == "portainer_stack_redeploy":
|
||||
_perform_portainer_redeploy(
|
||||
app=app,
|
||||
container_config=container_config,
|
||||
state=state,
|
||||
result=result,
|
||||
notifier=notifier,
|
||||
eventstore=eventstore,
|
||||
portainer_client=portainer_client,
|
||||
stack_cooldowns=stack_cooldowns,
|
||||
dry_run=dry_run,
|
||||
width=width,
|
||||
now=now,
|
||||
)
|
||||
return
|
||||
|
||||
if action == "restart_then_redeploy":
|
||||
_perform_restart_then_redeploy(
|
||||
app=app,
|
||||
container_config=container_config,
|
||||
state=state,
|
||||
result=result,
|
||||
notifier=notifier,
|
||||
eventstore=eventstore,
|
||||
portainer_client=portainer_client,
|
||||
stack_cooldowns=stack_cooldowns,
|
||||
client=client,
|
||||
dry_run=dry_run,
|
||||
width=width,
|
||||
now=now,
|
||||
)
|
||||
return
|
||||
|
||||
_perform_docker_action(
|
||||
action=action,
|
||||
container_config=container_config,
|
||||
state=state,
|
||||
result=result,
|
||||
notifier=notifier,
|
||||
eventstore=eventstore,
|
||||
client=client,
|
||||
dry_run=dry_run,
|
||||
width=width,
|
||||
now=now,
|
||||
)
|
||||
|
||||
|
||||
def _perform_restart_then_redeploy(
|
||||
*,
|
||||
app: AppConfig,
|
||||
container_config: ContainerConfig,
|
||||
state: ContainerState,
|
||||
result: HealthResult,
|
||||
notifier: GotifyNotifier,
|
||||
eventstore: EventStore,
|
||||
portainer_client: PortainerClient,
|
||||
stack_cooldowns: dict[str, datetime],
|
||||
client,
|
||||
dry_run: bool,
|
||||
width: int,
|
||||
now: datetime,
|
||||
) -> None:
|
||||
"""Tiered recovery: try `docker restart` first, escalate to a Portainer stack
|
||||
redeploy only when needed.
|
||||
|
||||
Tier 1 (first failure episode): attempt a plain `docker restart`. A VPN-routed
|
||||
container whose sidecar was merely *restarted* (same container id) re-establishes
|
||||
its network this way, avoiding a full redeploy.
|
||||
|
||||
Escalate to redeploy when either:
|
||||
- the restart command itself errors (e.g. the sidecar was *recreated*, so the
|
||||
stored `network_mode: container:<old-id>` is stale -> "No such container"), or
|
||||
- the container is still failing on a later recovery cycle
|
||||
(`state.restarted_this_episode` is already set), meaning the restart didn't stick.
|
||||
|
||||
The episode flag resets when the container is next seen HEALTHY (see main._check_one).
|
||||
"""
|
||||
name = container_config.name
|
||||
|
||||
def redeploy() -> None:
|
||||
_perform_portainer_redeploy(
|
||||
app=app,
|
||||
container_config=container_config,
|
||||
state=state,
|
||||
result=result,
|
||||
notifier=notifier,
|
||||
eventstore=eventstore,
|
||||
portainer_client=portainer_client,
|
||||
stack_cooldowns=stack_cooldowns,
|
||||
dry_run=dry_run,
|
||||
width=width,
|
||||
now=now,
|
||||
)
|
||||
|
||||
# Already restarted this episode and still failing -> escalate to redeploy.
|
||||
if state.restarted_this_episode:
|
||||
log.warning(
|
||||
"%-*s still failing after a restart; escalating to Portainer stack redeploy",
|
||||
width, name,
|
||||
)
|
||||
redeploy()
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
log.warning("%-*s DRY RUN: would restart (tier 1 of restart_then_redeploy)", width, name)
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
state.restarted_this_episode = True
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART_DRYRUN,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"would restart due to {result.status} (tier 1, then redeploy if still failing)",
|
||||
)
|
||||
return
|
||||
|
||||
log.warning(
|
||||
"%-*s restarting (tier 1) due to %d consecutive %s checks",
|
||||
width, name, state.error_count, result.status,
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"🔁 Inspector: restarting {name}",
|
||||
message=(
|
||||
f"Restarting **{name}** after **{state.error_count}** consecutive "
|
||||
f"**{result.status}** check(s).\n\n"
|
||||
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
|
||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
|
||||
success, err = restart_container(client, name)
|
||||
|
||||
if success:
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
state.restarted_this_episode = True
|
||||
state.last_notification_state = "RESTARTED"
|
||||
state.restart_limit_notified = False
|
||||
log.info(
|
||||
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
|
||||
width, name, state.cooldown_until.isoformat(),
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} restart completed",
|
||||
message=(
|
||||
f"**{name}** was successfully restarted by Docker.\n\n"
|
||||
f"- Previous status: *{result.status}*\n"
|
||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
return
|
||||
|
||||
# Restart failed -> escalate to redeploy this same round.
|
||||
log.error("%-*s restart failed: %s; escalating to Portainer stack redeploy", width, name, err)
|
||||
if is_stale_vpn_namespace_error(err):
|
||||
log.info(
|
||||
"%-*s restart failure is a stale VPN namespace error; redeploy is required",
|
||||
width, name,
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART_FAILED,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"restart failed ({err}); escalating to redeploy",
|
||||
)
|
||||
redeploy()
|
||||
|
||||
|
||||
def _perform_docker_action(
|
||||
*,
|
||||
action: str,
|
||||
container_config: ContainerConfig,
|
||||
state: ContainerState,
|
||||
result: HealthResult,
|
||||
notifier: GotifyNotifier,
|
||||
eventstore: EventStore,
|
||||
client,
|
||||
dry_run: bool,
|
||||
width: int,
|
||||
now: datetime,
|
||||
) -> None:
|
||||
name = container_config.name
|
||||
verb = "restart" if action == "restart_container" else "start"
|
||||
|
||||
if dry_run:
|
||||
log.warning("%-*s DRY RUN: would %s due to %s status", width, name, verb, result.status)
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART_DRYRUN,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"would {verb} due to {result.status}",
|
||||
)
|
||||
return
|
||||
|
||||
log.warning(
|
||||
"%-*s %sing due to %d consecutive %s checks",
|
||||
width, name, verb.capitalize(), state.error_count, result.status,
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"🔁 Inspector: {verb}ing {name}",
|
||||
message=(
|
||||
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
|
||||
f"**{result.status}** check(s).\n\n"
|
||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
|
||||
if action == "restart_container":
|
||||
success, err = restart_container(client, name)
|
||||
else:
|
||||
success, err = start_container(client, name)
|
||||
|
||||
if success:
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
state.last_notification_state = "RESTARTED"
|
||||
state.restart_limit_notified = False
|
||||
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat())
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} {verb} completed",
|
||||
message=(
|
||||
f"**{name}** was successfully {verb}ed by Docker.\n\n"
|
||||
f"- Previous status: *{result.status}*\n"
|
||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
else:
|
||||
log.error("%-*s %s failed: %s", width, name, verb, err)
|
||||
notifier.notify(
|
||||
title=f"🛑 Inspector: {name} {verb} FAILED",
|
||||
message=(
|
||||
f"{verb.capitalize()} of **{name}** failed.\n\n"
|
||||
f"- Status: **{result.status}**\n"
|
||||
f"- Error count: **{state.error_count}**\n"
|
||||
f"- Time: {format_timestamp(now)}\n\n"
|
||||
f"```\n{err}\n```"
|
||||
),
|
||||
priority=9,
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART_FAILED,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=str(err),
|
||||
)
|
||||
|
||||
|
||||
def _perform_portainer_redeploy(
|
||||
*,
|
||||
app: AppConfig,
|
||||
container_config: ContainerConfig,
|
||||
state: ContainerState,
|
||||
result: HealthResult,
|
||||
notifier: GotifyNotifier,
|
||||
eventstore: EventStore,
|
||||
portainer_client: PortainerClient,
|
||||
stack_cooldowns: dict[str, datetime],
|
||||
dry_run: bool,
|
||||
width: int,
|
||||
now: datetime,
|
||||
) -> None:
|
||||
name = container_config.name
|
||||
portainer_config = app.integrations.portainer
|
||||
stack_name = container_config.portainer_stack_name
|
||||
|
||||
if is_stale_vpn_namespace_error(result.message):
|
||||
log.info("%-*s detected stale VPN network namespace error", width, name)
|
||||
|
||||
if not portainer_config.enabled:
|
||||
log.warning(
|
||||
"%-*s requests recovery_action 'portainer_stack_redeploy' but "
|
||||
"integrations.portainer.enabled is false; skipping",
|
||||
width, name,
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_PORTAINER_REDEPLOY_SKIPPED,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message="integrations.portainer.enabled is false",
|
||||
)
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
return
|
||||
|
||||
environment, error = resolve_portainer_environment(container_config, portainer_config)
|
||||
if error:
|
||||
log.error("%-*s %s", width, name, error)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_PORTAINER_REDEPLOY_SKIPPED,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=error,
|
||||
)
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
return
|
||||
|
||||
key = stack_cooldown_key(environment, stack_name)
|
||||
|
||||
if in_stack_cooldown(stack_cooldowns, key, now):
|
||||
log.info("Portainer stack redeploy skipped due to cooldown for %s", key)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_PORTAINER_REDEPLOY_SKIPPED,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"cooldown active for {key}",
|
||||
)
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
return
|
||||
|
||||
wait_seconds = portainer_config.post_redeploy_wait_seconds
|
||||
|
||||
if dry_run:
|
||||
log.warning("%-*s DRY RUN: would resolve Portainer environment %s", width, name, environment)
|
||||
log.warning("%-*s DRY RUN: would redeploy Portainer stack %s", width, name, key)
|
||||
log.warning("%-*s DRY RUN: would wait %d seconds after stack redeploy", width, name, wait_seconds)
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART_DRYRUN,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"would redeploy Portainer stack {key}",
|
||||
)
|
||||
return
|
||||
|
||||
log.info("resolved Portainer environment %s for %s", environment, name)
|
||||
log.warning("triggering Portainer stack redeploy for environment %s stack %s", environment, stack_name)
|
||||
notifier.notify(
|
||||
title=f"🔁 Inspector: redeploying Portainer stack {stack_name}",
|
||||
message=(
|
||||
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
|
||||
f"to recover **{name}**.\n\n"
|
||||
f"- Status: **{result.status}**\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
|
||||
redeploy_result = portainer_client.redeploy_stack(environment, stack_name)
|
||||
|
||||
if not redeploy_result.ok:
|
||||
log.error("%-*s Portainer stack redeploy failed for %s: %s", width, name, key, redeploy_result.message)
|
||||
notifier.notify(
|
||||
title=f"🛑 Inspector: Portainer redeploy FAILED for {stack_name}",
|
||||
message=(
|
||||
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
|
||||
f"- Triggered by: **{name}** ({result.status})\n"
|
||||
f"- Time: {format_timestamp(now)}\n\n"
|
||||
f"```\n{redeploy_result.message}\n```"
|
||||
),
|
||||
priority=9,
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_PORTAINER_REDEPLOY_FAILED,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=redeploy_result.message,
|
||||
)
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
return
|
||||
|
||||
stack_cooldowns[key] = now + timedelta(seconds=portainer_config.stack_redeploy_cooldown_seconds)
|
||||
log.info("%-*s Portainer stack redeploy succeeded for %s", width, name, key)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_PORTAINER_REDEPLOY,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"redeployed stack {key}",
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: Portainer stack {stack_name} redeployed",
|
||||
message=(
|
||||
f"Portainer stack **{stack_name}** (environment **{environment}**) was "
|
||||
f"successfully redeployed to recover **{name}**.\n\n"
|
||||
f"- Triggered by: *{result.status}*\n"
|
||||
f"- Post-redeploy wait: {wait_seconds}s\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=4,
|
||||
)
|
||||
|
||||
log.info("waiting %d seconds after stack redeploy", wait_seconds)
|
||||
time.sleep(wait_seconds)
|
||||
|
||||
if container_config.http_check.enabled:
|
||||
recheck = check_http(container_config.http_check)
|
||||
if recheck.ok:
|
||||
log.info("recheck succeeded for %s", name)
|
||||
else:
|
||||
log.warning("recheck failed for %s: %s", name, recheck.message)
|
||||
notifier.notify(
|
||||
title=f"⚠️ Inspector: {name} recheck failed after redeploy",
|
||||
message=(
|
||||
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
|
||||
f"for **{name}** still reports a failure.\n\n"
|
||||
f"- Recheck result: *{recheck.message}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
Executable
+79
@@ -0,0 +1,79 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from .config import ContainerConfig
|
||||
from .health import NO_HEALTH_CHECK, STARTING
|
||||
|
||||
|
||||
def utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def localnow() -> datetime:
|
||||
return datetime.now()
|
||||
|
||||
|
||||
@dataclass
|
||||
class ContainerState:
|
||||
name: str
|
||||
current_status: str | None = None
|
||||
previous_status: str | None = None
|
||||
error_count: int = 0
|
||||
cooldown_until: datetime | None = None
|
||||
last_restart_at: datetime | None = None
|
||||
restart_history: deque[datetime] = field(default_factory=lambda: deque(maxlen=64))
|
||||
last_notification_state: str | None = None
|
||||
restart_limit_notified: bool = False
|
||||
restarted_this_episode: bool = False
|
||||
last_check_at: datetime | None = None
|
||||
|
||||
def in_cooldown(self, now: datetime) -> bool:
|
||||
return self.cooldown_until is not None and now < self.cooldown_until
|
||||
|
||||
def in_grace(self, now: datetime, grace_seconds: int) -> bool:
|
||||
if self.last_restart_at is None or grace_seconds <= 0:
|
||||
return False
|
||||
return now < self.last_restart_at + timedelta(seconds=grace_seconds)
|
||||
|
||||
def restarts_last_hour(self, now: datetime) -> int:
|
||||
cutoff = now - timedelta(hours=1)
|
||||
return sum(1 for ts in self.restart_history if ts >= cutoff)
|
||||
|
||||
def record_restart(self, now: datetime, cooldown_seconds: int) -> None:
|
||||
self.last_restart_at = now
|
||||
self.cooldown_until = now + timedelta(seconds=cooldown_seconds)
|
||||
self.restart_history.append(now)
|
||||
self.error_count = 0
|
||||
|
||||
|
||||
def should_count_failure(
|
||||
status: str,
|
||||
container_config: ContainerConfig,
|
||||
in_grace: bool,
|
||||
) -> bool:
|
||||
if status == STARTING and STARTING not in container_config.restart_on:
|
||||
return False
|
||||
if status == NO_HEALTH_CHECK and NO_HEALTH_CHECK not in container_config.restart_on:
|
||||
return False
|
||||
if status not in container_config.restart_on:
|
||||
return False
|
||||
if in_grace:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def should_restart(
|
||||
state: ContainerState,
|
||||
container_config: ContainerConfig,
|
||||
now: datetime,
|
||||
) -> tuple[bool, str | None]:
|
||||
if state.in_cooldown(now):
|
||||
return False, "in cooldown"
|
||||
if state.error_count < container_config.max_error_count:
|
||||
return False, "below threshold"
|
||||
if state.restarts_last_hour(now) >= container_config.max_restarts_per_hour:
|
||||
return False, "hourly restart limit reached"
|
||||
return True, None
|
||||
Executable
+76
@@ -0,0 +1,76 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
|
||||
import requests
|
||||
|
||||
from .config import UptimeKumaConfig
|
||||
|
||||
log = logging.getLogger("inspector.uptime_kuma")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HeartbeatSnapshot:
|
||||
up_by_monitor: dict[str, bool]
|
||||
|
||||
def is_up(self, monitor_name: str) -> bool | None:
|
||||
return self.up_by_monitor.get(monitor_name)
|
||||
|
||||
|
||||
_EMPTY = HeartbeatSnapshot(up_by_monitor={})
|
||||
|
||||
|
||||
class UptimeKumaClient:
|
||||
"""Polls a public Uptime Kuma status-page heartbeat endpoint.
|
||||
|
||||
Always safe to construct and call: disabled config, network errors, or
|
||||
an unexpected response shape all yield an empty snapshot (every
|
||||
`is_up()` returns None, meaning "no signal") rather than raising.
|
||||
"""
|
||||
|
||||
def __init__(self, config: UptimeKumaConfig, timeout: float = 5.0):
|
||||
self._config = config
|
||||
self._timeout = timeout
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return (
|
||||
self._config.enabled
|
||||
and bool(self._config.url)
|
||||
and bool(self._config.status_page_slug)
|
||||
)
|
||||
|
||||
def fetch(self) -> HeartbeatSnapshot:
|
||||
if not self.enabled:
|
||||
return _EMPTY
|
||||
|
||||
url = f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
|
||||
try:
|
||||
resp = requests.get(url, timeout=self._timeout)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return self._parse(data)
|
||||
except Exception as exc:
|
||||
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", url, exc)
|
||||
return _EMPTY
|
||||
|
||||
def _parse(self, data: dict) -> HeartbeatSnapshot:
|
||||
# Expected shape:
|
||||
# {"publicGroupList": [{"monitorList": [{"id": 1, "name": "..."}]}],
|
||||
# "heartbeatList": {"1": [{"status": 1, ...}, ...]}} # 1 == up
|
||||
up_by_monitor: dict[str, bool] = {}
|
||||
id_to_name: dict[int, str] = {}
|
||||
for group in data.get("publicGroupList") or []:
|
||||
for monitor in group.get("monitorList") or []:
|
||||
mid, mname = monitor.get("id"), monitor.get("name")
|
||||
if mid is not None and mname:
|
||||
id_to_name[mid] = mname
|
||||
|
||||
heartbeat_list = data.get("heartbeatList") or {}
|
||||
for mid, name in id_to_name.items():
|
||||
beats = heartbeat_list.get(str(mid)) or heartbeat_list.get(mid)
|
||||
if beats:
|
||||
up_by_monitor[name] = beats[-1].get("status") == 1
|
||||
|
||||
return HeartbeatSnapshot(up_by_monitor=up_by_monitor)
|
||||
Reference in new issue
Block a user