Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2a9c57b7d7 | ||
|
|
6aed0cf9a7 | ||
|
|
57100d0a7b | ||
|
|
b4da527732 | ||
|
|
284ba2ea15 | ||
|
|
bd78835f4a | ||
|
|
7013a13cba | ||
|
|
c603e86e75 | ||
|
|
4c05a4d565 | ||
|
|
88f6d06df9 | ||
|
|
c6d2a67a4b | ||
|
|
48e913e5d5 |
No files matched your search
@@ -7,10 +7,334 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
|
||||
|
||||
---
|
||||
|
||||
## [0.9.5] — 2026-10-28
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A stack redeploy no longer freezes Inspector.** After a successful Portainer
|
||||
redeploy, Inspector used to stop everything for `post_redeploy_wait_seconds`
|
||||
(default 90 s): no other container was checked, dashboard times stood still, and
|
||||
stopping Inspector during the wait wasn't clean. Now the redeployed app is left to
|
||||
settle while every other container keeps being checked, and its one-time HTTP
|
||||
re-check (with the "recheck failed" alert) happens on its first check after the wait.
|
||||
- The redeployed app's cooldown now starts after the settle time. Before, the wait
|
||||
ate into it (with the defaults, 300 s cooldown became 210 s).
|
||||
|
||||
---
|
||||
|
||||
## [0.9.4] — 2026-10-28
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Containers without a Docker health check now "recover" properly.** Many images
|
||||
don't define a health check, so Docker reports them as `NO HEALTH CHECK` even when
|
||||
they work. Inspector only treated `HEALTHY` as back to normal, so for these
|
||||
containers:
|
||||
- the error count never reset, and one-off failures days apart added up to a restart;
|
||||
- no "recovered" alert was sent;
|
||||
- with *Restart, then redeploy*, the next unrelated problem skipped the restart and
|
||||
redeployed the whole stack.
|
||||
|
||||
Now `NO HEALTH CHECK` counts as working (unless you made it a restart trigger): the
|
||||
count resets, the "recovered" alert is sent ("back to running (no health check)"),
|
||||
and the next problem starts with a plain restart. Containers with a health check
|
||||
behave exactly as before.
|
||||
|
||||
---
|
||||
|
||||
## [0.9.3] — 2026-10-28
|
||||
|
||||
### Changed
|
||||
|
||||
- **Every time follows the `TZ` of the Inspector stack.** Before, the dashboard,
|
||||
Events page and alerts were always in US Pacific time while the maintenance window
|
||||
and logs used `TZ`, so anyone outside Pacific saw times hours off (e.g. a 02:00
|
||||
Berlin window shown as 5:00 PM). Now the dashboard, the Events page (including
|
||||
where each day starts), alert times, cooldown times and logs all use `TZ`.
|
||||
- Alert times include the zone abbreviation (`10-28 09:27 AM CET`), and "Cooldown
|
||||
until" in alerts, event messages and logs is a local time instead of a UTC
|
||||
timestamp.
|
||||
- The active time zone is shown at the bottom of every Settings page, on the Events
|
||||
page and on the Maintenance page.
|
||||
- **If you never set `TZ`**, times are now shown in UTC instead of Pacific. Set `TZ`
|
||||
in the stack (e.g. `America/New_York`). An unrecognized name logs a warning and
|
||||
falls back to UTC.
|
||||
|
||||
### Docs
|
||||
|
||||
- INSTALL: `TZ` is now marked as needed, with a "Why the time zone matters" section
|
||||
(maintenance window, alerts, dashboard, logs), how to find your zone name, and a
|
||||
troubleshooting row. Guide: a Time zone section and a first-login check. Compose
|
||||
and README updated.
|
||||
|
||||
---
|
||||
|
||||
## [0.9.2] — 2026-10-28
|
||||
|
||||
Fixes from a code and security review. None of them change how containers are
|
||||
monitored or recovered.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A bad value saved from Settings can no longer stop Inspector from starting.**
|
||||
The forms now reject invalid times (maintenance start/end), unknown log levels and
|
||||
out-of-range numbers (for example `interval_seconds: 0`, which made the loop spin).
|
||||
If `config.yml` already has such a value, Inspector starts with the default and
|
||||
logs a warning instead of crashing in a restart loop.
|
||||
- **The dashboard password can no longer be cleared.** Saving an empty password (or
|
||||
username) is refused, and an empty password never signs anyone in. Before, clearing
|
||||
it left the running dashboard open to anyone.
|
||||
- **Passwords with non-ASCII characters (é, ü, £ …) now work.** Before, every login
|
||||
attempt with one crashed.
|
||||
- The Events page no longer errors for a moment after a container is removed.
|
||||
- The HTTP check hint no longer suggests `302` as an expected status; redirects are
|
||||
followed, so the final page's code is what counts.
|
||||
|
||||
### Security
|
||||
|
||||
- **Sign-in throttling:** after 5 failed sign-ins (form or Basic Auth) from one address
|
||||
within 5 minutes, that address is refused for the rest of the window.
|
||||
- **Changing the dashboard username or password signs out existing sessions.**
|
||||
Expired sessions are cleaned up.
|
||||
- Request bodies are limited to 1 MB; an invalid `Content-Length` gets a 400.
|
||||
- The Gotify token is sent in the `X-Gotify-Key` header instead of the URL, so it no
|
||||
longer appears in Gotify or reverse-proxy access logs.
|
||||
- Portainer token environment variables must be named `PORTAINER_…` (a name already in
|
||||
your `config.yml` keeps working), so Test connection can't be used to send other
|
||||
container secrets to an arbitrary URL.
|
||||
- Single quotes are escaped in HTML, and the delete confirmations build their text safely.
|
||||
- Two settings saves at the same moment can no longer overwrite each other.
|
||||
- `requests` updated to 2.32.5 (CVE-2024-47081).
|
||||
|
||||
---
|
||||
|
||||
## [0.9.1] — 2026-10-28
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A dependency on a disabled container no longer blocks recovery forever.** If
|
||||
`depends_on` named a container that is disabled in Inspector, the dependent app was
|
||||
never restarted. Inspector now only waits for dependencies it is watching, and the
|
||||
container page warns when the selected dependency is disabled.
|
||||
- **`PORTAINER_API_TOKEN` now reaches the container.** The public `compose.yml` didn't
|
||||
pass it in, so setting it as a Portainer stack variable had no effect. Existing
|
||||
stacks need the line `- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}` added under
|
||||
`environment:` (Stacks → inspector → Editor).
|
||||
- The "depends_on … is disabled or not found" warning was logged on every check
|
||||
(since 0.8.0). It is now logged once, and again only when the container settings
|
||||
change.
|
||||
|
||||
### Changed
|
||||
|
||||
- Guide: new sections **6. Apps behind a VPN container** (why depends_on matters, how to
|
||||
spot a VPN-routed app, setup, what Inspector does) and **7. Portainer stack redeploy**
|
||||
(why it's needed, what a redeploy does to the whole stack, requirements, where to find
|
||||
every value, adding the token, the other settings, Test connection errors).
|
||||
- Clearer form hints: `vpn_routed` only affects logging; `depends_on` works for any
|
||||
dependency; where to find the endpoint ID.
|
||||
|
||||
---
|
||||
|
||||
## [0.9.0] — 2026-10-28
|
||||
|
||||
### Added
|
||||
|
||||
- **Every container option is now in the dashboard.** The container page has three
|
||||
new groups:
|
||||
- **Reachability**: HTTP check (URL, timeout, expected status) and Uptime Kuma
|
||||
monitor.
|
||||
- **Dependencies**: `depends_on` (pick from your containers) and `vpn_routed`.
|
||||
- **Recovery**: `recovery_action` with plain-language descriptions, plus the
|
||||
Portainer environment and stack name for stack redeploys.
|
||||
- **Integrations settings.** New **Uptime Kuma** and **Portainer** pages under
|
||||
Settings. Portainer environments can be added, edited, renamed (containers
|
||||
that use them are updated) and deleted (refused while a container uses one).
|
||||
The API token can be an environment variable name (recommended) or a masked
|
||||
value saved in `config.yml`.
|
||||
- **Test connection** buttons for Uptime Kuma (lists monitors) and each Portainer
|
||||
environment (lists stacks on the endpoint), using the values in the form even
|
||||
before saving.
|
||||
- The All containers grid shows each container's extras (HTTP check, Uptime Kuma,
|
||||
dependency, recovery action).
|
||||
- Guide: new sections on reachability checks and apps behind a VPN container.
|
||||
|
||||
### Changed
|
||||
|
||||
- Uptime Kuma and Portainer changes take effect on the next check; Inspector
|
||||
rebuilds those clients when their settings change instead of needing a restart.
|
||||
- Saving a container form unchanged now leaves `config.yml` byte-identical. Blank
|
||||
optional fields are removed instead of written as empty values (`color: ''`,
|
||||
`maintenance_window: false`), and an unchanged `restart_on` list keeps its
|
||||
order and the comments after it. An explicit `recovery_action: restart_container`
|
||||
is kept.
|
||||
|
||||
---
|
||||
|
||||
## [0.8.0] — 2026-10-27
|
||||
|
||||
### Added
|
||||
|
||||
- **Install from Portainer with no command line.** Mount one folder per purpose
|
||||
(`/app/config`, `/app/logs`, `/app/data`) and deploy. If there's no config yet,
|
||||
Inspector writes a starter `config.yml` (dashboard on, event history on, no
|
||||
containers). Dashboard login comes from `INSPECTOR_USERNAME` / `INSPECTOR_PASSWORD`;
|
||||
without a password, one is generated and printed once in the container logs.
|
||||
- **Docker socket access is automatic.** A new entrypoint starts as root only long
|
||||
enough to read the socket's group and fix ownership of the mounted folders, then
|
||||
runs Inspector as `PUID:PGID` (default `1000:1000`) with that group added.
|
||||
`DOCKER_GID` / `group_add` are no longer needed.
|
||||
- **In-app guide** at `/guide/` with screenshots, linked from a new **Help** tab
|
||||
in the header. Also in the repo at `src/inspector/assets/guide/guide.html`.
|
||||
- **`maintenance_window` switch** on the container form, so containers can opt in
|
||||
to the daily maintenance window without editing YAML.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Containers added from the UI are now watched on the next check. Before, the
|
||||
loop only logged that a restart was needed, despite what the UI said.
|
||||
Containers removed or disabled from the UI are dropped the same way.
|
||||
- Status page could fail to render while a container was being added or removed
|
||||
(it iterated the live state dict and assumed every entry was still in config).
|
||||
|
||||
### Changed
|
||||
|
||||
- Config path: `--config`, then `$INSPECTOR_CONFIG`, then `/app/config.yml` if
|
||||
present (existing single-file setups keep working unchanged), then
|
||||
`/app/config/config.yml`.
|
||||
- Public `compose.yml`, `INSTALL.md`, and `README.md` rewritten for Portainer
|
||||
stacks; README gains the logo and screenshots.
|
||||
|
||||
### Upgrade notes
|
||||
|
||||
- Existing deployments need no changes. To adopt the simpler layout, move
|
||||
`config.yml` into the new config folder and switch to the new compose file.
|
||||
|
||||
---
|
||||
|
||||
## [0.7.4] — 2026-10-27
|
||||
|
||||
### Changed
|
||||
|
||||
- **Settings rail container list is compact.** Per-container entries are indented
|
||||
under "All containers" and fit on one line: colored dot, name, then enabled /
|
||||
disabled in small muted text. The max-error stat is gone. Long names truncate
|
||||
with an ellipsis, and the status stays visible.
|
||||
|
||||
---
|
||||
|
||||
## [0.7.3] — 2026-10-27
|
||||
|
||||
### Changed
|
||||
|
||||
- **Containers moved into Settings.** The top-nav Containers tab is gone; the
|
||||
settings rail now has an "All containers" entry (the tile grid at `/containers`)
|
||||
plus one entry per container (`/containers/<name>`). Both views render inside
|
||||
the same rail layout, and the Settings tab stays highlighted while you browse them.
|
||||
- **Per-channel priority reference.** The shared priority block is replaced by one
|
||||
reference per platform: Gotify (0 silent · 1–3 low · 4–7 normal · 8–10 high) and
|
||||
Pushover (-2 Lowest · -1 Low · 0 Normal · 1 High · 2 Emergency), with the
|
||||
Inspector → Pushover mapping noted under the Pushover one.
|
||||
- **restart_on redesign.** The pill row is replaced by a grid of toggleable cards,
|
||||
each with a color accent and a plain-language description of the status, plus
|
||||
"Recommended" (UNHEALTHY + NOT RUNNING) and "Observe only" quick buttons.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Unchecking every `restart_on` status now saves `restart_on: []` (observe only).
|
||||
Previously an all-unchecked submit silently kept the old list.
|
||||
|
||||
---
|
||||
|
||||
## [0.7.2] — 2026-10-27
|
||||
|
||||
### Added
|
||||
|
||||
- **Container CRUD from the UI.** `/containers/new` adds a new container to
|
||||
`config.yml`; a delete button on `/containers/<name>` removes one. The config
|
||||
change is picked up on the next polling cycle without a restart.
|
||||
- **Test-notification buttons** on the Gotify and Pushover sub-sections. The
|
||||
button posts the current (unsaved) form values to `POST /api/test-notify/<channel>`
|
||||
and shows the result inline. Unlike the regular notifier it does not swallow
|
||||
errors — HTTP failures are reported back to the UI.
|
||||
- **Priority reference** block on the Notifications section, including the
|
||||
Inspector → Pushover `-2..2` mapping.
|
||||
- **Config diagnostics strip** at the bottom of every settings page: shows the
|
||||
live `config.yml` path, whether it's writable, and the Inspector version. If
|
||||
the bind mount is read-only (a common deployment misconfiguration — a `:ro`
|
||||
mount means saves fail silently) a red `READ-ONLY` badge appears with a hint.
|
||||
- **Containers listed in the settings left rail** under their own group, plus a
|
||||
"+ Add container" link, so you can jump between sections and containers from
|
||||
the same navigation.
|
||||
- **Color input upgrade** on container edit / new forms: ten preset color swatches,
|
||||
a native color picker, and the existing hex text field, all bound together.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Toggle switches** replace the stock checkbox + "requires restart" layout for
|
||||
all `enabled`-style fields (notifications, dashboard, database, health summary,
|
||||
maintenance, notification mute, container enabled, Inspector `dry_run` /
|
||||
`heartbeat`). Reads much cleaner.
|
||||
- **Status page** no longer prints "auto-refreshes every 30s" (the `<meta refresh>`
|
||||
still ticks every 30s).
|
||||
|
||||
### Internal
|
||||
|
||||
- `_apply_container_create()` + `_apply_container_delete()` helpers alongside the
|
||||
existing `_apply_settings_update()`; container names are validated against
|
||||
`^[a-zA-Z0-9][a-zA-Z0-9_-]{0,62}$` and rejected if duplicate.
|
||||
- `_render_diag_strip()` uses `os.access(path, os.W_OK)` to detect read-only mounts.
|
||||
- Test-notification bypasses `GotifyNotifier.notify()` / `PushoverNotifier.notify()`
|
||||
so HTTP failures are surfaced (the real notifier intentionally swallows them to
|
||||
avoid breaking the inspector loop).
|
||||
|
||||
---
|
||||
|
||||
## [0.7.1] — 2026-10-27
|
||||
|
||||
### Changed
|
||||
|
||||
- **Settings page redesigned** to a classic SaaS layout (left rail + content pane):
|
||||
- Left rail groups sections under Core / Data / Scheduling with a live subtitle
|
||||
under each item (e.g. "60s · dry-run on", "gotify on · pushover off",
|
||||
"on · :8080") so you can read state without clicking in.
|
||||
- Each section is URL-addressable: `/settings/inspector`,
|
||||
`/settings/notifications`, `/settings/dashboard`, `/settings/database`,
|
||||
`/settings/health_summary`, `/settings/maintenance`, `/settings/notification_mute`.
|
||||
- Gotify and Pushover now share a single **Notifications** section with
|
||||
one Save button per channel, so the fan-out model is visible at a glance.
|
||||
- Save flash is a prominent green banner with a success check.
|
||||
- **Containers moved to a dedicated top-level route** (`/containers`): a grid
|
||||
of tiles showing status, enabled, max errors, cooldown, and restart_on at a
|
||||
glance. Click a tile for the per-container edit form at `/containers/<name>`.
|
||||
Keeps the Settings page small when buddies have many containers.
|
||||
- Top navigation adds a **Containers** tab alongside Status / Events / Settings.
|
||||
|
||||
### Internal
|
||||
|
||||
- `POST /settings/gotify` and `POST /settings/pushover` now redirect back to
|
||||
`/settings/notifications` (the grouped page).
|
||||
- `POST /containers/<name>` added as a parallel entry point to the existing
|
||||
settings backend for container edits.
|
||||
|
||||
---
|
||||
|
||||
## [0.7.0] — 2026-10-26
|
||||
|
||||
### Added
|
||||
|
||||
- **Public packaging** under `euronvault/inspector` on Gitea with a
|
||||
dedicated image at `gitea.euronvault.com/euronvault/inspector:0.7.0`
|
||||
(plus `:latest` and `:sha-<short>` tags). Dev continues on
|
||||
`eugene/inspector`.
|
||||
- `scripts/sync_public.sh` with an explicit allowlist
|
||||
(`scripts/public_allowlist.txt`) and grep-based denylist
|
||||
(`scripts/public_denylist_patterns.txt`) that refuses to publish any
|
||||
file matching a personal-string pattern (lab IPs, lab container
|
||||
names, lab env var names, agent-workflow scaffolding paths).
|
||||
- Sanitized `public/` staging tree: generic README, INSTALL guide,
|
||||
MIT LICENSE (Copyright (c) 2026 Europium 63), `config.example.yml`
|
||||
with two neutral example containers, and a `compose.yml` that pulls
|
||||
from `gitea.euronvault.com/euronvault/inspector`.
|
||||
- **Web settings page** at `/settings` for editing `config.yml` without SSH:
|
||||
per-section forms for Inspector loop, Gotify, Pushover, Dashboard,
|
||||
Database, Health summary, Maintenance, Notification mute, and per-container
|
||||
|
||||
@@ -19,9 +19,14 @@ ARG INSPECTOR_UID=1000
|
||||
ARG INSPECTOR_GID=1000
|
||||
RUN groupadd --gid ${INSPECTOR_GID} inspector \
|
||||
&& useradd --uid ${INSPECTOR_UID} --gid inspector --home-dir /app --shell /usr/sbin/nologin inspector \
|
||||
&& mkdir -p /app/logs /app/data \
|
||||
&& chown -R inspector:inspector /app/logs /app/data
|
||||
&& mkdir -p /app/config /app/logs /app/data \
|
||||
&& chown -R inspector:inspector /app/config /app/logs /app/data
|
||||
|
||||
USER inspector
|
||||
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
|
||||
RUN chmod 755 /usr/local/bin/docker-entrypoint.sh
|
||||
|
||||
CMD ["python", "-m", "inspector.main", "--config", "/app/config.yml"]
|
||||
# Starts as root only to detect the Docker socket's group and fix mounted
|
||||
# folder ownership, then drops to PUID:PGID (default 1000:1000).
|
||||
# See docker-entrypoint.sh.
|
||||
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
|
||||
CMD ["python", "-m", "inspector.main"]
|
||||
@@ -1,70 +1,164 @@
|
||||
# Install
|
||||
# Installing Inspector with Portainer
|
||||
|
||||
Inspector watches Docker containers, restarts the ones you tell it to,
|
||||
and notifies you via Gotify and/or Pushover. All configuration lives in
|
||||
a single `config.yml` that you can edit by hand or from the built-in web
|
||||
settings page.
|
||||
This guide uses only the Portainer web UI. You don't need SSH, a terminal, or a
|
||||
config file: Inspector creates its own settings on first start and everything
|
||||
else is done in its dashboard.
|
||||
|
||||
## Requirements
|
||||
**You need:** a Docker host managed by Portainer (CE or BE, 2.x), and a few
|
||||
minutes. Gotify or Pushover are optional, for phone alerts.
|
||||
|
||||
- Docker Engine 24+ and `docker compose` v2.
|
||||
- A machine that can reach `/var/run/docker.sock` (any Docker host).
|
||||
- Optional: a Gotify server and/or a Pushover account for notifications.
|
||||
---
|
||||
|
||||
## Quickstart
|
||||
## 1. Create the stack
|
||||
|
||||
```bash
|
||||
# 1. Grab the compose file and example config
|
||||
mkdir -p ~/docker/inspector && cd ~/docker/inspector
|
||||
curl -L -o compose.yml https://gitea.euronvault.com/euronvault/inspector/raw/branch/main/compose.yml
|
||||
curl -L -o config.yml https://gitea.euronvault.com/euronvault/inspector/raw/branch/main/config.example.yml
|
||||
1. In Portainer, open your environment (usually **local**) and click
|
||||
**Stacks → + Add stack**.
|
||||
2. **Name:** `inspector`
|
||||
3. **Build method:** *Web editor*. Paste the contents of
|
||||
[`compose.yml`](./compose.yml):
|
||||
|
||||
# 2. Edit config.yml — at minimum, list the containers you want watched
|
||||
# under the `containers:` key.
|
||||
$EDITOR config.yml
|
||||
```yaml
|
||||
services:
|
||||
inspector:
|
||||
image: gitea.euronvault.com/euronvault/inspector:${INSPECTOR_TAG:-latest}
|
||||
container_name: inspector
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${INSPECTOR_PORT:-8080}:8080"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- ${INSPECTOR_DATA:-/opt/inspector}/config:/app/config
|
||||
- ${INSPECTOR_DATA:-/opt/inspector}/logs:/app/logs
|
||||
- ${INSPECTOR_DATA:-/opt/inspector}/data:/app/data
|
||||
environment:
|
||||
- TZ=${TZ:-UTC}
|
||||
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
|
||||
- PUID=${PUID:-1000}
|
||||
- PGID=${PGID:-1000}
|
||||
```
|
||||
|
||||
# 3. Set your Docker group GID so the container can read the socket.
|
||||
echo "DOCKER_GID=$(getent group docker | cut -d: -f3)" > .env
|
||||
## 2. Set the environment variables
|
||||
|
||||
# 4. Pull and run
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
docker compose logs -f inspector
|
||||
Scroll down to **Environment variables**. Click **Advanced mode** and paste,
|
||||
then edit the values:
|
||||
|
||||
```
|
||||
INSPECTOR_DATA=/mnt/user/appdata/inspector
|
||||
INSPECTOR_PASSWORD=pick-a-password
|
||||
TZ=America/Los_Angeles
|
||||
```
|
||||
|
||||
If `dashboard.enabled: true` in your config, visit
|
||||
`http://<host>:8080/` after startup to see the dashboard, and
|
||||
`http://<host>:8080/settings` to edit the config from the browser.
|
||||
| Variable | What to put | Required |
|
||||
|---|---|---|
|
||||
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
|
||||
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
|
||||
| `TZ` | **Your time zone**, e.g. `America/New_York` or `Europe/Berlin`. Without it Inspector uses UTC, so the maintenance window, alerts and dashboard times are off by your UTC offset. See [Why the time zone matters](#why-the-time-zone-matters). | **Yes** (default `UTC`) |
|
||||
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
|
||||
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
|
||||
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
|
||||
| `PORTAINER_API_TOKEN` | A Portainer access token. Only needed if you use **Portainer stack redeploy** for VPN-routed apps; see the in-app guide (Help → Portainer). | No |
|
||||
|
||||
## Pinning a version
|
||||
### Why the time zone matters
|
||||
|
||||
Prefer an immutable tag over `latest` in production:
|
||||
Inspector uses `TZ` for every time it shows or acts on. If it's wrong, things
|
||||
still work, but at the wrong hour or with confusing times:
|
||||
|
||||
```bash
|
||||
# .env
|
||||
INSPECTOR_TAG=0.7.0
|
||||
```
|
||||
- **The maintenance window runs in this time zone.** A window of `02:00–05:00`
|
||||
with `TZ` left at UTC runs at 02:00 UTC, which is the middle of the evening in
|
||||
the Americas or mid-morning in Asia. Inspector would then restart containers
|
||||
during your nightly backups, and ignore real problems during the day.
|
||||
- **Alerts** (Gotify / Pushover) show the time of the problem in this zone, with
|
||||
its abbreviation, e.g. `10-28 03:15 AM CET`.
|
||||
- **The dashboard and Events page** show times in this zone, and the Events
|
||||
page's days run midnight to midnight in this zone.
|
||||
- **Logs** are stamped in this zone.
|
||||
|
||||
Images are published at `gitea.euronvault.com/euronvault/inspector`.
|
||||
**How to set it:** use your zone's name from the
|
||||
[list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
|
||||
("TZ identifier" column), e.g. `America/New_York`, `America/Chicago`,
|
||||
`Europe/London`, `Europe/Berlin`, `Australia/Sydney`. Abbreviations like `EST` or
|
||||
`CET` don't work. On Unraid, **Settings → Date and Time** shows the zone your
|
||||
server uses.
|
||||
|
||||
## Upgrading
|
||||
**How to check it:** the bottom of every Settings page shows
|
||||
`time zone: <name>`. If it says `UTC` and you're not on UTC, set `TZ` in the
|
||||
stack and click **Update the stack**. If the name you set isn't recognized,
|
||||
Inspector logs a warning and uses UTC.
|
||||
|
||||
```bash
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
## 3. Deploy and sign in
|
||||
|
||||
Your `config.yml` is persisted outside the container, so upgrades don't
|
||||
touch it. The settings page keeps a `.bak` of the previous file next to
|
||||
`config.yml` on every save (when the directory is bind-mounted; see the
|
||||
README for the Docker caveat).
|
||||
1. Click **Deploy the stack**.
|
||||
2. Open `http://<your-server-ip>:8080` (or the port you chose).
|
||||
3. Sign in as `admin` with your `INSPECTOR_PASSWORD`.
|
||||
- No password set? Go to **Containers → inspector → Logs** in Portainer. Near
|
||||
the top is a box labeled *Inspector dashboard login* with the generated
|
||||
password.
|
||||
4. Click **Help** in the dashboard header for the full how-to guide. The short
|
||||
version:
|
||||
- **Settings → + Add container**: add each container you want watched, using
|
||||
its exact name from Portainer, and click **Recommended** for restart rules.
|
||||
- **Settings → Notifications**: turn on Gotify and/or Pushover, click
|
||||
**Send test**, then **Save**.
|
||||
- Restart Inspector once after changing anything marked
|
||||
**requires restart** (Portainer → Containers → tick `inspector` → Restart).
|
||||
|
||||
## Uninstall
|
||||
That's it. Inspector picks up new containers on its next check (within a minute).
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
rm -rf logs data config.yml compose.yml .env
|
||||
```
|
||||
---
|
||||
|
||||
Inspector doesn't store anything outside the directory it was started
|
||||
from.
|
||||
## 4. Updating
|
||||
|
||||
1. **Stacks → inspector → Editor**.
|
||||
2. Click **Update the stack** and turn on **Re-pull image and redeploy**.
|
||||
3. Confirm.
|
||||
|
||||
Your settings and history live in `INSPECTOR_DATA`, so updates never touch them.
|
||||
|
||||
**Pinning a version.** `latest` always gets the newest release. To stay on a
|
||||
specific version, set `INSPECTOR_TAG` (e.g. `0.8.0`) in the stack's environment
|
||||
variables, then update the stack. Available versions are on the
|
||||
[releases page](https://gitea.euronvault.com/euronvault/inspector/releases)
|
||||
and in the [CHANGELOG](./CHANGELOG.md).
|
||||
|
||||
**Rolling back.** Set `INSPECTOR_TAG` to the previous version and update the
|
||||
stack with **Re-pull image and redeploy** on.
|
||||
|
||||
## 5. Uninstalling
|
||||
|
||||
1. **Stacks → inspector → Delete this stack**.
|
||||
2. Delete the `INSPECTOR_DATA` folder (on Unraid, through the appdata share) if
|
||||
you don't want to keep the settings and history.
|
||||
|
||||
Inspector never changes your other containers' files or settings. It only
|
||||
starts and restarts the ones you add to it.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| What you see | What to do |
|
||||
|---|---|
|
||||
| Stack fails with `pull access denied` or `not found` | Check the image line is exactly `gitea.euronvault.com/euronvault/inspector:...` and that `INSPECTOR_TAG` (if set) is a real version. |
|
||||
| Port already in use | Set `INSPECTOR_PORT` to a free port (e.g. `8095`) and update the stack. |
|
||||
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
|
||||
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
|
||||
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
|
||||
| Times are a few hours off, or the maintenance window runs at the wrong time | `TZ` is missing or wrong. Check `time zone:` at the bottom of Settings, set `TZ` to your zone name (e.g. `Europe/London`), and update the stack. |
|
||||
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
|
||||
|
||||
## Security notes
|
||||
|
||||
- Inspector needs the Docker socket to restart containers. Anything with access
|
||||
to that socket effectively has full control of the host, so keep the
|
||||
dashboard on your home network (or behind a VPN / reverse proxy with auth)
|
||||
and use a real password.
|
||||
- Inspector's process runs as an unprivileged user. It starts as root only long
|
||||
enough to detect the Docker socket's group and fix ownership of its own
|
||||
folders.
|
||||
|
||||
## Moving from an older (0.7.x) setup
|
||||
|
||||
Older setups mounted a single file at `/app/config.yml` and set `group_add` /
|
||||
`DOCKER_GID`. That still works unchanged. To switch to the simpler layout:
|
||||
move your `config.yml` into `<INSPECTOR_DATA>/config/`, replace the stack with
|
||||
the compose above, and update the stack.
|
||||
@@ -1,139 +1,109 @@
|
||||
# Inspector
|
||||
<p align="center">
|
||||
<img src="docs/logo_outline.png" alt="Inspector logo" width="140">
|
||||
</p>
|
||||
|
||||
Inspector is a small, opinionated Docker-watchdog. It polls the Docker
|
||||
daemon, applies per-container restart rules, pings Gotify and/or
|
||||
Pushover when things go wrong, and gives you a dashboard that shows
|
||||
what's happening right now and what happened earlier today.
|
||||
<h1 align="center">Inspector</h1>
|
||||
|
||||
Built for home-lab and small-fleet operators who want "it just works"
|
||||
container recovery without standing up a full observability stack.
|
||||
<p align="center"><b>Monitor. Restart. Report.</b><br>
|
||||
A small Docker watchdog for home servers: it watches the containers you pick,
|
||||
restarts them when they break, and tells your phone about it.</p>
|
||||
|
||||
## What Inspector does
|
||||
<p align="center">
|
||||
<a href="./INSTALL.md">Install with Portainer</a> ·
|
||||
<a href="./src/inspector/assets/guide/guide.html">How-to guide</a> ·
|
||||
<a href="./CHANGELOG.md">Changelog</a>
|
||||
</p>
|
||||
|
||||
- Polls each configured container on a fixed interval and classifies it:
|
||||
`HEALTHY`, `UNHEALTHY`, `STARTING`, `NOT RUNNING`, `ERROR`,
|
||||
`UNREACHABLE`, `CONTAINER NOT FOUND`, `NO HEALTH CHECK`, `UNKNOWN`.
|
||||
- Restarts a container when it crosses a configured error threshold —
|
||||
with cooldowns, grace periods, and an hourly restart cap so a flapping
|
||||
container doesn't DDoS your host.
|
||||
- Opt-in HTTP reachability checks per container (useful for VPN-routed
|
||||
containers whose tunnel drops while Docker still reports "healthy").
|
||||
- Opt-in Uptime Kuma integration as a second reachability source.
|
||||
- Advanced recovery via Portainer stack redeploy for containers whose
|
||||
network namespace is tied to a VPN sidecar.
|
||||
- Sends notifications on transitions, not on every poll — one line when
|
||||
something breaks, one line when it recovers.
|
||||
- Optional SQLite event history + a daily/weekly digest.
|
||||
- Scheduled maintenance windows (don't act during 2–6 AM).
|
||||
- Manual notification mute per container, from the dashboard.
|
||||
- Embedded web dashboard + events timeline + **web settings page** for
|
||||
editing `config.yml` without SSH.
|
||||

|
||||
|
||||
## What Inspector is not
|
||||
## What it does
|
||||
|
||||
- Not a metrics system. It doesn't scrape Prometheus or push
|
||||
dashboards; use Grafana / Netdata / your real monitoring stack for
|
||||
that.
|
||||
- Not an orchestrator. It only acts on containers you've explicitly
|
||||
listed in `config.yml`.
|
||||
- Not multi-host. One Inspector per Docker socket.
|
||||
- **Watches the containers you choose.** Each check sorts a container into
|
||||
healthy, unhealthy, not running, starting, and so on.
|
||||
- **Restarts them when they break**, with sensible brakes: a container has to fail
|
||||
several checks in a row first, there's a cooldown between restarts, a grace
|
||||
period while it boots, and an hourly cap so a crash-looping app can't take over
|
||||
your server.
|
||||
- **Alerts you** through [Gotify](https://gotify.net) and/or
|
||||
[Pushover](https://pushover.net): once when something breaks, once when it's
|
||||
restarted, once when it recovers. Not on every check.
|
||||
- **Is set up entirely in the browser.** Add, edit, and remove containers, set up
|
||||
notifications (with a **Send test** button), and change every setting from the
|
||||
dashboard. No YAML editing needed.
|
||||
- **Keeps a history.** The Events page shows a day-by-day timeline of failures,
|
||||
restarts, and recoveries, and an optional digest summarizes it for you.
|
||||
- **Stays out of the way when you're working.** Put a container in maintenance or
|
||||
mute its alerts with one click, or set a nightly maintenance window for backups
|
||||
and updates.
|
||||
- **Works in your time zone.** Set `TZ` in the stack and the maintenance window,
|
||||
alerts, dashboard and logs all follow it.
|
||||
|
||||
## Install
|
||||
|
||||
See [`INSTALL.md`](./INSTALL.md). TL;DR:
|
||||
Inspector is built to be installed from **Portainer**: paste one compose file into
|
||||
**Stacks → Add stack**, set a folder and a password, deploy. Inspector creates its
|
||||
own config and sets up Docker access by itself.
|
||||
|
||||
```bash
|
||||
mkdir -p ~/docker/inspector && cd ~/docker/inspector
|
||||
curl -L -o compose.yml https://gitea.euronvault.com/euronvault/inspector/raw/branch/main/compose.yml
|
||||
curl -L -o config.yml https://gitea.euronvault.com/euronvault/inspector/raw/branch/main/config.example.yml
|
||||
$EDITOR config.yml
|
||||
echo "DOCKER_GID=$(getent group docker | cut -d: -f3)" > .env
|
||||
docker compose up -d
|
||||
```
|
||||
**→ [Step-by-step Portainer install](./INSTALL.md)**
|
||||
|
||||
## Configuration
|
||||
## Using it
|
||||
|
||||
Everything lives in `config.yml` next to the compose file. Start from
|
||||
[`config.example.yml`](./config.example.yml) — every key is commented.
|
||||
Highlights:
|
||||
Once it's running, click **Help** in the dashboard header for the illustrated
|
||||
guide (the same guide is in this repo at
|
||||
[`src/inspector/assets/guide/guide.html`](./src/inspector/assets/guide/guide.html);
|
||||
download it and open it in a browser to read it here).
|
||||
|
||||
- `inspector.interval_seconds` — poll cadence.
|
||||
- `inspector.dry_run` — classify + log, never restart.
|
||||
- `notifications.gotify` / `notifications.pushover` — enable either,
|
||||
both, or neither.
|
||||
- `containers.<name>.max_error_count` / `cooldown_seconds` /
|
||||
`grace_period_seconds` / `max_restarts_per_hour` — the restart knobs.
|
||||
- `containers.<name>.restart_on` — which statuses trigger a restart
|
||||
(leave empty for observe-only).
|
||||
- `containers.<name>.http_check` — opt-in HTTP ping.
|
||||
- `containers.<name>.recovery_action` — `restart_container` (default)
|
||||
or the advanced Portainer-based options.
|
||||
| Add a container | Restart rules |
|
||||
| ------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
||||
|  |  |
|
||||
|
||||
### Dashboard and settings page
|
||||
| Notifications | Events history |
|
||||
| ------------------------------------------------------------------------------- | --------------------------------------------------------- |
|
||||
|  |  |
|
||||
|
||||
If `dashboard.enabled: true`, Inspector exposes a small web UI on
|
||||
`:8080`:
|
||||
## What it isn't
|
||||
|
||||
- `/` — container cards with current status, error bar, phase
|
||||
(cooldown / grace / failing / observe only / maintenance), and
|
||||
maintenance / mute toggles.
|
||||
- `/events` — per-day timeline of status changes, restarts, recoveries,
|
||||
and failures (requires `database.enabled: true`).
|
||||
- `/settings` — forms for every top-level config section plus one card
|
||||
per container. Saves preserve comments and key order in `config.yml`
|
||||
and keep a `.bak`. Fields that take effect on the next loop iteration
|
||||
(per-container thresholds, `dry_run`, `heartbeat`, `interval_seconds`)
|
||||
hot-apply; fields bound at startup are labeled **requires restart**.
|
||||
- **Not a monitoring/metrics system.** No graphs of CPU or memory; use Grafana,
|
||||
Netdata, or Uptime Kuma for that.
|
||||
- **Not an orchestrator.** It only touches the containers you add, and only to
|
||||
restart them.
|
||||
- **One Docker host per Inspector.** Run one copy on each server.
|
||||
|
||||
Note on `.bak` inside Docker: when `config.yml` is bind-mounted as a
|
||||
single file (the compose default), the `.bak` lives in the container's
|
||||
overlay fs — the live file still updates cleanly, but the backup is
|
||||
lost when the container is removed. To keep the `.bak` on the host,
|
||||
bind-mount the directory instead of the single file.
|
||||
## Advanced options
|
||||
|
||||
### Notifications
|
||||
Most people never need these. They are all on each container's page in the
|
||||
dashboard (and in `config.yml`, see
|
||||
[`config.example.yml`](./config.example.yml) for every option with comments).
|
||||
|
||||
Both channels are independent. If both are enabled, every alert goes to
|
||||
both. Inspector sends on transitions:
|
||||
- **HTTP checks**: mark a container `UNREACHABLE` when a URL stops answering,
|
||||
even if Docker says it's healthy (useful for apps behind a VPN container).
|
||||
- **Uptime Kuma**: use a Kuma status page as an extra reachability signal.
|
||||
**Test connection** lists the monitors Inspector can see.
|
||||
- **Dependencies** (`depends_on`): don't restart an app while the container it
|
||||
relies on (e.g. a VPN) is down.
|
||||
- **Portainer stack redeploy**: for containers that share a VPN container's
|
||||
network, recover by redeploying the whole stack instead of restarting one
|
||||
container. Set up the Portainer connection under **Settings → Portainer**;
|
||||
**Test connection** lists your stacks.
|
||||
|
||||
```
|
||||
🚨 first failure threshold crossed
|
||||
🔁 container restart triggered
|
||||
✅ restart / start completed
|
||||
✅ container recovered
|
||||
🛑 restart failed / hourly limit exceeded
|
||||
❓ configured container not found
|
||||
```
|
||||
## Security
|
||||
|
||||
Gotify messages are Markdown; Pushover doesn't render Markdown so the
|
||||
`**bold**` passthrough is cosmetic — bodies are identical.
|
||||
Inspector needs the Docker socket (`/var/run/docker.sock`) to restart containers,
|
||||
and that socket is effectively root on your server. So:
|
||||
|
||||
Pushover priorities map from Inspector's Gotify-style scale (default 5)
|
||||
to Pushover's `-2..2` range: `<=2 → -1`, `3..4 → 0`, `5..6 → 1`,
|
||||
`>=7 → 2`. Priority 2 (emergency) ships default `retry` / `expire` and
|
||||
your Pushover client prompts for acknowledgement.
|
||||
- Keep the dashboard on your home network, or put it behind a VPN or a reverse
|
||||
proxy with authentication. Don't expose port 8080 to the internet.
|
||||
- Use a real dashboard password. Repeated failed sign-ins from one address are
|
||||
blocked for a few minutes.
|
||||
- Inspector itself runs as an unprivileged user; it starts as root only to detect
|
||||
the socket's permissions and fix ownership of its own folders.
|
||||
|
||||
## Docker socket security
|
||||
## Versions
|
||||
|
||||
Inspector talks to `/var/run/docker.sock`. That socket is effectively
|
||||
root on your host: anything that can write to it can start, stop,
|
||||
inspect, and (through bind mounts) read any file as root. Inspector
|
||||
mounts it read-write because it needs to restart containers.
|
||||
|
||||
Follow the usual home-lab precautions:
|
||||
|
||||
- Don't publish the dashboard port on an untrusted network. Use a
|
||||
reverse proxy with auth, a tailnet, or SSH tunneling.
|
||||
- Set a strong `dashboard.password`.
|
||||
- Store Portainer / Gotify / Pushover tokens in the config file on a
|
||||
host-only path with permissions `600`.
|
||||
|
||||
## Versioning
|
||||
|
||||
Semantic versioning (`MAJOR.MINOR.PATCH`). Image tags are published for
|
||||
every release; `latest` tracks the newest tag. Pin a specific version
|
||||
in production — see `INSTALL.md`.
|
||||
Releases follow `MAJOR.MINOR.PATCH`. Every release is published as an image tag;
|
||||
`latest` is the newest. See the [CHANGELOG](./CHANGELOG.md) for what changed, and
|
||||
[INSTALL.md](./INSTALL.md#4-updating) for updating, pinning, and rolling back.
|
||||
|
||||
## License
|
||||
|
||||
[MIT](./LICENSE).
|
||||
[MIT](./LICENSE) © 2026 Europium 63
|
||||
@@ -1,29 +1,43 @@
|
||||
# Inspector compose file for buddies / self-hosters.
|
||||
# Pulls the published image from gitea.euronvault.com.
|
||||
# Inspector — Portainer stack
|
||||
#
|
||||
# Before `docker compose up -d`:
|
||||
# 1. Copy config.example.yml -> config.yml and edit it.
|
||||
# 2. Set DOCKER_GID to your host's docker group GID:
|
||||
# getent group docker | cut -d: -f3
|
||||
# Export it in your shell or put it in a .env file next to this file.
|
||||
# 3. Optionally set INSPECTOR_TAG to pin a version
|
||||
# (default: latest). Prefer a pinned X.Y.Z for production.
|
||||
# Paste this into Portainer: Stacks -> Add stack -> Web editor.
|
||||
# Then set these under "Environment variables" (Advanced mode lets you paste them):
|
||||
#
|
||||
# INSPECTOR_DATA Folder on your server for Inspector's files.
|
||||
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
|
||||
# INSPECTOR_PASSWORD Dashboard password (username is "admin").
|
||||
# Only used on first start; change it later in Settings.
|
||||
# TZ Your time zone name, e.g. America/New_York, Europe/Berlin.
|
||||
# Important: the maintenance window, alerts, dashboard and
|
||||
# logs all use it. Left unset, everything is in UTC.
|
||||
#
|
||||
# Optional:
|
||||
# INSPECTOR_PORT Port for the dashboard on your server (default 8080).
|
||||
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
|
||||
# PUID / PGID User/group Inspector runs as (default 1000/1000).
|
||||
# Unraid users often prefer 99 / 100.
|
||||
# PORTAINER_API_TOKEN Portainer access token, only if you use stack redeploy
|
||||
# (Settings -> Portainer in the dashboard).
|
||||
#
|
||||
# No config file to create, no docker group ID to look up: Inspector creates
|
||||
# its config on first start and detects Docker socket permissions by itself.
|
||||
services:
|
||||
inspector:
|
||||
image: gitea.euronvault.com/euronvault/inspector:${INSPECTOR_TAG:-latest}
|
||||
container_name: inspector
|
||||
restart: unless-stopped
|
||||
# GID of the docker group on the host so the non-root user inside the
|
||||
# container can read /var/run/docker.sock. Required.
|
||||
group_add:
|
||||
- "${DOCKER_GID:-999}"
|
||||
ports:
|
||||
# Only published if dashboard.enabled is true in config.yml.
|
||||
- "${INSPECTOR_PORT:-8080}:8080"
|
||||
volumes:
|
||||
# Lets Inspector see and restart your containers. Required.
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- ./config.yml:/app/config.yml
|
||||
- ./logs:/app/logs
|
||||
- ./data:/app/data
|
||||
# Settings (config.yml), logs, and event history. Kept across updates.
|
||||
- ${INSPECTOR_DATA:-/opt/inspector}/config:/app/config
|
||||
- ${INSPECTOR_DATA:-/opt/inspector}/logs:/app/logs
|
||||
- ${INSPECTOR_DATA:-/opt/inspector}/data:/app/data
|
||||
environment:
|
||||
- TZ=${TZ:-UTC}
|
||||
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
|
||||
- PUID=${PUID:-1000}
|
||||
- PGID=${PGID:-1000}
|
||||
- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}
|
||||
@@ -1,5 +1,9 @@
|
||||
# Inspector example configuration.
|
||||
# Copy to config.yml, edit, and mount into the container at /app/config.yml.
|
||||
# Inspector example configuration: every option, with comments.
|
||||
#
|
||||
# You normally don't need this file. Inspector writes its own config.yml to
|
||||
# INSPECTOR_DATA/config/ on first start, and the dashboard edits it for you.
|
||||
# Use this as a reference for the advanced options that are YAML-only
|
||||
# (http_check, uptime_kuma, depends_on, Portainer stack redeploy).
|
||||
|
||||
inspector:
|
||||
interval_seconds: 60
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
#!/bin/sh
|
||||
# Inspector container entrypoint.
|
||||
#
|
||||
# Started as root (the default): find the group that owns the Docker socket,
|
||||
# make the mounted folders writable, then drop to an unprivileged user with
|
||||
# that group added. No DOCKER_GID / group_add needed.
|
||||
#
|
||||
# Started as non-root (compose `user:` / older group_add setups): run as-is.
|
||||
set -e
|
||||
|
||||
if [ "$(id -u)" != "0" ]; then
|
||||
exec "$@"
|
||||
fi
|
||||
|
||||
PUID="${PUID:-1000}"
|
||||
PGID="${PGID:-1000}"
|
||||
SOCK=/var/run/docker.sock
|
||||
|
||||
groups="$PGID"
|
||||
if [ -S "$SOCK" ]; then
|
||||
sock_gid="$(stat -c %g "$SOCK")"
|
||||
groups="$groups,$sock_gid"
|
||||
else
|
||||
echo "warning: $SOCK is not mounted; Inspector cannot see any containers" >&2
|
||||
fi
|
||||
|
||||
# Mounted folders are often created by Docker as root. Hand them (not their
|
||||
# contents) to the runtime user so Inspector can write config, logs and data.
|
||||
for d in /app/config /app/logs /app/data; do
|
||||
if [ -d "$d" ] && [ "$(stat -c %u "$d")" != "$PUID" ]; then
|
||||
chown "$PUID:$PGID" "$d" 2>/dev/null || echo "warning: could not chown $d" >&2
|
||||
fi
|
||||
done
|
||||
if [ -f /app/config/config.yml ] && [ "$(stat -c %u /app/config/config.yml)" != "$PUID" ]; then
|
||||
chown "$PUID:$PGID" /app/config/config.yml 2>/dev/null || true
|
||||
fi
|
||||
|
||||
exec setpriv --reuid="$PUID" --regid="$PGID" --groups="$groups" "$@"
|
||||
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 25 KiB |
@@ -1,5 +1,5 @@
|
||||
docker==7.1.0
|
||||
PyYAML==6.0.2
|
||||
requests==2.32.3
|
||||
requests==2.32.5
|
||||
colorlog==6.9.0
|
||||
ruamel.yaml==0.18.6
|
||||
@@ -1 +1 @@
|
||||
__version__ = "0.7.0"
|
||||
__version__ = "0.9.5"
|
||||
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 70 KiB |
|
After Width: | Height: | Size: 73 KiB |
|
After Width: | Height: | Size: 45 KiB |
|
After Width: | Height: | Size: 27 KiB |
@@ -0,0 +1,462 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Inspector — How to use</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0b1220; --card: #111c30; --card-2: #0f1829; --border: #1e293b;
|
||||
--ink: #e2e8f0; --ink-2: #cbd5e1; --muted: #8494a9;
|
||||
--blue: #60a5fa; --green: #34d399; --amber: #fbbf24; --red: #f87171; --violet: #a78bfa;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { margin: 0; background: var(--bg); color: var(--ink);
|
||||
font-family: Inter, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", sans-serif;
|
||||
font-size: 16px; line-height: 1.6; }
|
||||
a { color: var(--blue); }
|
||||
.top { position: sticky; top: 0; z-index: 5; background: #060b16; border-bottom: 1px solid var(--border); }
|
||||
.top .in { max-width: 1100px; margin: 0 auto; padding: 0.8rem 1rem; display: flex; align-items: center; gap: 1rem; }
|
||||
.top .brand { font-weight: 800; font-size: 1.05rem; }
|
||||
.top .brand span { color: var(--muted); font-weight: 400; margin-left: 0.4rem; }
|
||||
.top a.back { margin-left: auto; font-size: 0.9rem; text-decoration: none; border: 1px solid var(--border);
|
||||
padding: 0.35rem 0.8rem; border-radius: 8px; color: var(--ink-2); }
|
||||
.wrap { max-width: 1100px; margin: 0 auto; padding: 2rem 1rem 4rem; display: grid;
|
||||
grid-template-columns: 220px minmax(0, 1fr); gap: 2.5rem; }
|
||||
nav.toc { position: sticky; top: 4.5rem; align-self: start; font-size: 0.88rem; }
|
||||
nav.toc b { display: block; color: var(--muted); text-transform: uppercase; font-size: 0.7rem;
|
||||
letter-spacing: 0.08em; margin-bottom: 0.5rem; }
|
||||
nav.toc a { display: block; color: var(--ink-2); text-decoration: none; padding: 0.25rem 0; }
|
||||
nav.toc a:hover { color: var(--blue); }
|
||||
main { min-width: 0; }
|
||||
h1 { font-size: 1.9rem; margin: 0 0 0.4rem; line-height: 1.2; }
|
||||
.lede { color: var(--ink-2); font-size: 1.05rem; margin: 0 0 2rem; }
|
||||
section { margin-bottom: 3rem; scroll-margin-top: 4.5rem; }
|
||||
h2 { font-size: 1.35rem; margin: 0 0 0.6rem; padding-top: 0.5rem; border-top: 1px solid var(--border); }
|
||||
h3 { font-size: 1.02rem; margin: 1.4rem 0 0.4rem; }
|
||||
p, li { color: var(--ink-2); }
|
||||
code { background: var(--card-2); border: 1px solid var(--border); padding: 0.05em 0.35em;
|
||||
border-radius: 5px; font-size: 0.88em; color: var(--ink); }
|
||||
figure { margin: 1rem 0 1.4rem; }
|
||||
figure img { width: 100%; height: auto; display: block; border-radius: 12px; border: 1px solid var(--border); }
|
||||
figcaption { color: var(--muted); font-size: 0.82rem; margin-top: 0.4rem; }
|
||||
.steps { counter-reset: s; list-style: none; padding: 0; }
|
||||
.steps li { counter-increment: s; position: relative; padding-left: 2.3rem; margin-bottom: 0.7rem; }
|
||||
.steps li::before { content: counter(s); position: absolute; left: 0; top: 0.05rem; width: 1.6rem; height: 1.6rem;
|
||||
border-radius: 50%; background: rgba(96,165,250,0.15); color: var(--blue); font-weight: 700;
|
||||
font-size: 0.85rem; display: flex; align-items: center; justify-content: center; }
|
||||
.note { background: var(--card); border: 1px solid var(--border); border-left: 3px solid var(--blue);
|
||||
border-radius: 10px; padding: 0.75rem 1rem; margin: 1rem 0; color: var(--ink-2); }
|
||||
.note.warn { border-left-color: var(--amber); }
|
||||
table { width: 100%; border-collapse: collapse; margin: 0.8rem 0 1.2rem; font-size: 0.92rem; }
|
||||
th, td { text-align: left; padding: 0.5rem 0.7rem; border-bottom: 1px solid var(--border); vertical-align: top; }
|
||||
th { color: var(--muted); font-weight: 600; font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.04em; }
|
||||
td:first-child { white-space: nowrap; }
|
||||
.tag { display: inline-block; font-size: 0.72rem; font-weight: 700; padding: 0.05rem 0.45rem; border-radius: 999px; }
|
||||
.t-green { background: rgba(52,211,153,0.15); color: var(--green); }
|
||||
.t-red { background: rgba(248,113,113,0.15); color: var(--red); }
|
||||
.t-amber { background: rgba(251,191,36,0.15); color: var(--amber); }
|
||||
.t-blue { background: rgba(96,165,250,0.15); color: var(--blue); }
|
||||
.t-violet { background: rgba(167,139,250,0.15); color: var(--violet); }
|
||||
@media (max-width: 820px) {
|
||||
.wrap { grid-template-columns: 1fr; gap: 1rem; padding-top: 1.2rem; }
|
||||
nav.toc { position: static; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="top"><div class="in">
|
||||
<div class="brand">Inspector<span>How to use</span></div>
|
||||
<a class="back" href="/">Open dashboard</a>
|
||||
</div></div>
|
||||
|
||||
<div class="wrap">
|
||||
<nav class="toc">
|
||||
<b>On this page</b>
|
||||
<a href="#first-login">1. First login</a>
|
||||
<a href="#add">2. Add your containers</a>
|
||||
<a href="#status">3. The Status page</a>
|
||||
<a href="#notifications">4. Notifications</a>
|
||||
<a href="#containers">5. Managing containers</a>
|
||||
<a href="#vpn">6. Apps behind a VPN</a>
|
||||
<a href="#portainer">7. Portainer stack redeploy</a>
|
||||
<a href="#events">8. Events history</a>
|
||||
<a href="#settings">9. Other settings</a>
|
||||
<a href="#restart">10. "Requires restart" fields</a>
|
||||
<a href="#trouble">11. Troubleshooting</a>
|
||||
</nav>
|
||||
|
||||
<main>
|
||||
<h1>Using Inspector</h1>
|
||||
<p class="lede">Inspector watches the Docker containers you pick, restarts them when they break,
|
||||
and pings your phone when something goes wrong or recovers. Everything is set from this web
|
||||
dashboard; you never need to touch a config file.</p>
|
||||
|
||||
<section id="first-login">
|
||||
<h2>1. First login</h2>
|
||||
<p>When Inspector starts for the first time it creates its config file and a dashboard login.</p>
|
||||
<ol class="steps">
|
||||
<li>If you set <code>INSPECTOR_PASSWORD</code> in the Portainer stack, sign in with
|
||||
<code>admin</code> (or your <code>INSPECTOR_USERNAME</code>) and that password.</li>
|
||||
<li>If you didn't, Inspector generated a password. In Portainer open
|
||||
<b>Containers → inspector → Logs</b>. Near the top you'll see a box labeled
|
||||
<i>Inspector dashboard login</i> with the username and password.</li>
|
||||
<li>Open <code>http://<your-server>:8080</code> and sign in.</li>
|
||||
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
|
||||
<li>Check the <b>time zone</b>: at the bottom of any Settings page it says
|
||||
<code>time zone: …</code>. If that's not your zone, see <a href="#settings">Time zone</a>.</li>
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
<section id="add">
|
||||
<h2>2. Add your containers</h2>
|
||||
<p>Inspector only watches containers you add. Go to <b>Settings</b> and click
|
||||
<b>+ Add container</b> at the bottom of the left menu (or <b>+ New container</b> on the
|
||||
All containers page).</p>
|
||||
<figure><img src="container-new.png" alt="New container form">
|
||||
<figcaption>The New container form.</figcaption></figure>
|
||||
<ol class="steps">
|
||||
<li><b>name</b>: the container's name exactly as Portainer shows it
|
||||
(for example <code>sonarr</code>). If it doesn't match, the card shows
|
||||
<span class="tag t-amber">CONTAINER NOT FOUND</span>.</li>
|
||||
<li>Leave <b>enabled</b> on.</li>
|
||||
<li>Under <b>Restart when status is…</b> click <b>Recommended</b>
|
||||
(restarts on <code>UNHEALTHY</code> and <code>NOT RUNNING</code>), or
|
||||
<b>Observe only</b> if you just want alerts without restarts.</li>
|
||||
<li>Click <b>Create container</b>. Inspector starts watching it on its next check
|
||||
(within a minute by default). No restart needed.</li>
|
||||
</ol>
|
||||
<h3>What the numbers mean</h3>
|
||||
<table>
|
||||
<tr><th>Field</th><th>Meaning</th><th>Default</th></tr>
|
||||
<tr><td>max_error_count</td><td>How many bad checks in a row before Inspector restarts the container. Stops one blip from causing a restart.</td><td>5</td></tr>
|
||||
<tr><td>cooldown_seconds</td><td>Minimum time between two restarts of the same container.</td><td>300</td></tr>
|
||||
<tr><td>grace_period_seconds</td><td>After a restart, ignore failures for this long so the app can boot.</td><td>120</td></tr>
|
||||
<tr><td>max_restarts_per_hour</td><td>Safety cap. After this many restarts in an hour Inspector stops restarting and alerts you instead.</td><td>3</td></tr>
|
||||
<tr><td>maintenance_window</td><td>Follow the daily maintenance window from Settings → Maintenance.</td><td>off</td></tr>
|
||||
<tr><td>color</td><td>Card color on the dashboard. Pick a swatch or leave blank for automatic.</td><td>auto</td></tr>
|
||||
</table>
|
||||
<div class="note">With the default 60-second check interval and <code>max_error_count</code> 5, a
|
||||
container has to be broken for about five minutes before Inspector restarts it. Lower the count
|
||||
for faster recovery.</div>
|
||||
</section>
|
||||
|
||||
<section id="status">
|
||||
<h2>3. The Status page</h2>
|
||||
<figure><img src="status.png" alt="Status page">
|
||||
<figcaption>Status: one card per watched container.</figcaption></figure>
|
||||
<p>The top row counts containers, healthy ones, ones with issues, and ones in maintenance.
|
||||
Each card shows:</p>
|
||||
<ul>
|
||||
<li><b>Status badge</b>: what Docker reports right now, e.g.
|
||||
<span class="tag t-green">HEALTHY</span> <span class="tag t-red">NOT RUNNING</span>
|
||||
<span class="tag t-violet">NO HEALTH CHECK</span> (the image has no health check; that's normal
|
||||
for many apps and only means Inspector can tell running from stopped, not healthy from unhealthy).</li>
|
||||
<li><b>Errors bar</b>: bad checks in a row out of <code>max_error_count</code>. When it fills,
|
||||
Inspector restarts the container.</li>
|
||||
<li><b>Cooldown until / Last restart</b>: when Inspector last restarted it and when it's allowed to
|
||||
again.</li>
|
||||
<li><b>Observe only</b>: shown when the container has no restart triggers; Inspector alerts but
|
||||
never restarts it.</li>
|
||||
<li><b>Enter maintenance</b>: pause Inspector for that container while you work on it
|
||||
(updates, migrations). Click again to exit.</li>
|
||||
<li><b>Mute notifications</b>: keep watching and restarting, but stop alerts for a while
|
||||
(12 hours by default).</li>
|
||||
</ul>
|
||||
<p>The page refreshes itself every 30 seconds.</p>
|
||||
</section>
|
||||
|
||||
<section id="notifications">
|
||||
<h2>4. Notifications</h2>
|
||||
<p>Inspector can alert you through <a href="https://gotify.net">Gotify</a>,
|
||||
<a href="https://pushover.net">Pushover</a>, or both. Set them up under
|
||||
<b>Settings → Notifications</b>.</p>
|
||||
<figure><img src="settings-notifications.png" alt="Notifications settings">
|
||||
<figcaption>Each channel has its own priority reference, a Send test button, and a Save button.</figcaption></figure>
|
||||
<ol class="steps">
|
||||
<li>Turn on <b>enabled</b> for the channel you use.</li>
|
||||
<li><b>Gotify</b>: enter your server URL and an <i>application</i> token (create one in Gotify
|
||||
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
|
||||
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
|
||||
the reason appears next to the button.</li>
|
||||
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 10</a>),
|
||||
because turning a channel on or off is a "requires restart" field.</li>
|
||||
</ol>
|
||||
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
|
||||
recovers. You won't get one on every check.</p>
|
||||
</section>
|
||||
|
||||
<section id="containers">
|
||||
<h2>5. Managing containers</h2>
|
||||
<p><b>Settings → All containers</b> shows every watched container as a tile. Click a tile (or a
|
||||
name in the left menu) to edit it.</p>
|
||||
<figure><img src="containers.png" alt="All containers grid">
|
||||
<figcaption>All containers.</figcaption></figure>
|
||||
<figure><img src="container-edit.png" alt="Container settings">
|
||||
<figcaption>Editing one container. Each restart trigger is a card you click on or off.</figcaption></figure>
|
||||
<p>Changes apply on the next check. <b>Delete container</b> stops Inspector watching it; it
|
||||
does <i>not</i> touch the container itself.</p>
|
||||
<h3>Restart triggers</h3>
|
||||
<table>
|
||||
<tr><th>Status</th><th>When it happens</th></tr>
|
||||
<tr><td>UNHEALTHY</td><td>The container's own health check is failing.</td></tr>
|
||||
<tr><td>NOT RUNNING</td><td>Stopped, exited, or crashed.</td></tr>
|
||||
<tr><td>UNREACHABLE</td><td>Running, but its HTTP check or Uptime Kuma monitor says it's down (see below).</td></tr>
|
||||
<tr><td>ERROR</td><td>Inspector couldn't check it (Docker API error).</td></tr>
|
||||
<tr><td>STARTING</td><td>Still in its health-check start period. Usually leave this off.</td></tr>
|
||||
<tr><td>HEALTHY / NO HEALTH CHECK</td><td>Normal states; almost never a reason to restart.</td></tr>
|
||||
</table>
|
||||
|
||||
<h3>Reachability: catching apps that look fine but aren't</h3>
|
||||
<p>Docker only knows whether a container is running and whether its own health check passes. An
|
||||
app can pass both and still be unusable, for example when it routes through a VPN container
|
||||
whose tunnel dropped. Under <b>Reachability</b> on the container page you can add:</p>
|
||||
<ul>
|
||||
<li><b>HTTP check</b>: a URL Inspector requests every check (for example
|
||||
<code>http://192.168.1.10:9696/ping</code>). Set <b>expected_status</b> to the code the app
|
||||
returns when it's healthy; most return 200, some return 401 or 302.</li>
|
||||
<li><b>Uptime Kuma monitor</b>: if you already run Uptime Kuma, name one of its monitors. Set up
|
||||
the connection first in <b>Settings → Uptime Kuma</b>; <b>Test connection</b> lists the
|
||||
monitor names it can see.</li>
|
||||
</ul>
|
||||
<p>A failed check reports <b>UNREACHABLE</b>. Tick UNREACHABLE in the restart triggers if you want
|
||||
Inspector to act on it.</p>
|
||||
<figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups">
|
||||
<figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure>
|
||||
|
||||
</section>
|
||||
|
||||
<section id="vpn">
|
||||
<h2>6. Apps behind a VPN container</h2>
|
||||
<h3>Why these apps need special handling</h3>
|
||||
<p>A common setup routes download and indexer apps through a VPN container. Those apps don't get a
|
||||
network of their own: their compose file says <code>network_mode: container:<vpn></code> (or
|
||||
<code>service:<vpn></code>) and all their traffic goes through the VPN container. That causes
|
||||
two problems for a watchdog:</p>
|
||||
<ul>
|
||||
<li><b>When the VPN breaks, every app behind it breaks at the same time.</b> Restarting the apps
|
||||
does nothing while the tunnel is down. Without help, Inspector would restart each app over and
|
||||
over, use up their hourly restart limits, and flood you with alerts, all for a problem in a
|
||||
different container.</li>
|
||||
<li><b>When the VPN container is recreated</b> (an image update, or someone redeploys its stack),
|
||||
it gets a new container ID. The apps still point at the old one, so a normal restart fails with
|
||||
errors like <i>"No such container"</i> or <i>"joining network namespace"</i>. The only fix is
|
||||
to recreate the apps too, which is what a Portainer stack redeploy does
|
||||
(<a href="#portainer">section 7</a>).</li>
|
||||
</ul>
|
||||
<h3>How to tell if an app is behind a VPN</h3>
|
||||
<ul>
|
||||
<li>In Portainer open <b>Stacks → your stack → Editor</b> and look for
|
||||
<code>network_mode: container:…</code> or <code>network_mode: service:…</code> on the app.</li>
|
||||
<li>Or open <b>Containers → the app</b>: under <b>Network</b> it shows the VPN container instead
|
||||
of a normal network, and it has no ports of its own (they're published on the VPN container).</li>
|
||||
</ul>
|
||||
<h3>Setting it up</h3>
|
||||
<ol class="steps">
|
||||
<li><b>Add the VPN container to Inspector and keep it enabled</b> (restart on UNHEALTHY and NOT
|
||||
RUNNING is a good start). Inspector can only wait for containers it is watching: if the VPN
|
||||
container is disabled, its apps are recovered normally, as if there were no dependency, and the
|
||||
container page shows a warning.</li>
|
||||
<li>On each app behind it, set <b>Dependencies → depends_on</b> to the VPN container.</li>
|
||||
<li>Turn on <b>vpn_routed</b>. This only changes the log: Inspector writes down when it holds off
|
||||
on the app because the VPN is down. The waiting itself comes from depends_on.</li>
|
||||
<li>Add an <b>HTTP check</b> (<a href="#containers">section 5</a>) using your server's IP and the
|
||||
port published on the VPN container, and tick <b>UNREACHABLE</b>. Docker often still reports
|
||||
these apps as healthy when the tunnel is down; the HTTP check notices.</li>
|
||||
<li>Under <b>Recovery</b>, choose <b>Restart, then redeploy stack if that fails</b> and enter the
|
||||
stack name. This needs the Portainer connection from <a href="#portainer">section 7</a>.</li>
|
||||
</ol>
|
||||
<h3>What Inspector then does</h3>
|
||||
<p>Every check, it looks at the VPN container first. While the VPN is anything other than healthy
|
||||
(or starting, or running without a health check), the apps that depend on it are not restarted;
|
||||
Inspector fixes the VPN first. Once the VPN is healthy again, the apps are handled normally: a
|
||||
restart first, and a stack redeploy if the restart fails or doesn't help.</p>
|
||||
<p class="note"><b>depends_on isn't only for VPNs.</b> Use it for any "B needs A" pair, such as an
|
||||
app and its database: Inspector won't restart the app while its database is down.</p>
|
||||
</section>
|
||||
|
||||
<section id="portainer">
|
||||
<h2>7. Portainer stack redeploy</h2>
|
||||
<h3>Why Inspector needs Portainer</h3>
|
||||
<p>Inspector talks to Docker directly, so it can restart or start a container. It can't
|
||||
<i>recreate</i> one: that needs the container's compose definition, which lives in Portainer. A
|
||||
stack redeploy asks Portainer to do exactly what its <b>Update the stack</b> button does. That's
|
||||
how VPN-routed apps get reattached to a recreated VPN container. You only need this if you use one
|
||||
of the two "redeploy" recovery actions.</p>
|
||||
<p class="note warn"><b>A redeploy affects the whole stack.</b> Portainer pulls the latest images
|
||||
and recreates <i>every</i> container in that stack, not just the failing app, so all of them blink
|
||||
off for a moment. That's why <b>Restart, then redeploy stack if that fails</b> is the recommended
|
||||
choice: most problems are fixed by the plain restart, and the redeploy is only used when that
|
||||
doesn't work.</p>
|
||||
<h3>Requirements</h3>
|
||||
<ul>
|
||||
<li>The app must belong to a stack that was <b>created in Portainer</b>. In <b>Stacks</b>, stacks
|
||||
created elsewhere (for example with <code>docker compose</code> on the command line) are marked
|
||||
<b>Limited</b>, and Portainer can't redeploy them.</li>
|
||||
<li>An access token for a Portainer user who can manage that stack. An administrator's token is
|
||||
the simplest.</li>
|
||||
</ul>
|
||||
|
||||
<h3>Step 1: find the values</h3>
|
||||
<table>
|
||||
<tr><th>Setting</th><th>Where to find it</th></tr>
|
||||
<tr><td>base_url</td><td>The address you open Portainer at, without anything after the port:
|
||||
usually <code>http://<server-ip>:9000</code> or <code>https://<server-ip>:9443</code>.
|
||||
Use the server's IP, not <code>localhost</code>: Inspector runs in its own container.</td></tr>
|
||||
<tr><td>endpoint_id</td><td>On Portainer's <b>Home</b> page, click your environment (often
|
||||
<b>local</b>). The browser address then ends like <code>#!/2/docker/dashboard</code>: the
|
||||
number after <code>#!/</code> is the endpoint ID (here <code>2</code>).</td></tr>
|
||||
<tr><td>stack name</td><td><b>Stacks</b> list, <b>Name</b> column, spelled exactly the same.
|
||||
This goes on each app's container page, not on the Portainer page.</td></tr>
|
||||
<tr><td>verify_ssl</td><td>Leave off for <code>http://</code> and for Portainer's built-in
|
||||
self-signed certificate on 9443. Turn on only if Portainer has a real certificate.</td></tr>
|
||||
<tr><td>API token</td><td>Click your username (top right) → <b>My account</b> →
|
||||
<b>Access tokens</b> → <b>Add access token</b>. Name it <code>inspector</code> and copy the
|
||||
token straight away: Portainer shows it only once.</td></tr>
|
||||
</table>
|
||||
|
||||
<h3>Step 2: give Inspector the token</h3>
|
||||
<p><b>Recommended: an environment variable.</b> The token then never lands in
|
||||
<code>config.yml</code> or its backups.</p>
|
||||
<ol class="steps">
|
||||
<li>In Portainer open <b>Stacks → inspector → Editor</b>.</li>
|
||||
<li>Check that the compose file has this line under <code>environment:</code> (it's in the
|
||||
compose file from version 0.9.1 on; add it if your stack is older):
|
||||
<code>- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}</code></li>
|
||||
<li>Under <b>Environment variables</b> click <b>Add an environment variable</b>: name
|
||||
<code>PORTAINER_API_TOKEN</code>, value the token.</li>
|
||||
<li>Click <b>Update the stack</b>. Inspector restarts with the token available.</li>
|
||||
</ol>
|
||||
<p><b>Alternative: save it in Inspector.</b> In the next step choose <b>Saved in config.yml</b>
|
||||
and paste the token. It is masked in the dashboard, but anyone who can read Inspector's config
|
||||
folder (or a backup of it) can read the token.</p>
|
||||
|
||||
<h3>Step 3: connect Inspector</h3>
|
||||
<ol class="steps">
|
||||
<li>Open <b>Settings → Portainer</b>. Under <b>Add environment</b>, give it a name (your own
|
||||
label, e.g. <code>home</code>) and fill in base_url, endpoint_id and verify_ssl.</li>
|
||||
<li><b>API token stored as</b>: <i>Environment variable</i> with the name
|
||||
<code>PORTAINER_API_TOKEN</code>, or <i>Saved in config.yml</i> with the token.</li>
|
||||
<li>Click <b>Test connection</b>. It should list the stacks on that endpoint; check that your
|
||||
app's stack is there. Then <b>Save new environment</b>.</li>
|
||||
<li>At the top of the page turn <b>enabled</b> on, pick the <b>default_environment</b>, and
|
||||
<b>Save Portainer</b>.</li>
|
||||
<li>On each app's container page, set the recovery action and stack name
|
||||
(<a href="#vpn">section 6</a>). Leave <b>portainer_environment</b> on <i>Default</i> unless
|
||||
you connected more than one Portainer.</li>
|
||||
</ol>
|
||||
<figure><img src="settings-portainer.png" alt="Portainer settings">
|
||||
<figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure>
|
||||
|
||||
<h3>The other settings on the page</h3>
|
||||
<table>
|
||||
<tr><th>Setting</th><th>What it does</th></tr>
|
||||
<tr><td>default_environment</td><td>Which connection apps use when they don't pick one. With a
|
||||
single environment you can leave it on Automatic.</td></tr>
|
||||
<tr><td>stack_redeploy_cooldown_seconds</td><td>Minimum time between two redeploys of the same
|
||||
stack (default 300). Several apps often share a stack; this stops them triggering a redeploy
|
||||
each.</td></tr>
|
||||
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector gives the stack to come up after
|
||||
a redeploy before judging that app again (default 90). Everything else keeps being checked
|
||||
meanwhile.</td></tr>
|
||||
</table>
|
||||
|
||||
<h3>If Test connection fails</h3>
|
||||
<table>
|
||||
<tr><th>Message</th><th>Fix</th></tr>
|
||||
<tr><td>environment variable … is not set</td><td>The compose line from step 2 is missing, or
|
||||
the stack wasn't updated after adding the variable.</td></tr>
|
||||
<tr><td>the API token was rejected</td><td>Token copied wrong, deleted, or from a user without
|
||||
access. Create a new one.</td></tr>
|
||||
<tr><td>no stacks found on endpoint N</td><td>Wrong endpoint ID; check the number in the
|
||||
Portainer address again.</td></tr>
|
||||
<tr><td>request failed …</td><td>Wrong address or port, <code>http</code> vs <code>https</code>
|
||||
mixed up, or <code>localhost</code> used instead of the server IP. With https and a
|
||||
self-signed certificate, turn verify_ssl off.</td></tr>
|
||||
</table>
|
||||
</section>
|
||||
|
||||
<section id="events">
|
||||
<h2>8. Events history</h2>
|
||||
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
|
||||
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
||||
<figure><img src="events.png" alt="Events timeline">
|
||||
<figcaption>Here a container stopped, Inspector restarted it, and it recovered a few seconds later.</figcaption></figure>
|
||||
</section>
|
||||
|
||||
<section id="settings">
|
||||
<h2>9. Other settings</h2>
|
||||
<figure><img src="settings-inspector.png" alt="Inspector loop settings">
|
||||
<figcaption>Settings → Inspector loop.</figcaption></figure>
|
||||
<table>
|
||||
<tr><th>Section</th><th>What it's for</th></tr>
|
||||
<tr><td>Inspector loop</td><td><b>interval_seconds</b>: how often to check (60 is fine).
|
||||
<b>dry_run</b>: log and alert but never actually restart; handy while you're trying it out.</td></tr>
|
||||
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
|
||||
<tr><td>Event history</td><td>How many days of events to keep.</td></tr>
|
||||
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
|
||||
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00, in your <a href="#settings">time zone</a>) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
|
||||
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
|
||||
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
|
||||
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#portainer">section 7</a>.</td></tr>
|
||||
</table>
|
||||
<p>At the bottom of every settings page is a strip showing where the config file lives,
|
||||
whether Inspector can write to it, and which time zone it is using.</p>
|
||||
|
||||
<h3>Time zone</h3>
|
||||
<p>Inspector uses the <code>TZ</code> setting of its Portainer stack for every time it shows or acts on:</p>
|
||||
<ul>
|
||||
<li><b>The maintenance window runs in this zone.</b> A 02:00–05:00 window with the wrong zone
|
||||
runs hours off, so Inspector could restart containers during your backups and ignore
|
||||
problems at other times.</li>
|
||||
<li>Alert times (with the zone, e.g. <code>03:15 AM CET</code>), dashboard times, the Events
|
||||
page's days (midnight to midnight), and logs.</li>
|
||||
</ul>
|
||||
<p>To change it: in Portainer open <b>Stacks → inspector → Editor</b>, set the <code>TZ</code>
|
||||
environment variable to your zone's name, e.g. <code>America/New_York</code>,
|
||||
<code>Europe/London</code> or <code>Australia/Sydney</code> (names from the
|
||||
<a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">list of time zones</a>;
|
||||
abbreviations like <code>EST</code> don't work), then click <b>Update the stack</b>. Without
|
||||
<code>TZ</code>, Inspector uses UTC.</p>
|
||||
</section>
|
||||
|
||||
<section id="restart">
|
||||
<h2>10. "Requires restart" fields</h2>
|
||||
<p>Most changes apply on the next check. Fields marked
|
||||
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
|
||||
dashboard on or off) only take effect after Inspector restarts:</p>
|
||||
<ol class="steps">
|
||||
<li>In Portainer go to <b>Containers</b>.</li>
|
||||
<li>Tick <b>inspector</b> and click <b>Restart</b>.</li>
|
||||
</ol>
|
||||
<p>Your settings and history are kept; they live in the folders mounted into the container.</p>
|
||||
</section>
|
||||
|
||||
<section id="trouble">
|
||||
<h2>11. Troubleshooting</h2>
|
||||
<table>
|
||||
<tr><th>Problem</th><th>Fix</th></tr>
|
||||
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
|
||||
<td>Inspector can't save changes. In the stack, make sure the config folder mount does not end in
|
||||
<code>:ro</code>, then update the stack.</td></tr>
|
||||
<tr><td>A card says <b>CONTAINER NOT FOUND</b></td>
|
||||
<td>The name doesn't match a container. Check the exact name in Portainer → Containers, delete
|
||||
the entry and add it again with the right name.</td></tr>
|
||||
<tr><td>Logs say <i>Permission denied on /var/run/docker.sock</i></td>
|
||||
<td>Remove any <code>user:</code> line from the stack so Inspector starts normally (it detects the
|
||||
socket permissions itself), then update the stack.</td></tr>
|
||||
<tr><td>Send test fails</td>
|
||||
<td>The message next to the button says why: wrong URL, wrong token (HTTP 401), or the server can't
|
||||
be reached from your Docker host.</td></tr>
|
||||
<tr><td>Forgot the dashboard password</td>
|
||||
<td>Open <code>config.yml</code> in the stack's config folder (e.g. through your Unraid share),
|
||||
set <code>dashboard: password:</code> to something new, and restart Inspector.</td></tr>
|
||||
<tr><td>No alerts arrive</td>
|
||||
<td>Check the channel is enabled, Send test works, Inspector was restarted after enabling it,
|
||||
and the container isn't muted or in maintenance.</td></tr>
|
||||
</table>
|
||||
</section>
|
||||
</main>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
After Width: | Height: | Size: 34 KiB |
|
After Width: | Height: | Size: 63 KiB |
|
After Width: | Height: | Size: 42 KiB |
|
After Width: | Height: | Size: 33 KiB |
@@ -0,0 +1,56 @@
|
||||
"""First-run helpers: find config.yml and write a starter one if it's missing.
|
||||
|
||||
Lets Inspector be installed from Portainer with nothing on the host but an
|
||||
empty folder mounted at /app/config.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
from importlib import resources
|
||||
from pathlib import Path
|
||||
|
||||
LEGACY_CONFIG_PATH = Path("/app/config.yml")
|
||||
DEFAULT_CONFIG_PATH = Path("/app/config/config.yml")
|
||||
|
||||
|
||||
def resolve_config_path(cli_value: str | None, env: dict | None = None) -> Path:
|
||||
"""--config wins, then $INSPECTOR_CONFIG, then the legacy single-file
|
||||
mount if present, then the folder mount."""
|
||||
env = os.environ if env is None else env
|
||||
if cli_value:
|
||||
return Path(cli_value)
|
||||
if env.get("INSPECTOR_CONFIG"):
|
||||
return Path(env["INSPECTOR_CONFIG"])
|
||||
if LEGACY_CONFIG_PATH.exists():
|
||||
return LEGACY_CONFIG_PATH
|
||||
return DEFAULT_CONFIG_PATH
|
||||
|
||||
|
||||
def write_starter_config(path: Path, env: dict | None = None) -> str | None:
|
||||
"""Write the starter config to `path`.
|
||||
|
||||
Dashboard credentials come from INSPECTOR_USERNAME / INSPECTOR_PASSWORD.
|
||||
Returns the generated password when none was supplied (so the caller can
|
||||
log it once), otherwise None.
|
||||
"""
|
||||
env = os.environ if env is None else env
|
||||
username = env.get("INSPECTOR_USERNAME") or "admin"
|
||||
password = env.get("INSPECTOR_PASSWORD") or ""
|
||||
generated = None
|
||||
if not password:
|
||||
generated = password = secrets.token_urlsafe(12)
|
||||
|
||||
template = (resources.files(__package__) / "starter_config.yml").read_text(encoding="utf-8")
|
||||
# json.dumps gives a valid YAML double-quoted scalar for any input.
|
||||
text = template.replace("__USERNAME__", json.dumps(username)).replace(
|
||||
"__PASSWORD__", json.dumps(password)
|
||||
)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
try:
|
||||
path.chmod(0o600)
|
||||
except OSError:
|
||||
pass
|
||||
return generated
|
||||
@@ -1,10 +1,12 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import threading
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import time
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
@@ -23,6 +25,30 @@ VALID_STATUSES = {
|
||||
"UNREACHABLE",
|
||||
}
|
||||
|
||||
log = logging.getLogger("inspector.config")
|
||||
|
||||
# Names logging.setLevel() accepts; anything else would crash at startup.
|
||||
VALID_LOG_LEVELS = {"CRITICAL", "FATAL", "ERROR", "WARN", "WARNING", "INFO", "DEBUG", "NOTSET"}
|
||||
|
||||
|
||||
def valid_hhmm(value: str) -> bool:
|
||||
"""True if `value` parses the way the maintenance tracker parses it ("HH:MM")."""
|
||||
try:
|
||||
hour, minute = str(value).split(":")
|
||||
time(int(hour), int(minute))
|
||||
return True
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _startup_safe(value, default, ok: bool, key: str):
|
||||
"""Return `value`, or `default` with a warning when it would crash Inspector."""
|
||||
if ok:
|
||||
return value
|
||||
log.warning("config: invalid %s %r; using %r instead", key, value, default)
|
||||
return default
|
||||
|
||||
|
||||
VALID_RECOVERY_ACTIONS = {
|
||||
"restart_container",
|
||||
"start_container",
|
||||
@@ -190,9 +216,11 @@ def build_config_from_raw(raw: dict) -> AppConfig:
|
||||
to validate an edited raw dict before writing it back to disk.
|
||||
"""
|
||||
insp_raw = raw.get("inspector", {}) or {}
|
||||
interval = int(insp_raw.get("interval_seconds", 60))
|
||||
log_level = str(insp_raw.get("log_level", "INFO")).upper()
|
||||
inspector = InspectorConfig(
|
||||
interval_seconds=int(insp_raw.get("interval_seconds", 60)),
|
||||
log_level=str(insp_raw.get("log_level", "INFO")).upper(),
|
||||
interval_seconds=_startup_safe(interval, 1, interval >= 1, "inspector.interval_seconds"),
|
||||
log_level=_startup_safe(log_level, "INFO", log_level in VALID_LOG_LEVELS, "inspector.log_level"),
|
||||
dry_run=bool(insp_raw.get("dry_run", False)),
|
||||
log_dir=str(insp_raw.get("log_dir", "/app/logs")),
|
||||
log_filename=str(insp_raw.get("log_filename", "inspector.log")),
|
||||
@@ -237,10 +265,12 @@ def build_config_from_raw(raw: dict) -> AppConfig:
|
||||
)
|
||||
|
||||
maint_raw = raw.get("maintenance") or {}
|
||||
maint_start = str(maint_raw.get("start", "00:00"))
|
||||
maint_end = str(maint_raw.get("end", "06:00"))
|
||||
maintenance = MaintenanceConfig(
|
||||
enabled=bool(maint_raw.get("enabled", False)),
|
||||
start=str(maint_raw.get("start", "00:00")),
|
||||
end=str(maint_raw.get("end", "06:00")),
|
||||
start=_startup_safe(maint_start, "00:00", valid_hhmm(maint_start), "maintenance.start"),
|
||||
end=_startup_safe(maint_end, "06:00", valid_hhmm(maint_end), "maintenance.end"),
|
||||
dir=str(maint_raw.get("dir", "/app/data/maintenance")),
|
||||
max_hours=int(maint_raw.get("max_hours", 8)),
|
||||
)
|
||||
|
||||
@@ -1,14 +1,49 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from zoneinfo import ZoneInfo
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timezone, tzinfo
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
|
||||
PACIFIC = ZoneInfo("America/Los_Angeles")
|
||||
log = logging.getLogger("inspector.formatting")
|
||||
|
||||
|
||||
def format_timestamp(value: datetime | str | None) -> str:
|
||||
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'.
|
||||
def _resolve_local_tz(env: dict | None = None) -> tuple[tzinfo, str]:
|
||||
"""The zone every displayed time uses, plus its name.
|
||||
|
||||
Follows the same rules as the C library (which drives log timestamps and
|
||||
the maintenance window), so all times Inspector shows agree:
|
||||
TZ set to a zone name -> that zone; TZ unset -> the container's
|
||||
/etc/localtime; anything else -> UTC.
|
||||
"""
|
||||
env = os.environ if env is None else env
|
||||
name = (env.get("TZ") or "").strip().lstrip(":")
|
||||
if name:
|
||||
try:
|
||||
return ZoneInfo(name), name
|
||||
except (ZoneInfoNotFoundError, ValueError):
|
||||
log.warning(
|
||||
"TZ=%r is not a known time zone name (e.g. Europe/Berlin); showing times in UTC",
|
||||
name,
|
||||
)
|
||||
return timezone.utc, "UTC"
|
||||
try:
|
||||
with open("/etc/localtime", "rb") as fh:
|
||||
zone = ZoneInfo.from_file(fh, key="localtime")
|
||||
label = datetime.now(zone).tzname() or "localtime"
|
||||
return zone, label
|
||||
except (OSError, ValueError):
|
||||
return timezone.utc, "UTC"
|
||||
|
||||
|
||||
# Resolved once at import; the container's TZ doesn't change while it runs.
|
||||
LOCAL_TZ, LOCAL_TZ_NAME = _resolve_local_tz()
|
||||
|
||||
|
||||
def format_timestamp(value: datetime | str | None, *, with_zone: bool = False) -> str:
|
||||
"""Render a timestamp in the configured local zone, e.g. '06-14 03:45 PM'
|
||||
(with_zone adds the abbreviation: '06-14 03:45 PM CET').
|
||||
|
||||
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
|
||||
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
|
||||
@@ -24,4 +59,8 @@ def format_timestamp(value: datetime | str | None) -> str:
|
||||
return value
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p")
|
||||
local = value.astimezone(LOCAL_TZ)
|
||||
text = local.strftime("%m-%d %I:%M %p")
|
||||
if with_zone:
|
||||
text += f" {local.tzname()}"
|
||||
return text
|
||||
@@ -2,6 +2,7 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
@@ -9,6 +10,7 @@ from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from . import __version__
|
||||
from .bootstrap import resolve_config_path, write_starter_config
|
||||
from .config import AppConfig, ConfigRef, ContainerConfig, load_config, load_raw
|
||||
from .dashboard import DashboardServer
|
||||
from .docker_client import build_client, get_container_status
|
||||
@@ -39,7 +41,7 @@ from .maintenance import MaintenanceTracker
|
||||
from .notification_mute import NotificationMuteTracker
|
||||
from .notifier import GotifyNotifier, MultiNotifier, NullNotifier, PushoverNotifier
|
||||
from .portainer_client import PortainerClient
|
||||
from .recovery import perform_recovery
|
||||
from .recovery import perform_recovery, run_post_redeploy_recheck
|
||||
from .uptime_kuma import HeartbeatSnapshot, UptimeKumaClient
|
||||
from .state import (
|
||||
ContainerState,
|
||||
@@ -61,6 +63,19 @@ def _request_stop(signum, _frame):
|
||||
_stop = True
|
||||
|
||||
|
||||
def _is_normal(status: str | None, container_config: ContainerConfig) -> bool:
|
||||
"""True when `status` means the container is working.
|
||||
|
||||
HEALTHY always counts. NO HEALTH CHECK (running, but the image defines no
|
||||
health check) counts too, unless the user made it a restart trigger.
|
||||
Many images have no health check, and without this their error count and
|
||||
restart episode would never reset.
|
||||
"""
|
||||
if status == HEALTHY:
|
||||
return True
|
||||
return status == NO_HEALTH_CHECK and NO_HEALTH_CHECK not in container_config.restart_on
|
||||
|
||||
|
||||
def _apply_reachability_override(
|
||||
container_config: ContainerConfig,
|
||||
result: HealthResult,
|
||||
@@ -138,6 +153,11 @@ def _check_one(
|
||||
if notification_mute.is_muted(name):
|
||||
notifier = NullNotifier()
|
||||
|
||||
if state.redeploy_recheck_at is not None and now >= state.redeploy_recheck_at:
|
||||
run_post_redeploy_recheck(
|
||||
container_config=container_config, state=state, notifier=notifier, now=now,
|
||||
)
|
||||
|
||||
if state.previous_status != state.current_status:
|
||||
log.info(
|
||||
"%-*s status: %s -> %s%s",
|
||||
@@ -156,19 +176,25 @@ def _check_one(
|
||||
message=result.message,
|
||||
)
|
||||
|
||||
is_normal = _is_normal(state.current_status, container_config)
|
||||
if (
|
||||
state.previous_status is not None
|
||||
and state.previous_status != HEALTHY
|
||||
and state.current_status == HEALTHY
|
||||
and not _is_normal(state.previous_status, container_config)
|
||||
and is_normal
|
||||
# HEALTHY here means "a recovery was already announced", for either status.
|
||||
and state.last_notification_state != HEALTHY
|
||||
):
|
||||
log.info("%-*s recovered: %s -> HEALTHY", width, name, state.previous_status)
|
||||
log.info("%-*s recovered: %s -> %s", width, name, state.previous_status, state.current_status)
|
||||
back_to = (
|
||||
"**HEALTHY**" if state.current_status == HEALTHY
|
||||
else "**running** (no health check)"
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} recovered",
|
||||
message=(
|
||||
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
|
||||
f"**{name}** is back to {back_to} (was *{state.previous_status}*).\n\n"
|
||||
f"- Errors before recovery: **{state.error_count}**\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
@@ -176,7 +202,7 @@ def _check_one(
|
||||
container=name,
|
||||
event_type=EVENT_RECOVERY,
|
||||
old_status=state.previous_status,
|
||||
new_status=HEALTHY,
|
||||
new_status=state.current_status,
|
||||
error_count=state.error_count,
|
||||
)
|
||||
state.last_notification_state = HEALTHY
|
||||
@@ -194,7 +220,7 @@ def _check_one(
|
||||
f"**{name}** is missing from Docker.\n\n"
|
||||
+ (f"- Detail: {result.message}\n" if result.message else "")
|
||||
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=7,
|
||||
)
|
||||
@@ -238,7 +264,7 @@ def _check_one(
|
||||
+ ".\n\n"
|
||||
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
|
||||
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=5,
|
||||
)
|
||||
@@ -251,9 +277,9 @@ def _check_one(
|
||||
message=result.message,
|
||||
)
|
||||
state.last_notification_state = result.status
|
||||
elif result.status == HEALTHY:
|
||||
elif is_normal:
|
||||
if state.error_count != 0:
|
||||
log.info("%-*s healthy again; resetting error count", width, name)
|
||||
log.info("%-*s back to normal; resetting error count", width, name)
|
||||
state.error_count = 0
|
||||
state.restarted_this_episode = False
|
||||
|
||||
@@ -274,7 +300,7 @@ def _check_one(
|
||||
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
|
||||
f"- Current status: **{result.status}**\n"
|
||||
f"- Restarts paused until the hourly window clears\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=9,
|
||||
)
|
||||
@@ -291,8 +317,10 @@ def _check_one(
|
||||
|
||||
if ok:
|
||||
dep = container_config.depends_on
|
||||
if dep:
|
||||
dep_status = states[dep].current_status if dep in states else None
|
||||
# Only wait on a dependency Inspector is watching. A disabled one has
|
||||
# no state, and waiting on it would block recovery forever.
|
||||
if dep and dep in states:
|
||||
dep_status = states[dep].current_status
|
||||
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
|
||||
if container_config.vpn_routed:
|
||||
log.info(
|
||||
@@ -398,8 +426,9 @@ def run_loop(config_ref: ConfigRef) -> int:
|
||||
except Exception as exc:
|
||||
if "Permission denied" in str(exc):
|
||||
log.critical(
|
||||
"Permission denied on /var/run/docker.sock. Set DOCKER_GID to "
|
||||
"the host docker group GID: stat -c '%g' /var/run/docker.sock"
|
||||
"Permission denied on /var/run/docker.sock. Start the container "
|
||||
"as root (the default) so it can detect the socket's group, or set "
|
||||
"group_add to the socket's GID: stat -c '%g' /var/run/docker.sock"
|
||||
)
|
||||
else:
|
||||
log.critical("Cannot reach Docker daemon: %s", exc)
|
||||
@@ -414,6 +443,7 @@ def run_loop(config_ref: ConfigRef) -> int:
|
||||
|
||||
check_order = _topological_order(enabled_configs)
|
||||
portainer_client = PortainerClient(app.integrations.portainer)
|
||||
integrations_cfg = app.integrations
|
||||
stack_cooldowns: dict[str, datetime] = {}
|
||||
|
||||
dashboard = DashboardServer(
|
||||
@@ -437,18 +467,31 @@ def run_loop(config_ref: ConfigRef) -> int:
|
||||
notifier = _build_notifier(app)
|
||||
|
||||
# Rebuild enabled_configs / check_order from current config so edits
|
||||
# to per-container settings apply next cycle. Containers added via
|
||||
# the settings page are logged as 'requires restart' because the
|
||||
# states dict is built once at startup.
|
||||
# to per-container settings apply next cycle. Containers added (or
|
||||
# re-enabled) from the settings page get a fresh state here; `states`
|
||||
# is shared with the dashboard, so it is updated in place.
|
||||
current_enabled = {c.name: c for c in app.containers if c.enabled}
|
||||
new_names = current_enabled.keys() - states.keys()
|
||||
if new_names:
|
||||
log.warning(
|
||||
"Containers added in config require Inspector restart to begin monitoring: %s",
|
||||
sorted(new_names),
|
||||
)
|
||||
enabled_configs = {n: c for n, c in current_enabled.items() if n in states}
|
||||
check_order = _topological_order(enabled_configs)
|
||||
for new_name in new_names:
|
||||
states[new_name] = ContainerState(name=new_name)
|
||||
name_width = max(name_width, *(len(n) for n in new_names))
|
||||
log.info("Now watching newly added container(s): %s", sorted(new_names))
|
||||
for gone in states.keys() - current_enabled.keys():
|
||||
states.pop(gone, None)
|
||||
log.info("Stopped watching container (removed or disabled): %s", gone)
|
||||
if current_enabled != enabled_configs:
|
||||
# Only re-order (and re-log ordering warnings) when the config changed.
|
||||
check_order = _topological_order(current_enabled)
|
||||
enabled_configs = current_enabled
|
||||
|
||||
# Integration clients hold their config; rebuild them when the
|
||||
# settings page changes Uptime Kuma / Portainer.
|
||||
if app.integrations != integrations_cfg:
|
||||
integrations_cfg = app.integrations
|
||||
uptime_kuma_client = UptimeKumaClient(integrations_cfg.uptime_kuma)
|
||||
portainer_client = PortainerClient(integrations_cfg.portainer)
|
||||
log.info("Integration settings changed; reloaded Uptime Kuma / Portainer clients")
|
||||
|
||||
uptime_kuma_snapshot = uptime_kuma_client.fetch()
|
||||
for name in check_order:
|
||||
@@ -495,14 +538,37 @@ def run_loop(config_ref: ConfigRef) -> int:
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(prog="inspector", description="Docker container health watcher")
|
||||
parser.add_argument("--config", "-c", default="/app/config.yml", help="Path to config.yml")
|
||||
parser.add_argument(
|
||||
"--config", "-c", default=None,
|
||||
help="Path to config.yml (default: $INSPECTOR_CONFIG, /app/config.yml if present, "
|
||||
"else /app/config/config.yml)",
|
||||
)
|
||||
parser.add_argument("--version", action="version", version=f"inspector {__version__}")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
config_path = Path(args.config)
|
||||
config_path = resolve_config_path(args.config)
|
||||
if not config_path.exists():
|
||||
print(f"config file not found: {config_path}", file=sys.stderr)
|
||||
# First run with an empty config folder: write a starter config so the
|
||||
# dashboard comes up and everything else can be set from the web UI.
|
||||
try:
|
||||
generated_password = write_starter_config(config_path)
|
||||
except OSError as exc:
|
||||
print(
|
||||
f"config file not found: {config_path} (and could not create a starter one: {exc})",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
print(f"Created starter config at {config_path}", file=sys.stderr)
|
||||
if generated_password:
|
||||
print(
|
||||
"=" * 64 + "\n"
|
||||
" Inspector dashboard login (generated on first start)\n"
|
||||
f" username: {os.environ.get('INSPECTOR_USERNAME') or 'admin'}\n"
|
||||
f" password: {generated_password}\n"
|
||||
" Change it under Settings -> Dashboard.\n"
|
||||
+ "=" * 64,
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
app = load_config(config_path)
|
||||
raw = load_raw(config_path)
|
||||
|
||||
@@ -33,9 +33,10 @@ class GotifyNotifier:
|
||||
},
|
||||
}
|
||||
try:
|
||||
# Header, not ?token=, so the token stays out of access logs.
|
||||
resp = requests.post(
|
||||
url,
|
||||
params={"token": self._config.token},
|
||||
headers={"X-Gotify-Key": self._config.token},
|
||||
json=payload,
|
||||
timeout=self._timeout,
|
||||
)
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from .config import AppConfig, ContainerConfig, PortainerConfig
|
||||
@@ -175,7 +174,8 @@ def _perform_restart_then_redeploy(
|
||||
- the container is still failing on a later recovery cycle
|
||||
(`state.restarted_this_episode` is already set), meaning the restart didn't stick.
|
||||
|
||||
The episode flag resets when the container is next seen HEALTHY (see main._check_one).
|
||||
The episode flag resets when the container is next seen working: HEALTHY, or
|
||||
NO HEALTH CHECK when that isn't a restart trigger (see main._is_normal).
|
||||
"""
|
||||
name = container_config.name
|
||||
|
||||
@@ -228,7 +228,7 @@ def _perform_restart_then_redeploy(
|
||||
f"**{result.status}** check(s).\n\n"
|
||||
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
|
||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
@@ -242,7 +242,7 @@ def _perform_restart_then_redeploy(
|
||||
state.restart_limit_notified = False
|
||||
log.info(
|
||||
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
|
||||
width, name, state.cooldown_until.isoformat(),
|
||||
width, name, format_timestamp(state.cooldown_until, with_zone=True),
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
@@ -250,15 +250,15 @@ def _perform_restart_then_redeploy(
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
||||
message=f"restart succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} restart completed",
|
||||
message=(
|
||||
f"**{name}** was successfully restarted by Docker.\n\n"
|
||||
f"- Previous status: *{result.status}*\n"
|
||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
@@ -321,7 +321,7 @@ def _perform_docker_action(
|
||||
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
|
||||
f"**{result.status}** check(s).\n\n"
|
||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
@@ -335,22 +335,22 @@ def _perform_docker_action(
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
state.last_notification_state = "RESTARTED"
|
||||
state.restart_limit_notified = False
|
||||
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat())
|
||||
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, format_timestamp(state.cooldown_until, with_zone=True))
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
||||
message=f"{verb} succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} {verb} completed",
|
||||
message=(
|
||||
f"**{name}** was successfully {verb}ed by Docker.\n\n"
|
||||
f"- Previous status: *{result.status}*\n"
|
||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
@@ -362,7 +362,7 @@ def _perform_docker_action(
|
||||
f"{verb.capitalize()} of **{name}** failed.\n\n"
|
||||
f"- Status: **{result.status}**\n"
|
||||
f"- Error count: **{state.error_count}**\n"
|
||||
f"- Time: {format_timestamp(now)}\n\n"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
|
||||
f"```\n{err}\n```"
|
||||
),
|
||||
priority=9,
|
||||
@@ -469,7 +469,7 @@ def _perform_portainer_redeploy(
|
||||
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
|
||||
f"to recover **{name}**.\n\n"
|
||||
f"- Status: **{result.status}**\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
@@ -483,7 +483,7 @@ def _perform_portainer_redeploy(
|
||||
message=(
|
||||
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
|
||||
f"- Triggered by: **{name}** ({result.status})\n"
|
||||
f"- Time: {format_timestamp(now)}\n\n"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
|
||||
f"```\n{redeploy_result.message}\n```"
|
||||
),
|
||||
priority=9,
|
||||
@@ -516,29 +516,51 @@ def _perform_portainer_redeploy(
|
||||
f"successfully redeployed to recover **{name}**.\n\n"
|
||||
f"- Triggered by: *{result.status}*\n"
|
||||
f"- Post-redeploy wait: {wait_seconds}s\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=4,
|
||||
)
|
||||
|
||||
log.info("waiting %d seconds after stack redeploy", wait_seconds)
|
||||
time.sleep(wait_seconds)
|
||||
|
||||
# Don't block the loop while the stack settles: keep this container in
|
||||
# cooldown for the wait plus its normal cooldown, and do the one-time HTTP
|
||||
# re-check on its first regular check after the wait (main._check_one).
|
||||
log.info(
|
||||
"%-*s letting stack %s settle for %ds; other containers keep being checked",
|
||||
width, name, key, wait_seconds,
|
||||
)
|
||||
state.record_restart(now, wait_seconds + container_config.cooldown_seconds)
|
||||
if container_config.http_check.enabled:
|
||||
state.redeploy_recheck_at = now + timedelta(seconds=wait_seconds)
|
||||
|
||||
|
||||
def run_post_redeploy_recheck(
|
||||
*,
|
||||
container_config: ContainerConfig,
|
||||
state: ContainerState,
|
||||
notifier: GotifyNotifier,
|
||||
now: datetime,
|
||||
) -> None:
|
||||
"""One HTTP re-check after a stack redeploy has had time to settle.
|
||||
|
||||
Called by main._check_one once `state.redeploy_recheck_at` has passed;
|
||||
alerts if the app still doesn't answer.
|
||||
"""
|
||||
state.redeploy_recheck_at = None
|
||||
name = container_config.name
|
||||
if not container_config.http_check.enabled:
|
||||
return
|
||||
recheck = check_http(container_config.http_check)
|
||||
if recheck.ok:
|
||||
log.info("recheck succeeded for %s", name)
|
||||
else:
|
||||
return
|
||||
log.warning("recheck failed for %s: %s", name, recheck.message)
|
||||
notifier.notify(
|
||||
title=f"⚠️ Inspector: {name} recheck failed after redeploy",
|
||||
message=(
|
||||
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
|
||||
f"Portainer stack **{container_config.portainer_stack_name}** was redeployed, but the HTTP recheck "
|
||||
f"for **{name}** still reports a failure.\n\n"
|
||||
f"- Recheck result: *{recheck.message}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
@@ -0,0 +1,64 @@
|
||||
# Inspector configuration.
|
||||
#
|
||||
# Inspector wrote this starter file on first start. You can edit everything
|
||||
# from the web UI (Settings), or by hand. Containers are added from
|
||||
# Settings -> Containers -> Add container.
|
||||
|
||||
inspector:
|
||||
interval_seconds: 60
|
||||
log_level: INFO
|
||||
dry_run: false
|
||||
log_dir: /app/logs
|
||||
log_filename: inspector.log
|
||||
log_retention_days: 30
|
||||
heartbeat: false
|
||||
|
||||
notifications:
|
||||
# Gotify: https://gotify.net
|
||||
gotify:
|
||||
enabled: false
|
||||
url: ""
|
||||
token: ""
|
||||
priority: 5
|
||||
|
||||
# Pushover: https://pushover.net
|
||||
pushover:
|
||||
enabled: false
|
||||
token: ""
|
||||
user_key: ""
|
||||
priority: 5
|
||||
device: ""
|
||||
sound: ""
|
||||
|
||||
# Event history (powers the Events page and the health summary digest).
|
||||
database:
|
||||
enabled: true
|
||||
path: /app/data/events.db
|
||||
retention_days: 30
|
||||
|
||||
health_summary:
|
||||
enabled: false
|
||||
interval_hours: 24
|
||||
priority: 3
|
||||
|
||||
maintenance:
|
||||
enabled: false
|
||||
start: "00:00"
|
||||
end: "06:00"
|
||||
dir: /app/data/maintenance
|
||||
max_hours: 8
|
||||
|
||||
notification_mute:
|
||||
enabled: true
|
||||
dir: /app/data/muted
|
||||
default_hours: 12
|
||||
|
||||
dashboard:
|
||||
enabled: true
|
||||
host: 0.0.0.0
|
||||
port: 8080
|
||||
username: __USERNAME__
|
||||
password: __PASSWORD__
|
||||
|
||||
# Containers Inspector watches. Add them from the web UI.
|
||||
containers:
|
||||
@@ -29,6 +29,8 @@ class ContainerState:
|
||||
restart_limit_notified: bool = False
|
||||
restarted_this_episode: bool = False
|
||||
last_check_at: datetime | None = None
|
||||
# After a stack redeploy: when to do the one-time HTTP re-check.
|
||||
redeploy_recheck_at: datetime | None = None
|
||||
|
||||
def in_cooldown(self, now: datetime) -> bool:
|
||||
return self.cooldown_until is not None and now < self.cooldown_until
|
||||
|
||||
@@ -41,20 +41,26 @@ class UptimeKumaClient:
|
||||
and bool(self._config.status_page_slug)
|
||||
)
|
||||
|
||||
def _heartbeat_url(self) -> str:
|
||||
return f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
|
||||
|
||||
def fetch(self) -> HeartbeatSnapshot:
|
||||
if not self.enabled:
|
||||
return _EMPTY
|
||||
|
||||
url = f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
|
||||
try:
|
||||
resp = requests.get(url, timeout=self._timeout)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return self._parse(data)
|
||||
return self.fetch_or_raise()
|
||||
except Exception as exc:
|
||||
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", url, exc)
|
||||
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", self._heartbeat_url(), exc)
|
||||
return _EMPTY
|
||||
|
||||
def fetch_or_raise(self) -> HeartbeatSnapshot:
|
||||
"""Like fetch(), but lets request/parse errors propagate (used by the
|
||||
settings page's "Test connection" button)."""
|
||||
resp = requests.get(self._heartbeat_url(), timeout=self._timeout)
|
||||
resp.raise_for_status()
|
||||
return self._parse(resp.json())
|
||||
|
||||
def _parse(self, data: dict) -> HeartbeatSnapshot:
|
||||
# Expected shape:
|
||||
# {"publicGroupList": [{"monitorList": [{"id": 1, "name": "..."}]}],
|
||||
|
||||