Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6aed0cf9a7 | ||
|
|
57100d0a7b | ||
|
|
b4da527732 | ||
|
|
284ba2ea15 | ||
|
|
bd78835f4a | ||
|
|
7013a13cba |
No files matched your search
+148
@@ -7,6 +7,154 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
|
||||
|
||||
---
|
||||
|
||||
## [0.9.4] — 2026-10-28
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Containers without a Docker health check now "recover" properly.** Many images
|
||||
don't define a health check, so Docker reports them as `NO HEALTH CHECK` even when
|
||||
they work. Inspector only treated `HEALTHY` as back to normal, so for these
|
||||
containers:
|
||||
- the error count never reset, and one-off failures days apart added up to a restart;
|
||||
- no "recovered" alert was sent;
|
||||
- with *Restart, then redeploy*, the next unrelated problem skipped the restart and
|
||||
redeployed the whole stack.
|
||||
|
||||
Now `NO HEALTH CHECK` counts as working (unless you made it a restart trigger): the
|
||||
count resets, the "recovered" alert is sent ("back to running (no health check)"),
|
||||
and the next problem starts with a plain restart. Containers with a health check
|
||||
behave exactly as before.
|
||||
|
||||
---
|
||||
|
||||
## [0.9.3] — 2026-10-28
|
||||
|
||||
### Changed
|
||||
|
||||
- **Every time follows the `TZ` of the Inspector stack.** Before, the dashboard,
|
||||
Events page and alerts were always in US Pacific time while the maintenance window
|
||||
and logs used `TZ`, so anyone outside Pacific saw times hours off (e.g. a 02:00
|
||||
Berlin window shown as 5:00 PM). Now the dashboard, the Events page (including
|
||||
where each day starts), alert times, cooldown times and logs all use `TZ`.
|
||||
- Alert times include the zone abbreviation (`10-28 09:27 AM CET`), and "Cooldown
|
||||
until" in alerts, event messages and logs is a local time instead of a UTC
|
||||
timestamp.
|
||||
- The active time zone is shown at the bottom of every Settings page, on the Events
|
||||
page and on the Maintenance page.
|
||||
- **If you never set `TZ`**, times are now shown in UTC instead of Pacific. Set `TZ`
|
||||
in the stack (e.g. `America/New_York`). An unrecognized name logs a warning and
|
||||
falls back to UTC.
|
||||
|
||||
### Docs
|
||||
|
||||
- INSTALL: `TZ` is now marked as needed, with a "Why the time zone matters" section
|
||||
(maintenance window, alerts, dashboard, logs), how to find your zone name, and a
|
||||
troubleshooting row. Guide: a Time zone section and a first-login check. Compose
|
||||
and README updated.
|
||||
|
||||
---
|
||||
|
||||
## [0.9.2] — 2026-10-28
|
||||
|
||||
Fixes from a code and security review. None of them change how containers are
|
||||
monitored or recovered.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A bad value saved from Settings can no longer stop Inspector from starting.**
|
||||
The forms now reject invalid times (maintenance start/end), unknown log levels and
|
||||
out-of-range numbers (for example `interval_seconds: 0`, which made the loop spin).
|
||||
If `config.yml` already has such a value, Inspector starts with the default and
|
||||
logs a warning instead of crashing in a restart loop.
|
||||
- **The dashboard password can no longer be cleared.** Saving an empty password (or
|
||||
username) is refused, and an empty password never signs anyone in. Before, clearing
|
||||
it left the running dashboard open to anyone.
|
||||
- **Passwords with non-ASCII characters (é, ü, £ …) now work.** Before, every login
|
||||
attempt with one crashed.
|
||||
- The Events page no longer errors for a moment after a container is removed.
|
||||
- The HTTP check hint no longer suggests `302` as an expected status; redirects are
|
||||
followed, so the final page's code is what counts.
|
||||
|
||||
### Security
|
||||
|
||||
- **Sign-in throttling:** after 5 failed sign-ins (form or Basic Auth) from one address
|
||||
within 5 minutes, that address is refused for the rest of the window.
|
||||
- **Changing the dashboard username or password signs out existing sessions.**
|
||||
Expired sessions are cleaned up.
|
||||
- Request bodies are limited to 1 MB; an invalid `Content-Length` gets a 400.
|
||||
- The Gotify token is sent in the `X-Gotify-Key` header instead of the URL, so it no
|
||||
longer appears in Gotify or reverse-proxy access logs.
|
||||
- Portainer token environment variables must be named `PORTAINER_…` (a name already in
|
||||
your `config.yml` keeps working), so Test connection can't be used to send other
|
||||
container secrets to an arbitrary URL.
|
||||
- Single quotes are escaped in HTML, and the delete confirmations build their text safely.
|
||||
- Two settings saves at the same moment can no longer overwrite each other.
|
||||
- `requests` updated to 2.32.5 (CVE-2024-47081).
|
||||
|
||||
---
|
||||
|
||||
## [0.9.1] — 2026-10-28
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A dependency on a disabled container no longer blocks recovery forever.** If
|
||||
`depends_on` named a container that is disabled in Inspector, the dependent app was
|
||||
never restarted. Inspector now only waits for dependencies it is watching, and the
|
||||
container page warns when the selected dependency is disabled.
|
||||
- **`PORTAINER_API_TOKEN` now reaches the container.** The public `compose.yml` didn't
|
||||
pass it in, so setting it as a Portainer stack variable had no effect. Existing
|
||||
stacks need the line `- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}` added under
|
||||
`environment:` (Stacks → inspector → Editor).
|
||||
- The "depends_on … is disabled or not found" warning was logged on every check
|
||||
(since 0.8.0). It is now logged once, and again only when the container settings
|
||||
change.
|
||||
|
||||
### Changed
|
||||
|
||||
- Guide: new sections **6. Apps behind a VPN container** (why depends_on matters, how to
|
||||
spot a VPN-routed app, setup, what Inspector does) and **7. Portainer stack redeploy**
|
||||
(why it's needed, what a redeploy does to the whole stack, requirements, where to find
|
||||
every value, adding the token, the other settings, Test connection errors).
|
||||
- Clearer form hints: `vpn_routed` only affects logging; `depends_on` works for any
|
||||
dependency; where to find the endpoint ID.
|
||||
|
||||
---
|
||||
|
||||
## [0.9.0] — 2026-10-28
|
||||
|
||||
### Added
|
||||
|
||||
- **Every container option is now in the dashboard.** The container page has three
|
||||
new groups:
|
||||
- **Reachability**: HTTP check (URL, timeout, expected status) and Uptime Kuma
|
||||
monitor.
|
||||
- **Dependencies**: `depends_on` (pick from your containers) and `vpn_routed`.
|
||||
- **Recovery**: `recovery_action` with plain-language descriptions, plus the
|
||||
Portainer environment and stack name for stack redeploys.
|
||||
- **Integrations settings.** New **Uptime Kuma** and **Portainer** pages under
|
||||
Settings. Portainer environments can be added, edited, renamed (containers
|
||||
that use them are updated) and deleted (refused while a container uses one).
|
||||
The API token can be an environment variable name (recommended) or a masked
|
||||
value saved in `config.yml`.
|
||||
- **Test connection** buttons for Uptime Kuma (lists monitors) and each Portainer
|
||||
environment (lists stacks on the endpoint), using the values in the form even
|
||||
before saving.
|
||||
- The All containers grid shows each container's extras (HTTP check, Uptime Kuma,
|
||||
dependency, recovery action).
|
||||
- Guide: new sections on reachability checks and apps behind a VPN container.
|
||||
|
||||
### Changed
|
||||
|
||||
- Uptime Kuma and Portainer changes take effect on the next check; Inspector
|
||||
rebuilds those clients when their settings change instead of needing a restart.
|
||||
- Saving a container form unchanged now leaves `config.yml` byte-identical. Blank
|
||||
optional fields are removed instead of written as empty values (`color: ''`,
|
||||
`maintenance_window: false`), and an unchanged `restart_on` list keeps its
|
||||
order and the comments after it. An explicit `recovery_action: restart_container`
|
||||
is kept.
|
||||
|
||||
---
|
||||
|
||||
## [0.8.0] — 2026-10-27
|
||||
|
||||
### Added
|
||||
|
||||
+30
-1
@@ -52,10 +52,38 @@ TZ=America/Los_Angeles
|
||||
|---|---|---|
|
||||
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
|
||||
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
|
||||
| `TZ` | Your [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), so times on the dashboard are local. | Recommended |
|
||||
| `TZ` | **Your time zone**, e.g. `America/New_York` or `Europe/Berlin`. Without it Inspector uses UTC, so the maintenance window, alerts and dashboard times are off by your UTC offset. See [Why the time zone matters](#why-the-time-zone-matters). | **Yes** (default `UTC`) |
|
||||
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
|
||||
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
|
||||
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
|
||||
| `PORTAINER_API_TOKEN` | A Portainer access token. Only needed if you use **Portainer stack redeploy** for VPN-routed apps; see the in-app guide (Help → Portainer). | No |
|
||||
|
||||
### Why the time zone matters
|
||||
|
||||
Inspector uses `TZ` for every time it shows or acts on. If it's wrong, things
|
||||
still work, but at the wrong hour or with confusing times:
|
||||
|
||||
- **The maintenance window runs in this time zone.** A window of `02:00–05:00`
|
||||
with `TZ` left at UTC runs at 02:00 UTC, which is the middle of the evening in
|
||||
the Americas or mid-morning in Asia. Inspector would then restart containers
|
||||
during your nightly backups, and ignore real problems during the day.
|
||||
- **Alerts** (Gotify / Pushover) show the time of the problem in this zone, with
|
||||
its abbreviation, e.g. `10-28 03:15 AM CET`.
|
||||
- **The dashboard and Events page** show times in this zone, and the Events
|
||||
page's days run midnight to midnight in this zone.
|
||||
- **Logs** are stamped in this zone.
|
||||
|
||||
**How to set it:** use your zone's name from the
|
||||
[list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
|
||||
("TZ identifier" column), e.g. `America/New_York`, `America/Chicago`,
|
||||
`Europe/London`, `Europe/Berlin`, `Australia/Sydney`. Abbreviations like `EST` or
|
||||
`CET` don't work. On Unraid, **Settings → Date and Time** shows the zone your
|
||||
server uses.
|
||||
|
||||
**How to check it:** the bottom of every Settings page shows
|
||||
`time zone: <name>`. If it says `UTC` and you're not on UTC, set `TZ` in the
|
||||
stack and click **Update the stack**. If the name you set isn't recognized,
|
||||
Inspector logs a warning and uses UTC.
|
||||
|
||||
## 3. Deploy and sign in
|
||||
|
||||
@@ -115,6 +143,7 @@ starts and restarts the ones you add to it.
|
||||
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
|
||||
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
|
||||
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
|
||||
| Times are a few hours off, or the maintenance window runs at the wrong time | `TZ` is missing or wrong. Check `time zone:` at the bottom of Settings, set `TZ` to your zone name (e.g. `Europe/London`), and update the stack. |
|
||||
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
|
||||
|
||||
## Security notes
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
<p align="center">
|
||||
<img src="docs/logo.png" alt="Inspector logo" width="140">
|
||||
<img src="docs/logo_outline.png" alt="Inspector logo" width="140">
|
||||
</p>
|
||||
|
||||
<h1 align="center">Inspector</h1>
|
||||
@@ -35,6 +35,8 @@ restarts them when they break, and tells your phone about it.</p>
|
||||
- **Stays out of the way when you're working.** Put a container in maintenance or
|
||||
mute its alerts with one click, or set a nightly maintenance window for backups
|
||||
and updates.
|
||||
- **Works in your time zone.** Set `TZ` in the stack and the maintenance window,
|
||||
alerts, dashboard and logs all follow it.
|
||||
|
||||
## Install
|
||||
|
||||
@@ -52,11 +54,11 @@ guide (the same guide is in this repo at
|
||||
download it and open it in a browser to read it here).
|
||||
|
||||
| Add a container | Restart rules |
|
||||
|---|---|
|
||||
| ------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
||||
|  |  |
|
||||
|
||||
| Notifications | Events history |
|
||||
|---|---|
|
||||
| ------------------------------------------------------------------------------- | --------------------------------------------------------- |
|
||||
|  |  |
|
||||
|
||||
## What it isn't
|
||||
@@ -69,18 +71,20 @@ download it and open it in a browser to read it here).
|
||||
|
||||
## Advanced options
|
||||
|
||||
Most people never need these. They live in `config.yml` (in your
|
||||
`INSPECTOR_DATA/config` folder) rather than the dashboard; see
|
||||
[`config.example.yml`](./config.example.yml) for every option with comments.
|
||||
Most people never need these. They are all on each container's page in the
|
||||
dashboard (and in `config.yml`, see
|
||||
[`config.example.yml`](./config.example.yml) for every option with comments).
|
||||
|
||||
- **HTTP checks**: mark a container `UNREACHABLE` when a URL stops answering,
|
||||
even if Docker says it's healthy (useful for apps behind a VPN container).
|
||||
- **Uptime Kuma**: use a Kuma status page as an extra reachability signal.
|
||||
**Test connection** lists the monitors Inspector can see.
|
||||
- **Dependencies** (`depends_on`): don't restart an app while the container it
|
||||
relies on (e.g. a VPN) is down.
|
||||
- **Portainer stack redeploy**: for containers that share a VPN container's
|
||||
network, recover by redeploying the whole stack instead of restarting one
|
||||
container.
|
||||
container. Set up the Portainer connection under **Settings → Portainer**;
|
||||
**Test connection** lists your stacks.
|
||||
|
||||
## Security
|
||||
|
||||
@@ -89,7 +93,8 @@ and that socket is effectively root on your server. So:
|
||||
|
||||
- Keep the dashboard on your home network, or put it behind a VPN or a reverse
|
||||
proxy with authentication. Don't expose port 8080 to the internet.
|
||||
- Use a real dashboard password.
|
||||
- Use a real dashboard password. Repeated failed sign-ins from one address are
|
||||
blocked for a few minutes.
|
||||
- Inspector itself runs as an unprivileged user; it starts as root only to detect
|
||||
the socket's permissions and fix ownership of its own folders.
|
||||
|
||||
|
||||
+6
-1
@@ -7,13 +7,17 @@
|
||||
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
|
||||
# INSPECTOR_PASSWORD Dashboard password (username is "admin").
|
||||
# Only used on first start; change it later in Settings.
|
||||
# TZ Your timezone, e.g. America/Los_Angeles
|
||||
# TZ Your time zone name, e.g. America/New_York, Europe/Berlin.
|
||||
# Important: the maintenance window, alerts, dashboard and
|
||||
# logs all use it. Left unset, everything is in UTC.
|
||||
#
|
||||
# Optional:
|
||||
# INSPECTOR_PORT Port for the dashboard on your server (default 8080).
|
||||
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
|
||||
# PUID / PGID User/group Inspector runs as (default 1000/1000).
|
||||
# Unraid users often prefer 99 / 100.
|
||||
# PORTAINER_API_TOKEN Portainer access token, only if you use stack redeploy
|
||||
# (Settings -> Portainer in the dashboard).
|
||||
#
|
||||
# No config file to create, no docker group ID to look up: Inspector creates
|
||||
# its config on first start and detects Docker socket permissions by itself.
|
||||
@@ -36,3 +40,4 @@ services:
|
||||
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
|
||||
- PUID=${PUID:-1000}
|
||||
- PGID=${PGID:-1000}
|
||||
- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}
|
||||
Executable
BIN
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
+1
-1
@@ -1,5 +1,5 @@
|
||||
docker==7.1.0
|
||||
PyYAML==6.0.2
|
||||
requests==2.32.3
|
||||
requests==2.32.5
|
||||
colorlog==6.9.0
|
||||
ruamel.yaml==0.18.6
|
||||
@@ -1 +1 @@
|
||||
__version__ = "0.8.0"
|
||||
__version__ = "0.9.4"
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
@@ -78,10 +78,12 @@
|
||||
<a href="#status">3. The Status page</a>
|
||||
<a href="#notifications">4. Notifications</a>
|
||||
<a href="#containers">5. Managing containers</a>
|
||||
<a href="#events">6. Events history</a>
|
||||
<a href="#settings">7. Other settings</a>
|
||||
<a href="#restart">8. "Requires restart" fields</a>
|
||||
<a href="#trouble">9. Troubleshooting</a>
|
||||
<a href="#vpn">6. Apps behind a VPN</a>
|
||||
<a href="#portainer">7. Portainer stack redeploy</a>
|
||||
<a href="#events">8. Events history</a>
|
||||
<a href="#settings">9. Other settings</a>
|
||||
<a href="#restart">10. "Requires restart" fields</a>
|
||||
<a href="#trouble">11. Troubleshooting</a>
|
||||
</nav>
|
||||
|
||||
<main>
|
||||
@@ -101,6 +103,8 @@ dashboard; you never need to touch a config file.</p>
|
||||
<i>Inspector dashboard login</i> with the username and password.</li>
|
||||
<li>Open <code>http://<your-server>:8080</code> and sign in.</li>
|
||||
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
|
||||
<li>Check the <b>time zone</b>: at the bottom of any Settings page it says
|
||||
<code>time zone: …</code>. If that's not your zone, see <a href="#settings">Time zone</a>.</li>
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
@@ -175,7 +179,7 @@ Each card shows:</p>
|
||||
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
|
||||
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
|
||||
the reason appears next to the button.</li>
|
||||
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 8</a>),
|
||||
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 10</a>),
|
||||
because turning a channel on or off is a "requires restart" field.</li>
|
||||
</ol>
|
||||
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
|
||||
@@ -197,15 +201,182 @@ does <i>not</i> touch the container itself.</p>
|
||||
<tr><th>Status</th><th>When it happens</th></tr>
|
||||
<tr><td>UNHEALTHY</td><td>The container's own health check is failing.</td></tr>
|
||||
<tr><td>NOT RUNNING</td><td>Stopped, exited, or crashed.</td></tr>
|
||||
<tr><td>UNREACHABLE</td><td>Running, but an HTTP check failed (advanced, set in config.yml).</td></tr>
|
||||
<tr><td>UNREACHABLE</td><td>Running, but its HTTP check or Uptime Kuma monitor says it's down (see below).</td></tr>
|
||||
<tr><td>ERROR</td><td>Inspector couldn't check it (Docker API error).</td></tr>
|
||||
<tr><td>STARTING</td><td>Still in its health-check start period. Usually leave this off.</td></tr>
|
||||
<tr><td>HEALTHY / NO HEALTH CHECK</td><td>Normal states; almost never a reason to restart.</td></tr>
|
||||
</table>
|
||||
|
||||
<h3>Reachability: catching apps that look fine but aren't</h3>
|
||||
<p>Docker only knows whether a container is running and whether its own health check passes. An
|
||||
app can pass both and still be unusable, for example when it routes through a VPN container
|
||||
whose tunnel dropped. Under <b>Reachability</b> on the container page you can add:</p>
|
||||
<ul>
|
||||
<li><b>HTTP check</b>: a URL Inspector requests every check (for example
|
||||
<code>http://192.168.1.10:9696/ping</code>). Set <b>expected_status</b> to the code the app
|
||||
returns when it's healthy; most return 200, some return 401 or 302.</li>
|
||||
<li><b>Uptime Kuma monitor</b>: if you already run Uptime Kuma, name one of its monitors. Set up
|
||||
the connection first in <b>Settings → Uptime Kuma</b>; <b>Test connection</b> lists the
|
||||
monitor names it can see.</li>
|
||||
</ul>
|
||||
<p>A failed check reports <b>UNREACHABLE</b>. Tick UNREACHABLE in the restart triggers if you want
|
||||
Inspector to act on it.</p>
|
||||
<figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups">
|
||||
<figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure>
|
||||
|
||||
</section>
|
||||
|
||||
<section id="vpn">
|
||||
<h2>6. Apps behind a VPN container</h2>
|
||||
<h3>Why these apps need special handling</h3>
|
||||
<p>A common setup routes download and indexer apps through a VPN container. Those apps don't get a
|
||||
network of their own: their compose file says <code>network_mode: container:<vpn></code> (or
|
||||
<code>service:<vpn></code>) and all their traffic goes through the VPN container. That causes
|
||||
two problems for a watchdog:</p>
|
||||
<ul>
|
||||
<li><b>When the VPN breaks, every app behind it breaks at the same time.</b> Restarting the apps
|
||||
does nothing while the tunnel is down. Without help, Inspector would restart each app over and
|
||||
over, use up their hourly restart limits, and flood you with alerts, all for a problem in a
|
||||
different container.</li>
|
||||
<li><b>When the VPN container is recreated</b> (an image update, or someone redeploys its stack),
|
||||
it gets a new container ID. The apps still point at the old one, so a normal restart fails with
|
||||
errors like <i>"No such container"</i> or <i>"joining network namespace"</i>. The only fix is
|
||||
to recreate the apps too, which is what a Portainer stack redeploy does
|
||||
(<a href="#portainer">section 7</a>).</li>
|
||||
</ul>
|
||||
<h3>How to tell if an app is behind a VPN</h3>
|
||||
<ul>
|
||||
<li>In Portainer open <b>Stacks → your stack → Editor</b> and look for
|
||||
<code>network_mode: container:…</code> or <code>network_mode: service:…</code> on the app.</li>
|
||||
<li>Or open <b>Containers → the app</b>: under <b>Network</b> it shows the VPN container instead
|
||||
of a normal network, and it has no ports of its own (they're published on the VPN container).</li>
|
||||
</ul>
|
||||
<h3>Setting it up</h3>
|
||||
<ol class="steps">
|
||||
<li><b>Add the VPN container to Inspector and keep it enabled</b> (restart on UNHEALTHY and NOT
|
||||
RUNNING is a good start). Inspector can only wait for containers it is watching: if the VPN
|
||||
container is disabled, its apps are recovered normally, as if there were no dependency, and the
|
||||
container page shows a warning.</li>
|
||||
<li>On each app behind it, set <b>Dependencies → depends_on</b> to the VPN container.</li>
|
||||
<li>Turn on <b>vpn_routed</b>. This only changes the log: Inspector writes down when it holds off
|
||||
on the app because the VPN is down. The waiting itself comes from depends_on.</li>
|
||||
<li>Add an <b>HTTP check</b> (<a href="#containers">section 5</a>) using your server's IP and the
|
||||
port published on the VPN container, and tick <b>UNREACHABLE</b>. Docker often still reports
|
||||
these apps as healthy when the tunnel is down; the HTTP check notices.</li>
|
||||
<li>Under <b>Recovery</b>, choose <b>Restart, then redeploy stack if that fails</b> and enter the
|
||||
stack name. This needs the Portainer connection from <a href="#portainer">section 7</a>.</li>
|
||||
</ol>
|
||||
<h3>What Inspector then does</h3>
|
||||
<p>Every check, it looks at the VPN container first. While the VPN is anything other than healthy
|
||||
(or starting, or running without a health check), the apps that depend on it are not restarted;
|
||||
Inspector fixes the VPN first. Once the VPN is healthy again, the apps are handled normally: a
|
||||
restart first, and a stack redeploy if the restart fails or doesn't help.</p>
|
||||
<p class="note"><b>depends_on isn't only for VPNs.</b> Use it for any "B needs A" pair, such as an
|
||||
app and its database: Inspector won't restart the app while its database is down.</p>
|
||||
</section>
|
||||
|
||||
<section id="portainer">
|
||||
<h2>7. Portainer stack redeploy</h2>
|
||||
<h3>Why Inspector needs Portainer</h3>
|
||||
<p>Inspector talks to Docker directly, so it can restart or start a container. It can't
|
||||
<i>recreate</i> one: that needs the container's compose definition, which lives in Portainer. A
|
||||
stack redeploy asks Portainer to do exactly what its <b>Update the stack</b> button does. That's
|
||||
how VPN-routed apps get reattached to a recreated VPN container. You only need this if you use one
|
||||
of the two "redeploy" recovery actions.</p>
|
||||
<p class="note warn"><b>A redeploy affects the whole stack.</b> Portainer pulls the latest images
|
||||
and recreates <i>every</i> container in that stack, not just the failing app, so all of them blink
|
||||
off for a moment. That's why <b>Restart, then redeploy stack if that fails</b> is the recommended
|
||||
choice: most problems are fixed by the plain restart, and the redeploy is only used when that
|
||||
doesn't work.</p>
|
||||
<h3>Requirements</h3>
|
||||
<ul>
|
||||
<li>The app must belong to a stack that was <b>created in Portainer</b>. In <b>Stacks</b>, stacks
|
||||
created elsewhere (for example with <code>docker compose</code> on the command line) are marked
|
||||
<b>Limited</b>, and Portainer can't redeploy them.</li>
|
||||
<li>An access token for a Portainer user who can manage that stack. An administrator's token is
|
||||
the simplest.</li>
|
||||
</ul>
|
||||
|
||||
<h3>Step 1: find the values</h3>
|
||||
<table>
|
||||
<tr><th>Setting</th><th>Where to find it</th></tr>
|
||||
<tr><td>base_url</td><td>The address you open Portainer at, without anything after the port:
|
||||
usually <code>http://<server-ip>:9000</code> or <code>https://<server-ip>:9443</code>.
|
||||
Use the server's IP, not <code>localhost</code>: Inspector runs in its own container.</td></tr>
|
||||
<tr><td>endpoint_id</td><td>On Portainer's <b>Home</b> page, click your environment (often
|
||||
<b>local</b>). The browser address then ends like <code>#!/2/docker/dashboard</code>: the
|
||||
number after <code>#!/</code> is the endpoint ID (here <code>2</code>).</td></tr>
|
||||
<tr><td>stack name</td><td><b>Stacks</b> list, <b>Name</b> column, spelled exactly the same.
|
||||
This goes on each app's container page, not on the Portainer page.</td></tr>
|
||||
<tr><td>verify_ssl</td><td>Leave off for <code>http://</code> and for Portainer's built-in
|
||||
self-signed certificate on 9443. Turn on only if Portainer has a real certificate.</td></tr>
|
||||
<tr><td>API token</td><td>Click your username (top right) → <b>My account</b> →
|
||||
<b>Access tokens</b> → <b>Add access token</b>. Name it <code>inspector</code> and copy the
|
||||
token straight away: Portainer shows it only once.</td></tr>
|
||||
</table>
|
||||
|
||||
<h3>Step 2: give Inspector the token</h3>
|
||||
<p><b>Recommended: an environment variable.</b> The token then never lands in
|
||||
<code>config.yml</code> or its backups.</p>
|
||||
<ol class="steps">
|
||||
<li>In Portainer open <b>Stacks → inspector → Editor</b>.</li>
|
||||
<li>Check that the compose file has this line under <code>environment:</code> (it's in the
|
||||
compose file from version 0.9.1 on; add it if your stack is older):
|
||||
<code>- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}</code></li>
|
||||
<li>Under <b>Environment variables</b> click <b>Add an environment variable</b>: name
|
||||
<code>PORTAINER_API_TOKEN</code>, value the token.</li>
|
||||
<li>Click <b>Update the stack</b>. Inspector restarts with the token available.</li>
|
||||
</ol>
|
||||
<p><b>Alternative: save it in Inspector.</b> In the next step choose <b>Saved in config.yml</b>
|
||||
and paste the token. It is masked in the dashboard, but anyone who can read Inspector's config
|
||||
folder (or a backup of it) can read the token.</p>
|
||||
|
||||
<h3>Step 3: connect Inspector</h3>
|
||||
<ol class="steps">
|
||||
<li>Open <b>Settings → Portainer</b>. Under <b>Add environment</b>, give it a name (your own
|
||||
label, e.g. <code>home</code>) and fill in base_url, endpoint_id and verify_ssl.</li>
|
||||
<li><b>API token stored as</b>: <i>Environment variable</i> with the name
|
||||
<code>PORTAINER_API_TOKEN</code>, or <i>Saved in config.yml</i> with the token.</li>
|
||||
<li>Click <b>Test connection</b>. It should list the stacks on that endpoint; check that your
|
||||
app's stack is there. Then <b>Save new environment</b>.</li>
|
||||
<li>At the top of the page turn <b>enabled</b> on, pick the <b>default_environment</b>, and
|
||||
<b>Save Portainer</b>.</li>
|
||||
<li>On each app's container page, set the recovery action and stack name
|
||||
(<a href="#vpn">section 6</a>). Leave <b>portainer_environment</b> on <i>Default</i> unless
|
||||
you connected more than one Portainer.</li>
|
||||
</ol>
|
||||
<figure><img src="settings-portainer.png" alt="Portainer settings">
|
||||
<figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure>
|
||||
|
||||
<h3>The other settings on the page</h3>
|
||||
<table>
|
||||
<tr><th>Setting</th><th>What it does</th></tr>
|
||||
<tr><td>default_environment</td><td>Which connection apps use when they don't pick one. With a
|
||||
single environment you can leave it on Automatic.</td></tr>
|
||||
<tr><td>stack_redeploy_cooldown_seconds</td><td>Minimum time between two redeploys of the same
|
||||
stack (default 300). Several apps often share a stack; this stops them triggering a redeploy
|
||||
each.</td></tr>
|
||||
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector waits after a redeploy for the
|
||||
stack to come up before judging it again (default 90).</td></tr>
|
||||
</table>
|
||||
|
||||
<h3>If Test connection fails</h3>
|
||||
<table>
|
||||
<tr><th>Message</th><th>Fix</th></tr>
|
||||
<tr><td>environment variable … is not set</td><td>The compose line from step 2 is missing, or
|
||||
the stack wasn't updated after adding the variable.</td></tr>
|
||||
<tr><td>the API token was rejected</td><td>Token copied wrong, deleted, or from a user without
|
||||
access. Create a new one.</td></tr>
|
||||
<tr><td>no stacks found on endpoint N</td><td>Wrong endpoint ID; check the number in the
|
||||
Portainer address again.</td></tr>
|
||||
<tr><td>request failed …</td><td>Wrong address or port, <code>http</code> vs <code>https</code>
|
||||
mixed up, or <code>localhost</code> used instead of the server IP. With https and a
|
||||
self-signed certificate, turn verify_ssl off.</td></tr>
|
||||
</table>
|
||||
</section>
|
||||
|
||||
<section id="events">
|
||||
<h2>6. Events history</h2>
|
||||
<h2>8. Events history</h2>
|
||||
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
|
||||
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
||||
<figure><img src="events.png" alt="Events timeline">
|
||||
@@ -213,7 +384,7 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
||||
</section>
|
||||
|
||||
<section id="settings">
|
||||
<h2>7. Other settings</h2>
|
||||
<h2>9. Other settings</h2>
|
||||
<figure><img src="settings-inspector.png" alt="Inspector loop settings">
|
||||
<figcaption>Settings → Inspector loop.</figcaption></figure>
|
||||
<table>
|
||||
@@ -223,15 +394,33 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
||||
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
|
||||
<tr><td>Event history</td><td>How many days of events to keep.</td></tr>
|
||||
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
|
||||
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
|
||||
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00, in your <a href="#settings">time zone</a>) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
|
||||
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
|
||||
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
|
||||
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#portainer">section 7</a>.</td></tr>
|
||||
</table>
|
||||
<p>At the bottom of every settings page is a strip showing where the config file lives and
|
||||
whether Inspector can write to it.</p>
|
||||
<p>At the bottom of every settings page is a strip showing where the config file lives,
|
||||
whether Inspector can write to it, and which time zone it is using.</p>
|
||||
|
||||
<h3>Time zone</h3>
|
||||
<p>Inspector uses the <code>TZ</code> setting of its Portainer stack for every time it shows or acts on:</p>
|
||||
<ul>
|
||||
<li><b>The maintenance window runs in this zone.</b> A 02:00–05:00 window with the wrong zone
|
||||
runs hours off, so Inspector could restart containers during your backups and ignore
|
||||
problems at other times.</li>
|
||||
<li>Alert times (with the zone, e.g. <code>03:15 AM CET</code>), dashboard times, the Events
|
||||
page's days (midnight to midnight), and logs.</li>
|
||||
</ul>
|
||||
<p>To change it: in Portainer open <b>Stacks → inspector → Editor</b>, set the <code>TZ</code>
|
||||
environment variable to your zone's name, e.g. <code>America/New_York</code>,
|
||||
<code>Europe/London</code> or <code>Australia/Sydney</code> (names from the
|
||||
<a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">list of time zones</a>;
|
||||
abbreviations like <code>EST</code> don't work), then click <b>Update the stack</b>. Without
|
||||
<code>TZ</code>, Inspector uses UTC.</p>
|
||||
</section>
|
||||
|
||||
<section id="restart">
|
||||
<h2>8. "Requires restart" fields</h2>
|
||||
<h2>10. "Requires restart" fields</h2>
|
||||
<p>Most changes apply on the next check. Fields marked
|
||||
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
|
||||
dashboard on or off) only take effect after Inspector restarts:</p>
|
||||
@@ -243,7 +432,7 @@ dashboard on or off) only take effect after Inspector restarts:</p>
|
||||
</section>
|
||||
|
||||
<section id="trouble">
|
||||
<h2>9. Troubleshooting</h2>
|
||||
<h2>11. Troubleshooting</h2>
|
||||
<table>
|
||||
<tr><th>Problem</th><th>Fix</th></tr>
|
||||
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
|
||||
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 42 KiB |
+34
-4
@@ -1,10 +1,12 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import threading
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import time
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
@@ -23,6 +25,30 @@ VALID_STATUSES = {
|
||||
"UNREACHABLE",
|
||||
}
|
||||
|
||||
log = logging.getLogger("inspector.config")
|
||||
|
||||
# Names logging.setLevel() accepts; anything else would crash at startup.
|
||||
VALID_LOG_LEVELS = {"CRITICAL", "FATAL", "ERROR", "WARN", "WARNING", "INFO", "DEBUG", "NOTSET"}
|
||||
|
||||
|
||||
def valid_hhmm(value: str) -> bool:
|
||||
"""True if `value` parses the way the maintenance tracker parses it ("HH:MM")."""
|
||||
try:
|
||||
hour, minute = str(value).split(":")
|
||||
time(int(hour), int(minute))
|
||||
return True
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _startup_safe(value, default, ok: bool, key: str):
|
||||
"""Return `value`, or `default` with a warning when it would crash Inspector."""
|
||||
if ok:
|
||||
return value
|
||||
log.warning("config: invalid %s %r; using %r instead", key, value, default)
|
||||
return default
|
||||
|
||||
|
||||
VALID_RECOVERY_ACTIONS = {
|
||||
"restart_container",
|
||||
"start_container",
|
||||
@@ -190,9 +216,11 @@ def build_config_from_raw(raw: dict) -> AppConfig:
|
||||
to validate an edited raw dict before writing it back to disk.
|
||||
"""
|
||||
insp_raw = raw.get("inspector", {}) or {}
|
||||
interval = int(insp_raw.get("interval_seconds", 60))
|
||||
log_level = str(insp_raw.get("log_level", "INFO")).upper()
|
||||
inspector = InspectorConfig(
|
||||
interval_seconds=int(insp_raw.get("interval_seconds", 60)),
|
||||
log_level=str(insp_raw.get("log_level", "INFO")).upper(),
|
||||
interval_seconds=_startup_safe(interval, 1, interval >= 1, "inspector.interval_seconds"),
|
||||
log_level=_startup_safe(log_level, "INFO", log_level in VALID_LOG_LEVELS, "inspector.log_level"),
|
||||
dry_run=bool(insp_raw.get("dry_run", False)),
|
||||
log_dir=str(insp_raw.get("log_dir", "/app/logs")),
|
||||
log_filename=str(insp_raw.get("log_filename", "inspector.log")),
|
||||
@@ -237,10 +265,12 @@ def build_config_from_raw(raw: dict) -> AppConfig:
|
||||
)
|
||||
|
||||
maint_raw = raw.get("maintenance") or {}
|
||||
maint_start = str(maint_raw.get("start", "00:00"))
|
||||
maint_end = str(maint_raw.get("end", "06:00"))
|
||||
maintenance = MaintenanceConfig(
|
||||
enabled=bool(maint_raw.get("enabled", False)),
|
||||
start=str(maint_raw.get("start", "00:00")),
|
||||
end=str(maint_raw.get("end", "06:00")),
|
||||
start=_startup_safe(maint_start, "00:00", valid_hhmm(maint_start), "maintenance.start"),
|
||||
end=_startup_safe(maint_end, "06:00", valid_hhmm(maint_end), "maintenance.end"),
|
||||
dir=str(maint_raw.get("dir", "/app/data/maintenance")),
|
||||
max_hours=int(maint_raw.get("max_hours", 8)),
|
||||
)
|
||||
|
||||
+788
-77
File diff suppressed because it is too large.
Load diff
@@ -1,14 +1,49 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from zoneinfo import ZoneInfo
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timezone, tzinfo
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
|
||||
PACIFIC = ZoneInfo("America/Los_Angeles")
|
||||
log = logging.getLogger("inspector.formatting")
|
||||
|
||||
|
||||
def format_timestamp(value: datetime | str | None) -> str:
|
||||
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'.
|
||||
def _resolve_local_tz(env: dict | None = None) -> tuple[tzinfo, str]:
|
||||
"""The zone every displayed time uses, plus its name.
|
||||
|
||||
Follows the same rules as the C library (which drives log timestamps and
|
||||
the maintenance window), so all times Inspector shows agree:
|
||||
TZ set to a zone name -> that zone; TZ unset -> the container's
|
||||
/etc/localtime; anything else -> UTC.
|
||||
"""
|
||||
env = os.environ if env is None else env
|
||||
name = (env.get("TZ") or "").strip().lstrip(":")
|
||||
if name:
|
||||
try:
|
||||
return ZoneInfo(name), name
|
||||
except (ZoneInfoNotFoundError, ValueError):
|
||||
log.warning(
|
||||
"TZ=%r is not a known time zone name (e.g. Europe/Berlin); showing times in UTC",
|
||||
name,
|
||||
)
|
||||
return timezone.utc, "UTC"
|
||||
try:
|
||||
with open("/etc/localtime", "rb") as fh:
|
||||
zone = ZoneInfo.from_file(fh, key="localtime")
|
||||
label = datetime.now(zone).tzname() or "localtime"
|
||||
return zone, label
|
||||
except (OSError, ValueError):
|
||||
return timezone.utc, "UTC"
|
||||
|
||||
|
||||
# Resolved once at import; the container's TZ doesn't change while it runs.
|
||||
LOCAL_TZ, LOCAL_TZ_NAME = _resolve_local_tz()
|
||||
|
||||
|
||||
def format_timestamp(value: datetime | str | None, *, with_zone: bool = False) -> str:
|
||||
"""Render a timestamp in the configured local zone, e.g. '06-14 03:45 PM'
|
||||
(with_zone adds the abbreviation: '06-14 03:45 PM CET').
|
||||
|
||||
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
|
||||
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
|
||||
@@ -24,4 +59,8 @@ def format_timestamp(value: datetime | str | None) -> str:
|
||||
return value
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p")
|
||||
local = value.astimezone(LOCAL_TZ)
|
||||
text = local.strftime("%m-%d %I:%M %p")
|
||||
if with_zone:
|
||||
text += f" {local.tzname()}"
|
||||
return text
|
||||
+46
-14
@@ -63,6 +63,19 @@ def _request_stop(signum, _frame):
|
||||
_stop = True
|
||||
|
||||
|
||||
def _is_normal(status: str | None, container_config: ContainerConfig) -> bool:
|
||||
"""True when `status` means the container is working.
|
||||
|
||||
HEALTHY always counts. NO HEALTH CHECK (running, but the image defines no
|
||||
health check) counts too, unless the user made it a restart trigger.
|
||||
Many images have no health check, and without this their error count and
|
||||
restart episode would never reset.
|
||||
"""
|
||||
if status == HEALTHY:
|
||||
return True
|
||||
return status == NO_HEALTH_CHECK and NO_HEALTH_CHECK not in container_config.restart_on
|
||||
|
||||
|
||||
def _apply_reachability_override(
|
||||
container_config: ContainerConfig,
|
||||
result: HealthResult,
|
||||
@@ -158,19 +171,25 @@ def _check_one(
|
||||
message=result.message,
|
||||
)
|
||||
|
||||
is_normal = _is_normal(state.current_status, container_config)
|
||||
if (
|
||||
state.previous_status is not None
|
||||
and state.previous_status != HEALTHY
|
||||
and state.current_status == HEALTHY
|
||||
and not _is_normal(state.previous_status, container_config)
|
||||
and is_normal
|
||||
# HEALTHY here means "a recovery was already announced", for either status.
|
||||
and state.last_notification_state != HEALTHY
|
||||
):
|
||||
log.info("%-*s recovered: %s -> HEALTHY", width, name, state.previous_status)
|
||||
log.info("%-*s recovered: %s -> %s", width, name, state.previous_status, state.current_status)
|
||||
back_to = (
|
||||
"**HEALTHY**" if state.current_status == HEALTHY
|
||||
else "**running** (no health check)"
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} recovered",
|
||||
message=(
|
||||
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
|
||||
f"**{name}** is back to {back_to} (was *{state.previous_status}*).\n\n"
|
||||
f"- Errors before recovery: **{state.error_count}**\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
@@ -178,7 +197,7 @@ def _check_one(
|
||||
container=name,
|
||||
event_type=EVENT_RECOVERY,
|
||||
old_status=state.previous_status,
|
||||
new_status=HEALTHY,
|
||||
new_status=state.current_status,
|
||||
error_count=state.error_count,
|
||||
)
|
||||
state.last_notification_state = HEALTHY
|
||||
@@ -196,7 +215,7 @@ def _check_one(
|
||||
f"**{name}** is missing from Docker.\n\n"
|
||||
+ (f"- Detail: {result.message}\n" if result.message else "")
|
||||
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=7,
|
||||
)
|
||||
@@ -240,7 +259,7 @@ def _check_one(
|
||||
+ ".\n\n"
|
||||
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
|
||||
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=5,
|
||||
)
|
||||
@@ -253,9 +272,9 @@ def _check_one(
|
||||
message=result.message,
|
||||
)
|
||||
state.last_notification_state = result.status
|
||||
elif result.status == HEALTHY:
|
||||
elif is_normal:
|
||||
if state.error_count != 0:
|
||||
log.info("%-*s healthy again; resetting error count", width, name)
|
||||
log.info("%-*s back to normal; resetting error count", width, name)
|
||||
state.error_count = 0
|
||||
state.restarted_this_episode = False
|
||||
|
||||
@@ -276,7 +295,7 @@ def _check_one(
|
||||
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
|
||||
f"- Current status: **{result.status}**\n"
|
||||
f"- Restarts paused until the hourly window clears\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=9,
|
||||
)
|
||||
@@ -293,8 +312,10 @@ def _check_one(
|
||||
|
||||
if ok:
|
||||
dep = container_config.depends_on
|
||||
if dep:
|
||||
dep_status = states[dep].current_status if dep in states else None
|
||||
# Only wait on a dependency Inspector is watching. A disabled one has
|
||||
# no state, and waiting on it would block recovery forever.
|
||||
if dep and dep in states:
|
||||
dep_status = states[dep].current_status
|
||||
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
|
||||
if container_config.vpn_routed:
|
||||
log.info(
|
||||
@@ -417,6 +438,7 @@ def run_loop(config_ref: ConfigRef) -> int:
|
||||
|
||||
check_order = _topological_order(enabled_configs)
|
||||
portainer_client = PortainerClient(app.integrations.portainer)
|
||||
integrations_cfg = app.integrations
|
||||
stack_cooldowns: dict[str, datetime] = {}
|
||||
|
||||
dashboard = DashboardServer(
|
||||
@@ -453,8 +475,18 @@ def run_loop(config_ref: ConfigRef) -> int:
|
||||
for gone in states.keys() - current_enabled.keys():
|
||||
states.pop(gone, None)
|
||||
log.info("Stopped watching container (removed or disabled): %s", gone)
|
||||
if current_enabled != enabled_configs:
|
||||
# Only re-order (and re-log ordering warnings) when the config changed.
|
||||
check_order = _topological_order(current_enabled)
|
||||
enabled_configs = current_enabled
|
||||
check_order = _topological_order(enabled_configs)
|
||||
|
||||
# Integration clients hold their config; rebuild them when the
|
||||
# settings page changes Uptime Kuma / Portainer.
|
||||
if app.integrations != integrations_cfg:
|
||||
integrations_cfg = app.integrations
|
||||
uptime_kuma_client = UptimeKumaClient(integrations_cfg.uptime_kuma)
|
||||
portainer_client = PortainerClient(integrations_cfg.portainer)
|
||||
log.info("Integration settings changed; reloaded Uptime Kuma / Portainer clients")
|
||||
|
||||
uptime_kuma_snapshot = uptime_kuma_client.fetch()
|
||||
for name in check_order:
|
||||
|
||||
@@ -33,9 +33,10 @@ class GotifyNotifier:
|
||||
},
|
||||
}
|
||||
try:
|
||||
# Header, not ?token=, so the token stays out of access logs.
|
||||
resp = requests.post(
|
||||
url,
|
||||
params={"token": self._config.token},
|
||||
headers={"X-Gotify-Key": self._config.token},
|
||||
json=payload,
|
||||
timeout=self._timeout,
|
||||
)
|
||||
|
||||
+17
-16
@@ -175,7 +175,8 @@ def _perform_restart_then_redeploy(
|
||||
- the container is still failing on a later recovery cycle
|
||||
(`state.restarted_this_episode` is already set), meaning the restart didn't stick.
|
||||
|
||||
The episode flag resets when the container is next seen HEALTHY (see main._check_one).
|
||||
The episode flag resets when the container is next seen working: HEALTHY, or
|
||||
NO HEALTH CHECK when that isn't a restart trigger (see main._is_normal).
|
||||
"""
|
||||
name = container_config.name
|
||||
|
||||
@@ -228,7 +229,7 @@ def _perform_restart_then_redeploy(
|
||||
f"**{result.status}** check(s).\n\n"
|
||||
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
|
||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
@@ -242,7 +243,7 @@ def _perform_restart_then_redeploy(
|
||||
state.restart_limit_notified = False
|
||||
log.info(
|
||||
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
|
||||
width, name, state.cooldown_until.isoformat(),
|
||||
width, name, format_timestamp(state.cooldown_until, with_zone=True),
|
||||
)
|
||||
eventstore.record(
|
||||
container=name,
|
||||
@@ -250,15 +251,15 @@ def _perform_restart_then_redeploy(
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
||||
message=f"restart succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} restart completed",
|
||||
message=(
|
||||
f"**{name}** was successfully restarted by Docker.\n\n"
|
||||
f"- Previous status: *{result.status}*\n"
|
||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
@@ -321,7 +322,7 @@ def _perform_docker_action(
|
||||
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
|
||||
f"**{result.status}** check(s).\n\n"
|
||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
@@ -335,22 +336,22 @@ def _perform_docker_action(
|
||||
state.record_restart(now, container_config.cooldown_seconds)
|
||||
state.last_notification_state = "RESTARTED"
|
||||
state.restart_limit_notified = False
|
||||
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat())
|
||||
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, format_timestamp(state.cooldown_until, with_zone=True))
|
||||
eventstore.record(
|
||||
container=name,
|
||||
event_type=EVENT_RESTART,
|
||||
old_status=state.previous_status,
|
||||
new_status=result.status,
|
||||
error_count=state.error_count,
|
||||
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
||||
message=f"{verb} succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
|
||||
)
|
||||
notifier.notify(
|
||||
title=f"✅ Inspector: {name} {verb} completed",
|
||||
message=(
|
||||
f"**{name}** was successfully {verb}ed by Docker.\n\n"
|
||||
f"- Previous status: *{result.status}*\n"
|
||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=3,
|
||||
)
|
||||
@@ -362,7 +363,7 @@ def _perform_docker_action(
|
||||
f"{verb.capitalize()} of **{name}** failed.\n\n"
|
||||
f"- Status: **{result.status}**\n"
|
||||
f"- Error count: **{state.error_count}**\n"
|
||||
f"- Time: {format_timestamp(now)}\n\n"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
|
||||
f"```\n{err}\n```"
|
||||
),
|
||||
priority=9,
|
||||
@@ -469,7 +470,7 @@ def _perform_portainer_redeploy(
|
||||
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
|
||||
f"to recover **{name}**.\n\n"
|
||||
f"- Status: **{result.status}**\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
@@ -483,7 +484,7 @@ def _perform_portainer_redeploy(
|
||||
message=(
|
||||
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
|
||||
f"- Triggered by: **{name}** ({result.status})\n"
|
||||
f"- Time: {format_timestamp(now)}\n\n"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
|
||||
f"```\n{redeploy_result.message}\n```"
|
||||
),
|
||||
priority=9,
|
||||
@@ -516,7 +517,7 @@ def _perform_portainer_redeploy(
|
||||
f"successfully redeployed to recover **{name}**.\n\n"
|
||||
f"- Triggered by: *{result.status}*\n"
|
||||
f"- Post-redeploy wait: {wait_seconds}s\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=4,
|
||||
)
|
||||
@@ -536,7 +537,7 @@ def _perform_portainer_redeploy(
|
||||
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
|
||||
f"for **{name}** still reports a failure.\n\n"
|
||||
f"- Recheck result: *{recheck.message}*\n"
|
||||
f"- Time: {format_timestamp(now)}"
|
||||
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||
),
|
||||
priority=6,
|
||||
)
|
||||
|
||||
@@ -41,20 +41,26 @@ class UptimeKumaClient:
|
||||
and bool(self._config.status_page_slug)
|
||||
)
|
||||
|
||||
def _heartbeat_url(self) -> str:
|
||||
return f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
|
||||
|
||||
def fetch(self) -> HeartbeatSnapshot:
|
||||
if not self.enabled:
|
||||
return _EMPTY
|
||||
|
||||
url = f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
|
||||
try:
|
||||
resp = requests.get(url, timeout=self._timeout)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return self._parse(data)
|
||||
return self.fetch_or_raise()
|
||||
except Exception as exc:
|
||||
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", url, exc)
|
||||
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", self._heartbeat_url(), exc)
|
||||
return _EMPTY
|
||||
|
||||
def fetch_or_raise(self) -> HeartbeatSnapshot:
|
||||
"""Like fetch(), but lets request/parse errors propagate (used by the
|
||||
settings page's "Test connection" button)."""
|
||||
resp = requests.get(self._heartbeat_url(), timeout=self._timeout)
|
||||
resp.raise_for_status()
|
||||
return self._parse(resp.json())
|
||||
|
||||
def _parse(self, data: dict) -> HeartbeatSnapshot:
|
||||
# Expected shape:
|
||||
# {"publicGroupList": [{"monitorList": [{"id": 1, "name": "..."}]}],
|
||||
|
||||
Reference in new issue
Block a user