7 Commits
18 changed files with 1398 additions and 158 deletions

No files matched your search

+163
View File
@@ -7,6 +7,169 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
---
## [0.9.5] — 2026-10-28
### Fixed
- **A stack redeploy no longer freezes Inspector.** After a successful Portainer
redeploy, Inspector used to stop everything for `post_redeploy_wait_seconds`
(default 90 s): no other container was checked, dashboard times stood still, and
stopping Inspector during the wait wasn't clean. Now the redeployed app is left to
settle while every other container keeps being checked, and its one-time HTTP
re-check (with the "recheck failed" alert) happens on its first check after the wait.
- The redeployed app's cooldown now starts after the settle time. Before, the wait
ate into it (with the defaults, 300 s cooldown became 210 s).
---
## [0.9.4] — 2026-10-28
### Fixed
- **Containers without a Docker health check now "recover" properly.** Many images
don't define a health check, so Docker reports them as `NO HEALTH CHECK` even when
they work. Inspector only treated `HEALTHY` as back to normal, so for these
containers:
- the error count never reset, and one-off failures days apart added up to a restart;
- no "recovered" alert was sent;
- with *Restart, then redeploy*, the next unrelated problem skipped the restart and
redeployed the whole stack.
Now `NO HEALTH CHECK` counts as working (unless you made it a restart trigger): the
count resets, the "recovered" alert is sent ("back to running (no health check)"),
and the next problem starts with a plain restart. Containers with a health check
behave exactly as before.
---
## [0.9.3] — 2026-10-28
### Changed
- **Every time follows the `TZ` of the Inspector stack.** Before, the dashboard,
Events page and alerts were always in US Pacific time while the maintenance window
and logs used `TZ`, so anyone outside Pacific saw times hours off (e.g. a 02:00
Berlin window shown as 5:00 PM). Now the dashboard, the Events page (including
where each day starts), alert times, cooldown times and logs all use `TZ`.
- Alert times include the zone abbreviation (`10-28 09:27 AM CET`), and "Cooldown
until" in alerts, event messages and logs is a local time instead of a UTC
timestamp.
- The active time zone is shown at the bottom of every Settings page, on the Events
page and on the Maintenance page.
- **If you never set `TZ`**, times are now shown in UTC instead of Pacific. Set `TZ`
in the stack (e.g. `America/New_York`). An unrecognized name logs a warning and
falls back to UTC.
### Docs
- INSTALL: `TZ` is now marked as needed, with a "Why the time zone matters" section
(maintenance window, alerts, dashboard, logs), how to find your zone name, and a
troubleshooting row. Guide: a Time zone section and a first-login check. Compose
and README updated.
---
## [0.9.2] — 2026-10-28
Fixes from a code and security review. None of them change how containers are
monitored or recovered.
### Fixed
- **A bad value saved from Settings can no longer stop Inspector from starting.**
The forms now reject invalid times (maintenance start/end), unknown log levels and
out-of-range numbers (for example `interval_seconds: 0`, which made the loop spin).
If `config.yml` already has such a value, Inspector starts with the default and
logs a warning instead of crashing in a restart loop.
- **The dashboard password can no longer be cleared.** Saving an empty password (or
username) is refused, and an empty password never signs anyone in. Before, clearing
it left the running dashboard open to anyone.
- **Passwords with non-ASCII characters (é, ü, £ …) now work.** Before, every login
attempt with one crashed.
- The Events page no longer errors for a moment after a container is removed.
- The HTTP check hint no longer suggests `302` as an expected status; redirects are
followed, so the final page's code is what counts.
### Security
- **Sign-in throttling:** after 5 failed sign-ins (form or Basic Auth) from one address
within 5 minutes, that address is refused for the rest of the window.
- **Changing the dashboard username or password signs out existing sessions.**
Expired sessions are cleaned up.
- Request bodies are limited to 1 MB; an invalid `Content-Length` gets a 400.
- The Gotify token is sent in the `X-Gotify-Key` header instead of the URL, so it no
longer appears in Gotify or reverse-proxy access logs.
- Portainer token environment variables must be named `PORTAINER_…` (a name already in
your `config.yml` keeps working), so Test connection can't be used to send other
container secrets to an arbitrary URL.
- Single quotes are escaped in HTML, and the delete confirmations build their text safely.
- Two settings saves at the same moment can no longer overwrite each other.
- `requests` updated to 2.32.5 (CVE-2024-47081).
---
## [0.9.1] — 2026-10-28
### Fixed
- **A dependency on a disabled container no longer blocks recovery forever.** If
`depends_on` named a container that is disabled in Inspector, the dependent app was
never restarted. Inspector now only waits for dependencies it is watching, and the
container page warns when the selected dependency is disabled.
- **`PORTAINER_API_TOKEN` now reaches the container.** The public `compose.yml` didn't
pass it in, so setting it as a Portainer stack variable had no effect. Existing
stacks need the line `- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}` added under
`environment:` (Stacks → inspector → Editor).
- The "depends_on … is disabled or not found" warning was logged on every check
(since 0.8.0). It is now logged once, and again only when the container settings
change.
### Changed
- Guide: new sections **6. Apps behind a VPN container** (why depends_on matters, how to
spot a VPN-routed app, setup, what Inspector does) and **7. Portainer stack redeploy**
(why it's needed, what a redeploy does to the whole stack, requirements, where to find
every value, adding the token, the other settings, Test connection errors).
- Clearer form hints: `vpn_routed` only affects logging; `depends_on` works for any
dependency; where to find the endpoint ID.
---
## [0.9.0] — 2026-10-28
### Added
- **Every container option is now in the dashboard.** The container page has three
new groups:
- **Reachability**: HTTP check (URL, timeout, expected status) and Uptime Kuma
monitor.
- **Dependencies**: `depends_on` (pick from your containers) and `vpn_routed`.
- **Recovery**: `recovery_action` with plain-language descriptions, plus the
Portainer environment and stack name for stack redeploys.
- **Integrations settings.** New **Uptime Kuma** and **Portainer** pages under
Settings. Portainer environments can be added, edited, renamed (containers
that use them are updated) and deleted (refused while a container uses one).
The API token can be an environment variable name (recommended) or a masked
value saved in `config.yml`.
- **Test connection** buttons for Uptime Kuma (lists monitors) and each Portainer
environment (lists stacks on the endpoint), using the values in the form even
before saving.
- The All containers grid shows each container's extras (HTTP check, Uptime Kuma,
dependency, recovery action).
- Guide: new sections on reachability checks and apps behind a VPN container.
### Changed
- Uptime Kuma and Portainer changes take effect on the next check; Inspector
rebuilds those clients when their settings change instead of needing a restart.
- Saving a container form unchanged now leaves `config.yml` byte-identical. Blank
optional fields are removed instead of written as empty values (`color: ''`,
`maintenance_window: false`), and an unchanged `restart_on` list keeps its
order and the comments after it. An explicit `recovery_action: restart_container`
is kept.
---
## [0.8.0] — 2026-10-27
### Added
+30 -1
View File
@@ -52,10 +52,38 @@ TZ=America/Los_Angeles
|---|---|---|
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
| `TZ` | Your [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), so times on the dashboard are local. | Recommended |
| `TZ` | **Your time zone**, e.g. `America/New_York` or `Europe/Berlin`. Without it Inspector uses UTC, so the maintenance window, alerts and dashboard times are off by your UTC offset. See [Why the time zone matters](#why-the-time-zone-matters). | **Yes** (default `UTC`) |
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
| `PORTAINER_API_TOKEN` | A Portainer access token. Only needed if you use **Portainer stack redeploy** for VPN-routed apps; see the in-app guide (Help → Portainer). | No |
### Why the time zone matters
Inspector uses `TZ` for every time it shows or acts on. If it's wrong, things
still work, but at the wrong hour or with confusing times:
- **The maintenance window runs in this time zone.** A window of `02:00–05:00`
with `TZ` left at UTC runs at 02:00 UTC, which is the middle of the evening in
the Americas or mid-morning in Asia. Inspector would then restart containers
during your nightly backups, and ignore real problems during the day.
- **Alerts** (Gotify / Pushover) show the time of the problem in this zone, with
its abbreviation, e.g. `10-28 03:15 AM CET`.
- **The dashboard and Events page** show times in this zone, and the Events
page's days run midnight to midnight in this zone.
- **Logs** are stamped in this zone.
**How to set it:** use your zone's name from the
[list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
("TZ identifier" column), e.g. `America/New_York`, `America/Chicago`,
`Europe/London`, `Europe/Berlin`, `Australia/Sydney`. Abbreviations like `EST` or
`CET` don't work. On Unraid, **Settings → Date and Time** shows the zone your
server uses.
**How to check it:** the bottom of every Settings page shows
`time zone: <name>`. If it says `UTC` and you're not on UTC, set `TZ` in the
stack and click **Update the stack**. If the name you set isn't recognized,
Inspector logs a warning and uses UTC.
## 3. Deploy and sign in
@@ -115,6 +143,7 @@ starts and restarts the ones you add to it.
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
| Times are a few hours off, or the maintenance window runs at the wrong time | `TZ` is missing or wrong. Check `time zone:` at the bottom of Settings, set `TZ` to your zone name (e.g. `Europe/London`), and update the stack. |
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
## Security notes
+13 -8
View File
@@ -1,5 +1,5 @@
<p align="center">
<img src="docs/logo.png" alt="Inspector logo" width="140">
<img src="docs/logo_outline.png" alt="Inspector logo" width="140">
</p>
<h1 align="center">Inspector</h1>
@@ -35,6 +35,8 @@ restarts them when they break, and tells your phone about it.</p>
- **Stays out of the way when you're working.** Put a container in maintenance or
mute its alerts with one click, or set a nightly maintenance window for backups
and updates.
- **Works in your time zone.** Set `TZ` in the stack and the maintenance window,
alerts, dashboard and logs all follow it.
## Install
@@ -52,11 +54,11 @@ guide (the same guide is in this repo at
download it and open it in a browser to read it here).
| Add a container | Restart rules |
|---|---|
| ------------------------------------------------------------------- | -------------------------------------------------------------------- |
| ![New container form](src/inspector/assets/guide/container-new.png) | ![Container settings](src/inspector/assets/guide/container-edit.png) |
| Notifications | Events history |
|---|---|
| ------------------------------------------------------------------------------- | --------------------------------------------------------- |
| ![Notification settings](src/inspector/assets/guide/settings-notifications.png) | ![Events timeline](src/inspector/assets/guide/events.png) |
## What it isn't
@@ -69,18 +71,20 @@ download it and open it in a browser to read it here).
## Advanced options
Most people never need these. They live in `config.yml` (in your
`INSPECTOR_DATA/config` folder) rather than the dashboard; see
[`config.example.yml`](./config.example.yml) for every option with comments.
Most people never need these. They are all on each container's page in the
dashboard (and in `config.yml`, see
[`config.example.yml`](./config.example.yml) for every option with comments).
- **HTTP checks**: mark a container `UNREACHABLE` when a URL stops answering,
even if Docker says it's healthy (useful for apps behind a VPN container).
- **Uptime Kuma**: use a Kuma status page as an extra reachability signal.
**Test connection** lists the monitors Inspector can see.
- **Dependencies** (`depends_on`): don't restart an app while the container it
relies on (e.g. a VPN) is down.
- **Portainer stack redeploy**: for containers that share a VPN container's
network, recover by redeploying the whole stack instead of restarting one
container.
container. Set up the Portainer connection under **Settings → Portainer**;
**Test connection** lists your stacks.
## Security
@@ -89,7 +93,8 @@ and that socket is effectively root on your server. So:
- Keep the dashboard on your home network, or put it behind a VPN or a reverse
proxy with authentication. Don't expose port 8080 to the internet.
- Use a real dashboard password.
- Use a real dashboard password. Repeated failed sign-ins from one address are
blocked for a few minutes.
- Inspector itself runs as an unprivileged user; it starts as root only to detect
the socket's permissions and fix ownership of its own folders.
+6 -1
View File
@@ -7,13 +7,17 @@
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
# INSPECTOR_PASSWORD Dashboard password (username is "admin").
# Only used on first start; change it later in Settings.
# TZ Your timezone, e.g. America/Los_Angeles
# TZ Your time zone name, e.g. America/New_York, Europe/Berlin.
# Important: the maintenance window, alerts, dashboard and
# logs all use it. Left unset, everything is in UTC.
#
# Optional:
# INSPECTOR_PORT Port for the dashboard on your server (default 8080).
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
# PUID / PGID User/group Inspector runs as (default 1000/1000).
# Unraid users often prefer 99 / 100.
# PORTAINER_API_TOKEN Portainer access token, only if you use stack redeploy
# (Settings -> Portainer in the dashboard).
#
# No config file to create, no docker group ID to look up: Inspector creates
# its config on first start and detects Docker socket permissions by itself.
@@ -36,3 +40,4 @@ services:
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
- PUID=${PUID:-1000}
- PGID=${PGID:-1000}
- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

+1 -1
View File
@@ -1,5 +1,5 @@
docker==7.1.0
PyYAML==6.0.2
requests==2.32.3
requests==2.32.5
colorlog==6.9.0
ruamel.yaml==0.18.6
+1 -1
View File
@@ -1 +1 @@
__version__ = "0.8.0"
__version__ = "0.9.5"
Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

+203 -13
View File
@@ -78,10 +78,12 @@
<a href="#status">3. The Status page</a>
<a href="#notifications">4. Notifications</a>
<a href="#containers">5. Managing containers</a>
<a href="#events">6. Events history</a>
<a href="#settings">7. Other settings</a>
<a href="#restart">8. "Requires restart" fields</a>
<a href="#trouble">9. Troubleshooting</a>
<a href="#vpn">6. Apps behind a VPN</a>
<a href="#portainer">7. Portainer stack redeploy</a>
<a href="#events">8. Events history</a>
<a href="#settings">9. Other settings</a>
<a href="#restart">10. "Requires restart" fields</a>
<a href="#trouble">11. Troubleshooting</a>
</nav>
<main>
@@ -101,6 +103,8 @@ dashboard; you never need to touch a config file.</p>
<i>Inspector dashboard login</i> with the username and password.</li>
<li>Open <code>http://&lt;your-server&gt;:8080</code> and sign in.</li>
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
<li>Check the <b>time zone</b>: at the bottom of any Settings page it says
<code>time zone: …</code>. If that's not your zone, see <a href="#settings">Time zone</a>.</li>
</ol>
</section>
@@ -175,7 +179,7 @@ Each card shows:</p>
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
the reason appears next to the button.</li>
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 8</a>),
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 10</a>),
because turning a channel on or off is a "requires restart" field.</li>
</ol>
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
@@ -197,15 +201,183 @@ does <i>not</i> touch the container itself.</p>
<tr><th>Status</th><th>When it happens</th></tr>
<tr><td>UNHEALTHY</td><td>The container's own health check is failing.</td></tr>
<tr><td>NOT RUNNING</td><td>Stopped, exited, or crashed.</td></tr>
<tr><td>UNREACHABLE</td><td>Running, but an HTTP check failed (advanced, set in config.yml).</td></tr>
<tr><td>UNREACHABLE</td><td>Running, but its HTTP check or Uptime Kuma monitor says it's down (see below).</td></tr>
<tr><td>ERROR</td><td>Inspector couldn't check it (Docker API error).</td></tr>
<tr><td>STARTING</td><td>Still in its health-check start period. Usually leave this off.</td></tr>
<tr><td>HEALTHY / NO HEALTH CHECK</td><td>Normal states; almost never a reason to restart.</td></tr>
</table>
<h3>Reachability: catching apps that look fine but aren't</h3>
<p>Docker only knows whether a container is running and whether its own health check passes. An
app can pass both and still be unusable, for example when it routes through a VPN container
whose tunnel dropped. Under <b>Reachability</b> on the container page you can add:</p>
<ul>
<li><b>HTTP check</b>: a URL Inspector requests every check (for example
<code>http://192.168.1.10:9696/ping</code>). Set <b>expected_status</b> to the code the app
returns when it's healthy; most return 200, some return 401 or 302.</li>
<li><b>Uptime Kuma monitor</b>: if you already run Uptime Kuma, name one of its monitors. Set up
the connection first in <b>Settings → Uptime Kuma</b>; <b>Test connection</b> lists the
monitor names it can see.</li>
</ul>
<p>A failed check reports <b>UNREACHABLE</b>. Tick UNREACHABLE in the restart triggers if you want
Inspector to act on it.</p>
<figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups">
<figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure>
</section>
<section id="vpn">
<h2>6. Apps behind a VPN container</h2>
<h3>Why these apps need special handling</h3>
<p>A common setup routes download and indexer apps through a VPN container. Those apps don't get a
network of their own: their compose file says <code>network_mode: container:&lt;vpn&gt;</code> (or
<code>service:&lt;vpn&gt;</code>) and all their traffic goes through the VPN container. That causes
two problems for a watchdog:</p>
<ul>
<li><b>When the VPN breaks, every app behind it breaks at the same time.</b> Restarting the apps
does nothing while the tunnel is down. Without help, Inspector would restart each app over and
over, use up their hourly restart limits, and flood you with alerts, all for a problem in a
different container.</li>
<li><b>When the VPN container is recreated</b> (an image update, or someone redeploys its stack),
it gets a new container ID. The apps still point at the old one, so a normal restart fails with
errors like <i>"No such container"</i> or <i>"joining network namespace"</i>. The only fix is
to recreate the apps too, which is what a Portainer stack redeploy does
(<a href="#portainer">section 7</a>).</li>
</ul>
<h3>How to tell if an app is behind a VPN</h3>
<ul>
<li>In Portainer open <b>Stacks → your stack → Editor</b> and look for
<code>network_mode: container:…</code> or <code>network_mode: service:…</code> on the app.</li>
<li>Or open <b>Containers → the app</b>: under <b>Network</b> it shows the VPN container instead
of a normal network, and it has no ports of its own (they're published on the VPN container).</li>
</ul>
<h3>Setting it up</h3>
<ol class="steps">
<li><b>Add the VPN container to Inspector and keep it enabled</b> (restart on UNHEALTHY and NOT
RUNNING is a good start). Inspector can only wait for containers it is watching: if the VPN
container is disabled, its apps are recovered normally, as if there were no dependency, and the
container page shows a warning.</li>
<li>On each app behind it, set <b>Dependencies → depends_on</b> to the VPN container.</li>
<li>Turn on <b>vpn_routed</b>. This only changes the log: Inspector writes down when it holds off
on the app because the VPN is down. The waiting itself comes from depends_on.</li>
<li>Add an <b>HTTP check</b> (<a href="#containers">section 5</a>) using your server's IP and the
port published on the VPN container, and tick <b>UNREACHABLE</b>. Docker often still reports
these apps as healthy when the tunnel is down; the HTTP check notices.</li>
<li>Under <b>Recovery</b>, choose <b>Restart, then redeploy stack if that fails</b> and enter the
stack name. This needs the Portainer connection from <a href="#portainer">section 7</a>.</li>
</ol>
<h3>What Inspector then does</h3>
<p>Every check, it looks at the VPN container first. While the VPN is anything other than healthy
(or starting, or running without a health check), the apps that depend on it are not restarted;
Inspector fixes the VPN first. Once the VPN is healthy again, the apps are handled normally: a
restart first, and a stack redeploy if the restart fails or doesn't help.</p>
<p class="note"><b>depends_on isn't only for VPNs.</b> Use it for any "B needs A" pair, such as an
app and its database: Inspector won't restart the app while its database is down.</p>
</section>
<section id="portainer">
<h2>7. Portainer stack redeploy</h2>
<h3>Why Inspector needs Portainer</h3>
<p>Inspector talks to Docker directly, so it can restart or start a container. It can't
<i>recreate</i> one: that needs the container's compose definition, which lives in Portainer. A
stack redeploy asks Portainer to do exactly what its <b>Update the stack</b> button does. That's
how VPN-routed apps get reattached to a recreated VPN container. You only need this if you use one
of the two "redeploy" recovery actions.</p>
<p class="note warn"><b>A redeploy affects the whole stack.</b> Portainer pulls the latest images
and recreates <i>every</i> container in that stack, not just the failing app, so all of them blink
off for a moment. That's why <b>Restart, then redeploy stack if that fails</b> is the recommended
choice: most problems are fixed by the plain restart, and the redeploy is only used when that
doesn't work.</p>
<h3>Requirements</h3>
<ul>
<li>The app must belong to a stack that was <b>created in Portainer</b>. In <b>Stacks</b>, stacks
created elsewhere (for example with <code>docker compose</code> on the command line) are marked
<b>Limited</b>, and Portainer can't redeploy them.</li>
<li>An access token for a Portainer user who can manage that stack. An administrator's token is
the simplest.</li>
</ul>
<h3>Step 1: find the values</h3>
<table>
<tr><th>Setting</th><th>Where to find it</th></tr>
<tr><td>base_url</td><td>The address you open Portainer at, without anything after the port:
usually <code>http://&lt;server-ip&gt;:9000</code> or <code>https://&lt;server-ip&gt;:9443</code>.
Use the server's IP, not <code>localhost</code>: Inspector runs in its own container.</td></tr>
<tr><td>endpoint_id</td><td>On Portainer's <b>Home</b> page, click your environment (often
<b>local</b>). The browser address then ends like <code>#!/2/docker/dashboard</code>: the
number after <code>#!/</code> is the endpoint ID (here <code>2</code>).</td></tr>
<tr><td>stack name</td><td><b>Stacks</b> list, <b>Name</b> column, spelled exactly the same.
This goes on each app's container page, not on the Portainer page.</td></tr>
<tr><td>verify_ssl</td><td>Leave off for <code>http://</code> and for Portainer's built-in
self-signed certificate on 9443. Turn on only if Portainer has a real certificate.</td></tr>
<tr><td>API token</td><td>Click your username (top right) → <b>My account</b> →
<b>Access tokens</b> → <b>Add access token</b>. Name it <code>inspector</code> and copy the
token straight away: Portainer shows it only once.</td></tr>
</table>
<h3>Step 2: give Inspector the token</h3>
<p><b>Recommended: an environment variable.</b> The token then never lands in
<code>config.yml</code> or its backups.</p>
<ol class="steps">
<li>In Portainer open <b>Stacks → inspector → Editor</b>.</li>
<li>Check that the compose file has this line under <code>environment:</code> (it's in the
compose file from version 0.9.1 on; add it if your stack is older):
<code>- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}</code></li>
<li>Under <b>Environment variables</b> click <b>Add an environment variable</b>: name
<code>PORTAINER_API_TOKEN</code>, value the token.</li>
<li>Click <b>Update the stack</b>. Inspector restarts with the token available.</li>
</ol>
<p><b>Alternative: save it in Inspector.</b> In the next step choose <b>Saved in config.yml</b>
and paste the token. It is masked in the dashboard, but anyone who can read Inspector's config
folder (or a backup of it) can read the token.</p>
<h3>Step 3: connect Inspector</h3>
<ol class="steps">
<li>Open <b>Settings → Portainer</b>. Under <b>Add environment</b>, give it a name (your own
label, e.g. <code>home</code>) and fill in base_url, endpoint_id and verify_ssl.</li>
<li><b>API token stored as</b>: <i>Environment variable</i> with the name
<code>PORTAINER_API_TOKEN</code>, or <i>Saved in config.yml</i> with the token.</li>
<li>Click <b>Test connection</b>. It should list the stacks on that endpoint; check that your
app's stack is there. Then <b>Save new environment</b>.</li>
<li>At the top of the page turn <b>enabled</b> on, pick the <b>default_environment</b>, and
<b>Save Portainer</b>.</li>
<li>On each app's container page, set the recovery action and stack name
(<a href="#vpn">section 6</a>). Leave <b>portainer_environment</b> on <i>Default</i> unless
you connected more than one Portainer.</li>
</ol>
<figure><img src="settings-portainer.png" alt="Portainer settings">
<figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure>
<h3>The other settings on the page</h3>
<table>
<tr><th>Setting</th><th>What it does</th></tr>
<tr><td>default_environment</td><td>Which connection apps use when they don't pick one. With a
single environment you can leave it on Automatic.</td></tr>
<tr><td>stack_redeploy_cooldown_seconds</td><td>Minimum time between two redeploys of the same
stack (default 300). Several apps often share a stack; this stops them triggering a redeploy
each.</td></tr>
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector gives the stack to come up after
a redeploy before judging that app again (default 90). Everything else keeps being checked
meanwhile.</td></tr>
</table>
<h3>If Test connection fails</h3>
<table>
<tr><th>Message</th><th>Fix</th></tr>
<tr><td>environment variable … is not set</td><td>The compose line from step 2 is missing, or
the stack wasn't updated after adding the variable.</td></tr>
<tr><td>the API token was rejected</td><td>Token copied wrong, deleted, or from a user without
access. Create a new one.</td></tr>
<tr><td>no stacks found on endpoint N</td><td>Wrong endpoint ID; check the number in the
Portainer address again.</td></tr>
<tr><td>request failed …</td><td>Wrong address or port, <code>http</code> vs <code>https</code>
mixed up, or <code>localhost</code> used instead of the server IP. With https and a
self-signed certificate, turn verify_ssl off.</td></tr>
</table>
</section>
<section id="events">
<h2>6. Events history</h2>
<h2>8. Events history</h2>
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
<figure><img src="events.png" alt="Events timeline">
@@ -213,7 +385,7 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
</section>
<section id="settings">
<h2>7. Other settings</h2>
<h2>9. Other settings</h2>
<figure><img src="settings-inspector.png" alt="Inspector loop settings">
<figcaption>Settings → Inspector loop.</figcaption></figure>
<table>
@@ -223,15 +395,33 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
<tr><td>Event history</td><td>How many days of events to keep.</td></tr>
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00, in your <a href="#settings">time zone</a>) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#portainer">section 7</a>.</td></tr>
</table>
<p>At the bottom of every settings page is a strip showing where the config file lives and
whether Inspector can write to it.</p>
<p>At the bottom of every settings page is a strip showing where the config file lives,
whether Inspector can write to it, and which time zone it is using.</p>
<h3>Time zone</h3>
<p>Inspector uses the <code>TZ</code> setting of its Portainer stack for every time it shows or acts on:</p>
<ul>
<li><b>The maintenance window runs in this zone.</b> A 02:00–05:00 window with the wrong zone
runs hours off, so Inspector could restart containers during your backups and ignore
problems at other times.</li>
<li>Alert times (with the zone, e.g. <code>03:15 AM CET</code>), dashboard times, the Events
page's days (midnight to midnight), and logs.</li>
</ul>
<p>To change it: in Portainer open <b>Stacks → inspector → Editor</b>, set the <code>TZ</code>
environment variable to your zone's name, e.g. <code>America/New_York</code>,
<code>Europe/London</code> or <code>Australia/Sydney</code> (names from the
<a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">list of time zones</a>;
abbreviations like <code>EST</code> don't work), then click <b>Update the stack</b>. Without
<code>TZ</code>, Inspector uses UTC.</p>
</section>
<section id="restart">
<h2>8. "Requires restart" fields</h2>
<h2>10. "Requires restart" fields</h2>
<p>Most changes apply on the next check. Fields marked
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
dashboard on or off) only take effect after Inspector restarts:</p>
@@ -243,7 +433,7 @@ dashboard on or off) only take effect after Inspector restarts:</p>
</section>
<section id="trouble">
<h2>9. Troubleshooting</h2>
<h2>11. Troubleshooting</h2>
<table>
<tr><th>Problem</th><th>Fix</th></tr>
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
Binary file not shown.

After

Width:  |  Height:  |  Size: 42 KiB

+34 -4
View File
@@ -1,10 +1,12 @@
from __future__ import annotations
import io
import logging
import os
import shutil
import threading
from dataclasses import dataclass, field
from datetime import time
from pathlib import Path
import yaml
@@ -23,6 +25,30 @@ VALID_STATUSES = {
"UNREACHABLE",
}
log = logging.getLogger("inspector.config")
# Names logging.setLevel() accepts; anything else would crash at startup.
VALID_LOG_LEVELS = {"CRITICAL", "FATAL", "ERROR", "WARN", "WARNING", "INFO", "DEBUG", "NOTSET"}
def valid_hhmm(value: str) -> bool:
"""True if `value` parses the way the maintenance tracker parses it ("HH:MM")."""
try:
hour, minute = str(value).split(":")
time(int(hour), int(minute))
return True
except (TypeError, ValueError):
return False
def _startup_safe(value, default, ok: bool, key: str):
"""Return `value`, or `default` with a warning when it would crash Inspector."""
if ok:
return value
log.warning("config: invalid %s %r; using %r instead", key, value, default)
return default
VALID_RECOVERY_ACTIONS = {
"restart_container",
"start_container",
@@ -190,9 +216,11 @@ def build_config_from_raw(raw: dict) -> AppConfig:
to validate an edited raw dict before writing it back to disk.
"""
insp_raw = raw.get("inspector", {}) or {}
interval = int(insp_raw.get("interval_seconds", 60))
log_level = str(insp_raw.get("log_level", "INFO")).upper()
inspector = InspectorConfig(
interval_seconds=int(insp_raw.get("interval_seconds", 60)),
log_level=str(insp_raw.get("log_level", "INFO")).upper(),
interval_seconds=_startup_safe(interval, 1, interval >= 1, "inspector.interval_seconds"),
log_level=_startup_safe(log_level, "INFO", log_level in VALID_LOG_LEVELS, "inspector.log_level"),
dry_run=bool(insp_raw.get("dry_run", False)),
log_dir=str(insp_raw.get("log_dir", "/app/logs")),
log_filename=str(insp_raw.get("log_filename", "inspector.log")),
@@ -237,10 +265,12 @@ def build_config_from_raw(raw: dict) -> AppConfig:
)
maint_raw = raw.get("maintenance") or {}
maint_start = str(maint_raw.get("start", "00:00"))
maint_end = str(maint_raw.get("end", "06:00"))
maintenance = MaintenanceConfig(
enabled=bool(maint_raw.get("enabled", False)),
start=str(maint_raw.get("start", "00:00")),
end=str(maint_raw.get("end", "06:00")),
start=_startup_safe(maint_start, "00:00", valid_hhmm(maint_start), "maintenance.start"),
end=_startup_safe(maint_end, "06:00", valid_hhmm(maint_end), "maintenance.end"),
dir=str(maint_raw.get("dir", "/app/data/maintenance")),
max_hours=int(maint_raw.get("max_hours", 8)),
)
File diff suppressed because it is too large. Load diff
+45 -6
View File
@@ -1,14 +1,49 @@
from __future__ import annotations
from datetime import datetime, timezone
from zoneinfo import ZoneInfo
import logging
import os
from datetime import datetime, timezone, tzinfo
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
PACIFIC = ZoneInfo("America/Los_Angeles")
log = logging.getLogger("inspector.formatting")
def format_timestamp(value: datetime | str | None) -> str:
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'.
def _resolve_local_tz(env: dict | None = None) -> tuple[tzinfo, str]:
"""The zone every displayed time uses, plus its name.
Follows the same rules as the C library (which drives log timestamps and
the maintenance window), so all times Inspector shows agree:
TZ set to a zone name -> that zone; TZ unset -> the container's
/etc/localtime; anything else -> UTC.
"""
env = os.environ if env is None else env
name = (env.get("TZ") or "").strip().lstrip(":")
if name:
try:
return ZoneInfo(name), name
except (ZoneInfoNotFoundError, ValueError):
log.warning(
"TZ=%r is not a known time zone name (e.g. Europe/Berlin); showing times in UTC",
name,
)
return timezone.utc, "UTC"
try:
with open("/etc/localtime", "rb") as fh:
zone = ZoneInfo.from_file(fh, key="localtime")
label = datetime.now(zone).tzname() or "localtime"
return zone, label
except (OSError, ValueError):
return timezone.utc, "UTC"
# Resolved once at import; the container's TZ doesn't change while it runs.
LOCAL_TZ, LOCAL_TZ_NAME = _resolve_local_tz()
def format_timestamp(value: datetime | str | None, *, with_zone: bool = False) -> str:
"""Render a timestamp in the configured local zone, e.g. '06-14 03:45 PM'
(with_zone adds the abbreviation: '06-14 03:45 PM CET').
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
@@ -24,4 +59,8 @@ def format_timestamp(value: datetime | str | None) -> str:
return value
if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc)
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p")
local = value.astimezone(LOCAL_TZ)
text = local.strftime("%m-%d %I:%M %p")
if with_zone:
text += f" {local.tzname()}"
return text
+52 -15
View File
@@ -41,7 +41,7 @@ from .maintenance import MaintenanceTracker
from .notification_mute import NotificationMuteTracker
from .notifier import GotifyNotifier, MultiNotifier, NullNotifier, PushoverNotifier
from .portainer_client import PortainerClient
from .recovery import perform_recovery
from .recovery import perform_recovery, run_post_redeploy_recheck
from .uptime_kuma import HeartbeatSnapshot, UptimeKumaClient
from .state import (
ContainerState,
@@ -63,6 +63,19 @@ def _request_stop(signum, _frame):
_stop = True
def _is_normal(status: str | None, container_config: ContainerConfig) -> bool:
"""True when `status` means the container is working.
HEALTHY always counts. NO HEALTH CHECK (running, but the image defines no
health check) counts too, unless the user made it a restart trigger.
Many images have no health check, and without this their error count and
restart episode would never reset.
"""
if status == HEALTHY:
return True
return status == NO_HEALTH_CHECK and NO_HEALTH_CHECK not in container_config.restart_on
def _apply_reachability_override(
container_config: ContainerConfig,
result: HealthResult,
@@ -140,6 +153,11 @@ def _check_one(
if notification_mute.is_muted(name):
notifier = NullNotifier()
if state.redeploy_recheck_at is not None and now >= state.redeploy_recheck_at:
run_post_redeploy_recheck(
container_config=container_config, state=state, notifier=notifier, now=now,
)
if state.previous_status != state.current_status:
log.info(
"%-*s status: %s -> %s%s",
@@ -158,19 +176,25 @@ def _check_one(
message=result.message,
)
is_normal = _is_normal(state.current_status, container_config)
if (
state.previous_status is not None
and state.previous_status != HEALTHY
and state.current_status == HEALTHY
and not _is_normal(state.previous_status, container_config)
and is_normal
# HEALTHY here means "a recovery was already announced", for either status.
and state.last_notification_state != HEALTHY
):
log.info("%-*s recovered: %s -> HEALTHY", width, name, state.previous_status)
log.info("%-*s recovered: %s -> %s", width, name, state.previous_status, state.current_status)
back_to = (
"**HEALTHY**" if state.current_status == HEALTHY
else "**running** (no health check)"
)
notifier.notify(
title=f"✅ Inspector: {name} recovered",
message=(
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
f"**{name}** is back to {back_to} (was *{state.previous_status}*).\n\n"
f"- Errors before recovery: **{state.error_count}**\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=3,
)
@@ -178,7 +202,7 @@ def _check_one(
container=name,
event_type=EVENT_RECOVERY,
old_status=state.previous_status,
new_status=HEALTHY,
new_status=state.current_status,
error_count=state.error_count,
)
state.last_notification_state = HEALTHY
@@ -196,7 +220,7 @@ def _check_one(
f"**{name}** is missing from Docker.\n\n"
+ (f"- Detail: {result.message}\n" if result.message else "")
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=7,
)
@@ -240,7 +264,7 @@ def _check_one(
+ ".\n\n"
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=5,
)
@@ -253,9 +277,9 @@ def _check_one(
message=result.message,
)
state.last_notification_state = result.status
elif result.status == HEALTHY:
elif is_normal:
if state.error_count != 0:
log.info("%-*s healthy again; resetting error count", width, name)
log.info("%-*s back to normal; resetting error count", width, name)
state.error_count = 0
state.restarted_this_episode = False
@@ -276,7 +300,7 @@ def _check_one(
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
f"- Current status: **{result.status}**\n"
f"- Restarts paused until the hourly window clears\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=9,
)
@@ -293,8 +317,10 @@ def _check_one(
if ok:
dep = container_config.depends_on
if dep:
dep_status = states[dep].current_status if dep in states else None
# Only wait on a dependency Inspector is watching. A disabled one has
# no state, and waiting on it would block recovery forever.
if dep and dep in states:
dep_status = states[dep].current_status
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
if container_config.vpn_routed:
log.info(
@@ -417,6 +443,7 @@ def run_loop(config_ref: ConfigRef) -> int:
check_order = _topological_order(enabled_configs)
portainer_client = PortainerClient(app.integrations.portainer)
integrations_cfg = app.integrations
stack_cooldowns: dict[str, datetime] = {}
dashboard = DashboardServer(
@@ -453,8 +480,18 @@ def run_loop(config_ref: ConfigRef) -> int:
for gone in states.keys() - current_enabled.keys():
states.pop(gone, None)
log.info("Stopped watching container (removed or disabled): %s", gone)
if current_enabled != enabled_configs:
# Only re-order (and re-log ordering warnings) when the config changed.
check_order = _topological_order(current_enabled)
enabled_configs = current_enabled
check_order = _topological_order(enabled_configs)
# Integration clients hold their config; rebuild them when the
# settings page changes Uptime Kuma / Portainer.
if app.integrations != integrations_cfg:
integrations_cfg = app.integrations
uptime_kuma_client = UptimeKumaClient(integrations_cfg.uptime_kuma)
portainer_client = PortainerClient(integrations_cfg.portainer)
log.info("Integration settings changed; reloaded Uptime Kuma / Portainer clients")
uptime_kuma_snapshot = uptime_kuma_client.fetch()
for name in check_order:
+2 -1
View File
@@ -33,9 +33,10 @@ class GotifyNotifier:
},
}
try:
# Header, not ?token=, so the token stays out of access logs.
resp = requests.post(
url,
params={"token": self._config.token},
headers={"X-Gotify-Key": self._config.token},
json=payload,
timeout=self._timeout,
)
+46 -24
View File
@@ -1,7 +1,6 @@
from __future__ import annotations
import logging
import time
from datetime import datetime, timedelta
from .config import AppConfig, ContainerConfig, PortainerConfig
@@ -175,7 +174,8 @@ def _perform_restart_then_redeploy(
- the container is still failing on a later recovery cycle
(`state.restarted_this_episode` is already set), meaning the restart didn't stick.
The episode flag resets when the container is next seen HEALTHY (see main._check_one).
The episode flag resets when the container is next seen working: HEALTHY, or
NO HEALTH CHECK when that isn't a restart trigger (see main._is_normal).
"""
name = container_config.name
@@ -228,7 +228,7 @@ def _perform_restart_then_redeploy(
f"**{result.status}** check(s).\n\n"
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
@@ -242,7 +242,7 @@ def _perform_restart_then_redeploy(
state.restart_limit_notified = False
log.info(
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
width, name, state.cooldown_until.isoformat(),
width, name, format_timestamp(state.cooldown_until, with_zone=True),
)
eventstore.record(
container=name,
@@ -250,15 +250,15 @@ def _perform_restart_then_redeploy(
old_status=state.previous_status,
new_status=result.status,
error_count=state.error_count,
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}",
message=f"restart succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
)
notifier.notify(
title=f"✅ Inspector: {name} restart completed",
message=(
f"**{name}** was successfully restarted by Docker.\n\n"
f"- Previous status: *{result.status}*\n"
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
f"- Time: {format_timestamp(now)}"
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=3,
)
@@ -321,7 +321,7 @@ def _perform_docker_action(
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
f"**{result.status}** check(s).\n\n"
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
@@ -335,22 +335,22 @@ def _perform_docker_action(
state.record_restart(now, container_config.cooldown_seconds)
state.last_notification_state = "RESTARTED"
state.restart_limit_notified = False
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat())
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, format_timestamp(state.cooldown_until, with_zone=True))
eventstore.record(
container=name,
event_type=EVENT_RESTART,
old_status=state.previous_status,
new_status=result.status,
error_count=state.error_count,
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}",
message=f"{verb} succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
)
notifier.notify(
title=f"✅ Inspector: {name} {verb} completed",
message=(
f"**{name}** was successfully {verb}ed by Docker.\n\n"
f"- Previous status: *{result.status}*\n"
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
f"- Time: {format_timestamp(now)}"
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=3,
)
@@ -362,7 +362,7 @@ def _perform_docker_action(
f"{verb.capitalize()} of **{name}** failed.\n\n"
f"- Status: **{result.status}**\n"
f"- Error count: **{state.error_count}**\n"
f"- Time: {format_timestamp(now)}\n\n"
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
f"```\n{err}\n```"
),
priority=9,
@@ -469,7 +469,7 @@ def _perform_portainer_redeploy(
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
f"to recover **{name}**.\n\n"
f"- Status: **{result.status}**\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
@@ -483,7 +483,7 @@ def _perform_portainer_redeploy(
message=(
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
f"- Triggered by: **{name}** ({result.status})\n"
f"- Time: {format_timestamp(now)}\n\n"
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
f"```\n{redeploy_result.message}\n```"
),
priority=9,
@@ -516,29 +516,51 @@ def _perform_portainer_redeploy(
f"successfully redeployed to recover **{name}**.\n\n"
f"- Triggered by: *{result.status}*\n"
f"- Post-redeploy wait: {wait_seconds}s\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=4,
)
log.info("waiting %d seconds after stack redeploy", wait_seconds)
time.sleep(wait_seconds)
# Don't block the loop while the stack settles: keep this container in
# cooldown for the wait plus its normal cooldown, and do the one-time HTTP
# re-check on its first regular check after the wait (main._check_one).
log.info(
"%-*s letting stack %s settle for %ds; other containers keep being checked",
width, name, key, wait_seconds,
)
state.record_restart(now, wait_seconds + container_config.cooldown_seconds)
if container_config.http_check.enabled:
state.redeploy_recheck_at = now + timedelta(seconds=wait_seconds)
def run_post_redeploy_recheck(
*,
container_config: ContainerConfig,
state: ContainerState,
notifier: GotifyNotifier,
now: datetime,
) -> None:
"""One HTTP re-check after a stack redeploy has had time to settle.
Called by main._check_one once `state.redeploy_recheck_at` has passed;
alerts if the app still doesn't answer.
"""
state.redeploy_recheck_at = None
name = container_config.name
if not container_config.http_check.enabled:
return
recheck = check_http(container_config.http_check)
if recheck.ok:
log.info("recheck succeeded for %s", name)
else:
return
log.warning("recheck failed for %s: %s", name, recheck.message)
notifier.notify(
title=f"⚠️ Inspector: {name} recheck failed after redeploy",
message=(
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
f"Portainer stack **{container_config.portainer_stack_name}** was redeployed, but the HTTP recheck "
f"for **{name}** still reports a failure.\n\n"
f"- Recheck result: *{recheck.message}*\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
state.record_restart(now, container_config.cooldown_seconds)
+2
View File
@@ -29,6 +29,8 @@ class ContainerState:
restart_limit_notified: bool = False
restarted_this_episode: bool = False
last_check_at: datetime | None = None
# After a stack redeploy: when to do the one-time HTTP re-check.
redeploy_recheck_at: datetime | None = None
def in_cooldown(self, now: datetime) -> bool:
return self.cooldown_until is not None and now < self.cooldown_until
+12 -6
View File
@@ -41,20 +41,26 @@ class UptimeKumaClient:
and bool(self._config.status_page_slug)
)
def _heartbeat_url(self) -> str:
return f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
def fetch(self) -> HeartbeatSnapshot:
if not self.enabled:
return _EMPTY
url = f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
try:
resp = requests.get(url, timeout=self._timeout)
resp.raise_for_status()
data = resp.json()
return self._parse(data)
return self.fetch_or_raise()
except Exception as exc:
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", url, exc)
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", self._heartbeat_url(), exc)
return _EMPTY
def fetch_or_raise(self) -> HeartbeatSnapshot:
"""Like fetch(), but lets request/parse errors propagate (used by the
settings page's "Test connection" button)."""
resp = requests.get(self._heartbeat_url(), timeout=self._timeout)
resp.raise_for_status()
return self._parse(resp.json())
def _parse(self, data: dict) -> HeartbeatSnapshot:
# Expected shape:
# {"publicGroupList": [{"monitorList": [{"id": 1, "name": "..."}]}],