5 Commits
17 changed files with 1299 additions and 143 deletions

No files matched your search

+128
View File
@@ -7,6 +7,134 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
--- ---
## [0.9.3] — 2026-10-28
### Changed
- **Every time follows the `TZ` of the Inspector stack.** Before, the dashboard,
Events page and alerts were always in US Pacific time while the maintenance window
and logs used `TZ`, so anyone outside Pacific saw times hours off (e.g. a 02:00
Berlin window shown as 5:00 PM). Now the dashboard, the Events page (including
where each day starts), alert times, cooldown times and logs all use `TZ`.
- Alert times include the zone abbreviation (`10-28 09:27 AM CET`), and "Cooldown
until" in alerts, event messages and logs is a local time instead of a UTC
timestamp.
- The active time zone is shown at the bottom of every Settings page, on the Events
page and on the Maintenance page.
- **If you never set `TZ`**, times are now shown in UTC instead of Pacific. Set `TZ`
in the stack (e.g. `America/New_York`). An unrecognized name logs a warning and
falls back to UTC.
### Docs
- INSTALL: `TZ` is now marked as needed, with a "Why the time zone matters" section
(maintenance window, alerts, dashboard, logs), how to find your zone name, and a
troubleshooting row. Guide: a Time zone section and a first-login check. Compose
and README updated.
---
## [0.9.2] — 2026-10-28
Fixes from a code and security review. None of them change how containers are
monitored or recovered.
### Fixed
- **A bad value saved from Settings can no longer stop Inspector from starting.**
The forms now reject invalid times (maintenance start/end), unknown log levels and
out-of-range numbers (for example `interval_seconds: 0`, which made the loop spin).
If `config.yml` already has such a value, Inspector starts with the default and
logs a warning instead of crashing in a restart loop.
- **The dashboard password can no longer be cleared.** Saving an empty password (or
username) is refused, and an empty password never signs anyone in. Before, clearing
it left the running dashboard open to anyone.
- **Passwords with non-ASCII characters (é, ü, £ …) now work.** Before, every login
attempt with one crashed.
- The Events page no longer errors for a moment after a container is removed.
- The HTTP check hint no longer suggests `302` as an expected status; redirects are
followed, so the final page's code is what counts.
### Security
- **Sign-in throttling:** after 5 failed sign-ins (form or Basic Auth) from one address
within 5 minutes, that address is refused for the rest of the window.
- **Changing the dashboard username or password signs out existing sessions.**
Expired sessions are cleaned up.
- Request bodies are limited to 1 MB; an invalid `Content-Length` gets a 400.
- The Gotify token is sent in the `X-Gotify-Key` header instead of the URL, so it no
longer appears in Gotify or reverse-proxy access logs.
- Portainer token environment variables must be named `PORTAINER_…` (a name already in
your `config.yml` keeps working), so Test connection can't be used to send other
container secrets to an arbitrary URL.
- Single quotes are escaped in HTML, and the delete confirmations build their text safely.
- Two settings saves at the same moment can no longer overwrite each other.
- `requests` updated to 2.32.5 (CVE-2024-47081).
---
## [0.9.1] — 2026-10-28
### Fixed
- **A dependency on a disabled container no longer blocks recovery forever.** If
`depends_on` named a container that is disabled in Inspector, the dependent app was
never restarted. Inspector now only waits for dependencies it is watching, and the
container page warns when the selected dependency is disabled.
- **`PORTAINER_API_TOKEN` now reaches the container.** The public `compose.yml` didn't
pass it in, so setting it as a Portainer stack variable had no effect. Existing
stacks need the line `- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}` added under
`environment:` (Stacks → inspector → Editor).
- The "depends_on … is disabled or not found" warning was logged on every check
(since 0.8.0). It is now logged once, and again only when the container settings
change.
### Changed
- Guide: new sections **6. Apps behind a VPN container** (why depends_on matters, how to
spot a VPN-routed app, setup, what Inspector does) and **7. Portainer stack redeploy**
(why it's needed, what a redeploy does to the whole stack, requirements, where to find
every value, adding the token, the other settings, Test connection errors).
- Clearer form hints: `vpn_routed` only affects logging; `depends_on` works for any
dependency; where to find the endpoint ID.
---
## [0.9.0] — 2026-10-28
### Added
- **Every container option is now in the dashboard.** The container page has three
new groups:
- **Reachability**: HTTP check (URL, timeout, expected status) and Uptime Kuma
monitor.
- **Dependencies**: `depends_on` (pick from your containers) and `vpn_routed`.
- **Recovery**: `recovery_action` with plain-language descriptions, plus the
Portainer environment and stack name for stack redeploys.
- **Integrations settings.** New **Uptime Kuma** and **Portainer** pages under
Settings. Portainer environments can be added, edited, renamed (containers
that use them are updated) and deleted (refused while a container uses one).
The API token can be an environment variable name (recommended) or a masked
value saved in `config.yml`.
- **Test connection** buttons for Uptime Kuma (lists monitors) and each Portainer
environment (lists stacks on the endpoint), using the values in the form even
before saving.
- The All containers grid shows each container's extras (HTTP check, Uptime Kuma,
dependency, recovery action).
- Guide: new sections on reachability checks and apps behind a VPN container.
### Changed
- Uptime Kuma and Portainer changes take effect on the next check; Inspector
rebuilds those clients when their settings change instead of needing a restart.
- Saving a container form unchanged now leaves `config.yml` byte-identical. Blank
optional fields are removed instead of written as empty values (`color: ''`,
`maintenance_window: false`), and an unchanged `restart_on` list keeps its
order and the comments after it. An explicit `recovery_action: restart_container`
is kept.
---
## [0.8.0] — 2026-10-27 ## [0.8.0] — 2026-10-27
### Added ### Added
+30 -1
View File
@@ -52,10 +52,38 @@ TZ=America/Los_Angeles
|---|---|---| |---|---|---|
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes | | `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended | | `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
| `TZ` | Your [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), so times on the dashboard are local. | Recommended | | `TZ` | **Your time zone**, e.g. `America/New_York` or `Europe/Berlin`. Without it Inspector uses UTC, so the maintenance window, alerts and dashboard times are off by your UTC offset. See [Why the time zone matters](#why-the-time-zone-matters). | **Yes** (default `UTC`) |
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) | | `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) | | `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) | | `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
| `PORTAINER_API_TOKEN` | A Portainer access token. Only needed if you use **Portainer stack redeploy** for VPN-routed apps; see the in-app guide (Help → Portainer). | No |
### Why the time zone matters
Inspector uses `TZ` for every time it shows or acts on. If it's wrong, things
still work, but at the wrong hour or with confusing times:
- **The maintenance window runs in this time zone.** A window of `02:00–05:00`
with `TZ` left at UTC runs at 02:00 UTC, which is the middle of the evening in
the Americas or mid-morning in Asia. Inspector would then restart containers
during your nightly backups, and ignore real problems during the day.
- **Alerts** (Gotify / Pushover) show the time of the problem in this zone, with
its abbreviation, e.g. `10-28 03:15 AM CET`.
- **The dashboard and Events page** show times in this zone, and the Events
page's days run midnight to midnight in this zone.
- **Logs** are stamped in this zone.
**How to set it:** use your zone's name from the
[list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
("TZ identifier" column), e.g. `America/New_York`, `America/Chicago`,
`Europe/London`, `Europe/Berlin`, `Australia/Sydney`. Abbreviations like `EST` or
`CET` don't work. On Unraid, **Settings → Date and Time** shows the zone your
server uses.
**How to check it:** the bottom of every Settings page shows
`time zone: <name>`. If it says `UTC` and you're not on UTC, set `TZ` in the
stack and click **Update the stack**. If the name you set isn't recognized,
Inspector logs a warning and uses UTC.
## 3. Deploy and sign in ## 3. Deploy and sign in
@@ -115,6 +143,7 @@ starts and restarts the ones you add to it.
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. | | Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. | | Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. | | Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
| Times are a few hours off, or the maintenance window runs at the wrong time | `TZ` is missing or wrong. Check `time zone:` at the bottom of Settings, set `TZ` to your zone name (e.g. `Europe/London`), and update the stack. |
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. | | A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
## Security notes ## Security notes
+15 -10
View File
@@ -1,5 +1,5 @@
<p align="center"> <p align="center">
<img src="docs/logo.png" alt="Inspector logo" width="140"> <img src="docs/logo_outline.png" alt="Inspector logo" width="140">
</p> </p>
<h1 align="center">Inspector</h1> <h1 align="center">Inspector</h1>
@@ -35,6 +35,8 @@ restarts them when they break, and tells your phone about it.</p>
- **Stays out of the way when you're working.** Put a container in maintenance or - **Stays out of the way when you're working.** Put a container in maintenance or
mute its alerts with one click, or set a nightly maintenance window for backups mute its alerts with one click, or set a nightly maintenance window for backups
and updates. and updates.
- **Works in your time zone.** Set `TZ` in the stack and the maintenance window,
alerts, dashboard and logs all follow it.
## Install ## Install
@@ -51,12 +53,12 @@ guide (the same guide is in this repo at
[`src/inspector/assets/guide/guide.html`](./src/inspector/assets/guide/guide.html); [`src/inspector/assets/guide/guide.html`](./src/inspector/assets/guide/guide.html);
download it and open it in a browser to read it here). download it and open it in a browser to read it here).
| Add a container | Restart rules | | Add a container | Restart rules |
|---|---| | ------------------------------------------------------------------- | -------------------------------------------------------------------- |
| ![New container form](src/inspector/assets/guide/container-new.png) | ![Container settings](src/inspector/assets/guide/container-edit.png) | | ![New container form](src/inspector/assets/guide/container-new.png) | ![Container settings](src/inspector/assets/guide/container-edit.png) |
| Notifications | Events history | | Notifications | Events history |
|---|---| | ------------------------------------------------------------------------------- | --------------------------------------------------------- |
| ![Notification settings](src/inspector/assets/guide/settings-notifications.png) | ![Events timeline](src/inspector/assets/guide/events.png) | | ![Notification settings](src/inspector/assets/guide/settings-notifications.png) | ![Events timeline](src/inspector/assets/guide/events.png) |
## What it isn't ## What it isn't
@@ -69,18 +71,20 @@ download it and open it in a browser to read it here).
## Advanced options ## Advanced options
Most people never need these. They live in `config.yml` (in your Most people never need these. They are all on each container's page in the
`INSPECTOR_DATA/config` folder) rather than the dashboard; see dashboard (and in `config.yml`, see
[`config.example.yml`](./config.example.yml) for every option with comments. [`config.example.yml`](./config.example.yml) for every option with comments).
- **HTTP checks**: mark a container `UNREACHABLE` when a URL stops answering, - **HTTP checks**: mark a container `UNREACHABLE` when a URL stops answering,
even if Docker says it's healthy (useful for apps behind a VPN container). even if Docker says it's healthy (useful for apps behind a VPN container).
- **Uptime Kuma**: use a Kuma status page as an extra reachability signal. - **Uptime Kuma**: use a Kuma status page as an extra reachability signal.
**Test connection** lists the monitors Inspector can see.
- **Dependencies** (`depends_on`): don't restart an app while the container it - **Dependencies** (`depends_on`): don't restart an app while the container it
relies on (e.g. a VPN) is down. relies on (e.g. a VPN) is down.
- **Portainer stack redeploy**: for containers that share a VPN container's - **Portainer stack redeploy**: for containers that share a VPN container's
network, recover by redeploying the whole stack instead of restarting one network, recover by redeploying the whole stack instead of restarting one
container. container. Set up the Portainer connection under **Settings → Portainer**;
**Test connection** lists your stacks.
## Security ## Security
@@ -89,7 +93,8 @@ and that socket is effectively root on your server. So:
- Keep the dashboard on your home network, or put it behind a VPN or a reverse - Keep the dashboard on your home network, or put it behind a VPN or a reverse
proxy with authentication. Don't expose port 8080 to the internet. proxy with authentication. Don't expose port 8080 to the internet.
- Use a real dashboard password. - Use a real dashboard password. Repeated failed sign-ins from one address are
blocked for a few minutes.
- Inspector itself runs as an unprivileged user; it starts as root only to detect - Inspector itself runs as an unprivileged user; it starts as root only to detect
the socket's permissions and fix ownership of its own folders. the socket's permissions and fix ownership of its own folders.
+6 -1
View File
@@ -7,13 +7,17 @@
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector # Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
# INSPECTOR_PASSWORD Dashboard password (username is "admin"). # INSPECTOR_PASSWORD Dashboard password (username is "admin").
# Only used on first start; change it later in Settings. # Only used on first start; change it later in Settings.
# TZ Your timezone, e.g. America/Los_Angeles # TZ Your time zone name, e.g. America/New_York, Europe/Berlin.
# Important: the maintenance window, alerts, dashboard and
# logs all use it. Left unset, everything is in UTC.
# #
# Optional: # Optional:
# INSPECTOR_PORT Port for the dashboard on your server (default 8080). # INSPECTOR_PORT Port for the dashboard on your server (default 8080).
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin). # INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
# PUID / PGID User/group Inspector runs as (default 1000/1000). # PUID / PGID User/group Inspector runs as (default 1000/1000).
# Unraid users often prefer 99 / 100. # Unraid users often prefer 99 / 100.
# PORTAINER_API_TOKEN Portainer access token, only if you use stack redeploy
# (Settings -> Portainer in the dashboard).
# #
# No config file to create, no docker group ID to look up: Inspector creates # No config file to create, no docker group ID to look up: Inspector creates
# its config on first start and detects Docker socket permissions by itself. # its config on first start and detects Docker socket permissions by itself.
@@ -36,3 +40,4 @@ services:
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-} - INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
- PUID=${PUID:-1000} - PUID=${PUID:-1000}
- PGID=${PGID:-1000} - PGID=${PGID:-1000}
- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

+1 -1
View File
@@ -1,5 +1,5 @@
docker==7.1.0 docker==7.1.0
PyYAML==6.0.2 PyYAML==6.0.2
requests==2.32.3 requests==2.32.5
colorlog==6.9.0 colorlog==6.9.0
ruamel.yaml==0.18.6 ruamel.yaml==0.18.6
+1 -1
View File
@@ -1 +1 @@
__version__ = "0.8.0" __version__ = "0.9.3"
Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

+202 -13
View File
@@ -78,10 +78,12 @@
<a href="#status">3. The Status page</a> <a href="#status">3. The Status page</a>
<a href="#notifications">4. Notifications</a> <a href="#notifications">4. Notifications</a>
<a href="#containers">5. Managing containers</a> <a href="#containers">5. Managing containers</a>
<a href="#events">6. Events history</a> <a href="#vpn">6. Apps behind a VPN</a>
<a href="#settings">7. Other settings</a> <a href="#portainer">7. Portainer stack redeploy</a>
<a href="#restart">8. "Requires restart" fields</a> <a href="#events">8. Events history</a>
<a href="#trouble">9. Troubleshooting</a> <a href="#settings">9. Other settings</a>
<a href="#restart">10. "Requires restart" fields</a>
<a href="#trouble">11. Troubleshooting</a>
</nav> </nav>
<main> <main>
@@ -101,6 +103,8 @@ dashboard; you never need to touch a config file.</p>
<i>Inspector dashboard login</i> with the username and password.</li> <i>Inspector dashboard login</i> with the username and password.</li>
<li>Open <code>http://&lt;your-server&gt;:8080</code> and sign in.</li> <li>Open <code>http://&lt;your-server&gt;:8080</code> and sign in.</li>
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li> <li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
<li>Check the <b>time zone</b>: at the bottom of any Settings page it says
<code>time zone: …</code>. If that's not your zone, see <a href="#settings">Time zone</a>.</li>
</ol> </ol>
</section> </section>
@@ -175,7 +179,7 @@ Each card shows:</p>
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li> under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails, <li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
the reason appears next to the button.</li> the reason appears next to the button.</li>
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 8</a>), <li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 10</a>),
because turning a channel on or off is a "requires restart" field.</li> because turning a channel on or off is a "requires restart" field.</li>
</ol> </ol>
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it <p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
@@ -197,15 +201,182 @@ does <i>not</i> touch the container itself.</p>
<tr><th>Status</th><th>When it happens</th></tr> <tr><th>Status</th><th>When it happens</th></tr>
<tr><td>UNHEALTHY</td><td>The container's own health check is failing.</td></tr> <tr><td>UNHEALTHY</td><td>The container's own health check is failing.</td></tr>
<tr><td>NOT RUNNING</td><td>Stopped, exited, or crashed.</td></tr> <tr><td>NOT RUNNING</td><td>Stopped, exited, or crashed.</td></tr>
<tr><td>UNREACHABLE</td><td>Running, but an HTTP check failed (advanced, set in config.yml).</td></tr> <tr><td>UNREACHABLE</td><td>Running, but its HTTP check or Uptime Kuma monitor says it's down (see below).</td></tr>
<tr><td>ERROR</td><td>Inspector couldn't check it (Docker API error).</td></tr> <tr><td>ERROR</td><td>Inspector couldn't check it (Docker API error).</td></tr>
<tr><td>STARTING</td><td>Still in its health-check start period. Usually leave this off.</td></tr> <tr><td>STARTING</td><td>Still in its health-check start period. Usually leave this off.</td></tr>
<tr><td>HEALTHY / NO HEALTH CHECK</td><td>Normal states; almost never a reason to restart.</td></tr> <tr><td>HEALTHY / NO HEALTH CHECK</td><td>Normal states; almost never a reason to restart.</td></tr>
</table> </table>
<h3>Reachability: catching apps that look fine but aren't</h3>
<p>Docker only knows whether a container is running and whether its own health check passes. An
app can pass both and still be unusable, for example when it routes through a VPN container
whose tunnel dropped. Under <b>Reachability</b> on the container page you can add:</p>
<ul>
<li><b>HTTP check</b>: a URL Inspector requests every check (for example
<code>http://192.168.1.10:9696/ping</code>). Set <b>expected_status</b> to the code the app
returns when it's healthy; most return 200, some return 401 or 302.</li>
<li><b>Uptime Kuma monitor</b>: if you already run Uptime Kuma, name one of its monitors. Set up
the connection first in <b>Settings → Uptime Kuma</b>; <b>Test connection</b> lists the
monitor names it can see.</li>
</ul>
<p>A failed check reports <b>UNREACHABLE</b>. Tick UNREACHABLE in the restart triggers if you want
Inspector to act on it.</p>
<figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups">
<figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure>
</section>
<section id="vpn">
<h2>6. Apps behind a VPN container</h2>
<h3>Why these apps need special handling</h3>
<p>A common setup routes download and indexer apps through a VPN container. Those apps don't get a
network of their own: their compose file says <code>network_mode: container:&lt;vpn&gt;</code> (or
<code>service:&lt;vpn&gt;</code>) and all their traffic goes through the VPN container. That causes
two problems for a watchdog:</p>
<ul>
<li><b>When the VPN breaks, every app behind it breaks at the same time.</b> Restarting the apps
does nothing while the tunnel is down. Without help, Inspector would restart each app over and
over, use up their hourly restart limits, and flood you with alerts, all for a problem in a
different container.</li>
<li><b>When the VPN container is recreated</b> (an image update, or someone redeploys its stack),
it gets a new container ID. The apps still point at the old one, so a normal restart fails with
errors like <i>"No such container"</i> or <i>"joining network namespace"</i>. The only fix is
to recreate the apps too, which is what a Portainer stack redeploy does
(<a href="#portainer">section 7</a>).</li>
</ul>
<h3>How to tell if an app is behind a VPN</h3>
<ul>
<li>In Portainer open <b>Stacks → your stack → Editor</b> and look for
<code>network_mode: container:…</code> or <code>network_mode: service:…</code> on the app.</li>
<li>Or open <b>Containers → the app</b>: under <b>Network</b> it shows the VPN container instead
of a normal network, and it has no ports of its own (they're published on the VPN container).</li>
</ul>
<h3>Setting it up</h3>
<ol class="steps">
<li><b>Add the VPN container to Inspector and keep it enabled</b> (restart on UNHEALTHY and NOT
RUNNING is a good start). Inspector can only wait for containers it is watching: if the VPN
container is disabled, its apps are recovered normally, as if there were no dependency, and the
container page shows a warning.</li>
<li>On each app behind it, set <b>Dependencies → depends_on</b> to the VPN container.</li>
<li>Turn on <b>vpn_routed</b>. This only changes the log: Inspector writes down when it holds off
on the app because the VPN is down. The waiting itself comes from depends_on.</li>
<li>Add an <b>HTTP check</b> (<a href="#containers">section 5</a>) using your server's IP and the
port published on the VPN container, and tick <b>UNREACHABLE</b>. Docker often still reports
these apps as healthy when the tunnel is down; the HTTP check notices.</li>
<li>Under <b>Recovery</b>, choose <b>Restart, then redeploy stack if that fails</b> and enter the
stack name. This needs the Portainer connection from <a href="#portainer">section 7</a>.</li>
</ol>
<h3>What Inspector then does</h3>
<p>Every check, it looks at the VPN container first. While the VPN is anything other than healthy
(or starting, or running without a health check), the apps that depend on it are not restarted;
Inspector fixes the VPN first. Once the VPN is healthy again, the apps are handled normally: a
restart first, and a stack redeploy if the restart fails or doesn't help.</p>
<p class="note"><b>depends_on isn't only for VPNs.</b> Use it for any "B needs A" pair, such as an
app and its database: Inspector won't restart the app while its database is down.</p>
</section>
<section id="portainer">
<h2>7. Portainer stack redeploy</h2>
<h3>Why Inspector needs Portainer</h3>
<p>Inspector talks to Docker directly, so it can restart or start a container. It can't
<i>recreate</i> one: that needs the container's compose definition, which lives in Portainer. A
stack redeploy asks Portainer to do exactly what its <b>Update the stack</b> button does. That's
how VPN-routed apps get reattached to a recreated VPN container. You only need this if you use one
of the two "redeploy" recovery actions.</p>
<p class="note warn"><b>A redeploy affects the whole stack.</b> Portainer pulls the latest images
and recreates <i>every</i> container in that stack, not just the failing app, so all of them blink
off for a moment. That's why <b>Restart, then redeploy stack if that fails</b> is the recommended
choice: most problems are fixed by the plain restart, and the redeploy is only used when that
doesn't work.</p>
<h3>Requirements</h3>
<ul>
<li>The app must belong to a stack that was <b>created in Portainer</b>. In <b>Stacks</b>, stacks
created elsewhere (for example with <code>docker compose</code> on the command line) are marked
<b>Limited</b>, and Portainer can't redeploy them.</li>
<li>An access token for a Portainer user who can manage that stack. An administrator's token is
the simplest.</li>
</ul>
<h3>Step 1: find the values</h3>
<table>
<tr><th>Setting</th><th>Where to find it</th></tr>
<tr><td>base_url</td><td>The address you open Portainer at, without anything after the port:
usually <code>http://&lt;server-ip&gt;:9000</code> or <code>https://&lt;server-ip&gt;:9443</code>.
Use the server's IP, not <code>localhost</code>: Inspector runs in its own container.</td></tr>
<tr><td>endpoint_id</td><td>On Portainer's <b>Home</b> page, click your environment (often
<b>local</b>). The browser address then ends like <code>#!/2/docker/dashboard</code>: the
number after <code>#!/</code> is the endpoint ID (here <code>2</code>).</td></tr>
<tr><td>stack name</td><td><b>Stacks</b> list, <b>Name</b> column, spelled exactly the same.
This goes on each app's container page, not on the Portainer page.</td></tr>
<tr><td>verify_ssl</td><td>Leave off for <code>http://</code> and for Portainer's built-in
self-signed certificate on 9443. Turn on only if Portainer has a real certificate.</td></tr>
<tr><td>API token</td><td>Click your username (top right) → <b>My account</b> →
<b>Access tokens</b> → <b>Add access token</b>. Name it <code>inspector</code> and copy the
token straight away: Portainer shows it only once.</td></tr>
</table>
<h3>Step 2: give Inspector the token</h3>
<p><b>Recommended: an environment variable.</b> The token then never lands in
<code>config.yml</code> or its backups.</p>
<ol class="steps">
<li>In Portainer open <b>Stacks → inspector → Editor</b>.</li>
<li>Check that the compose file has this line under <code>environment:</code> (it's in the
compose file from version 0.9.1 on; add it if your stack is older):
<code>- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}</code></li>
<li>Under <b>Environment variables</b> click <b>Add an environment variable</b>: name
<code>PORTAINER_API_TOKEN</code>, value the token.</li>
<li>Click <b>Update the stack</b>. Inspector restarts with the token available.</li>
</ol>
<p><b>Alternative: save it in Inspector.</b> In the next step choose <b>Saved in config.yml</b>
and paste the token. It is masked in the dashboard, but anyone who can read Inspector's config
folder (or a backup of it) can read the token.</p>
<h3>Step 3: connect Inspector</h3>
<ol class="steps">
<li>Open <b>Settings → Portainer</b>. Under <b>Add environment</b>, give it a name (your own
label, e.g. <code>home</code>) and fill in base_url, endpoint_id and verify_ssl.</li>
<li><b>API token stored as</b>: <i>Environment variable</i> with the name
<code>PORTAINER_API_TOKEN</code>, or <i>Saved in config.yml</i> with the token.</li>
<li>Click <b>Test connection</b>. It should list the stacks on that endpoint; check that your
app's stack is there. Then <b>Save new environment</b>.</li>
<li>At the top of the page turn <b>enabled</b> on, pick the <b>default_environment</b>, and
<b>Save Portainer</b>.</li>
<li>On each app's container page, set the recovery action and stack name
(<a href="#vpn">section 6</a>). Leave <b>portainer_environment</b> on <i>Default</i> unless
you connected more than one Portainer.</li>
</ol>
<figure><img src="settings-portainer.png" alt="Portainer settings">
<figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure>
<h3>The other settings on the page</h3>
<table>
<tr><th>Setting</th><th>What it does</th></tr>
<tr><td>default_environment</td><td>Which connection apps use when they don't pick one. With a
single environment you can leave it on Automatic.</td></tr>
<tr><td>stack_redeploy_cooldown_seconds</td><td>Minimum time between two redeploys of the same
stack (default 300). Several apps often share a stack; this stops them triggering a redeploy
each.</td></tr>
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector waits after a redeploy for the
stack to come up before judging it again (default 90).</td></tr>
</table>
<h3>If Test connection fails</h3>
<table>
<tr><th>Message</th><th>Fix</th></tr>
<tr><td>environment variable … is not set</td><td>The compose line from step 2 is missing, or
the stack wasn't updated after adding the variable.</td></tr>
<tr><td>the API token was rejected</td><td>Token copied wrong, deleted, or from a user without
access. Create a new one.</td></tr>
<tr><td>no stacks found on endpoint N</td><td>Wrong endpoint ID; check the number in the
Portainer address again.</td></tr>
<tr><td>request failed …</td><td>Wrong address or port, <code>http</code> vs <code>https</code>
mixed up, or <code>localhost</code> used instead of the server IP. With https and a
self-signed certificate, turn verify_ssl off.</td></tr>
</table>
</section> </section>
<section id="events"> <section id="events">
<h2>6. Events history</h2> <h2>8. Events history</h2>
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and <p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p> maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
<figure><img src="events.png" alt="Events timeline"> <figure><img src="events.png" alt="Events timeline">
@@ -213,7 +384,7 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
</section> </section>
<section id="settings"> <section id="settings">
<h2>7. Other settings</h2> <h2>9. Other settings</h2>
<figure><img src="settings-inspector.png" alt="Inspector loop settings"> <figure><img src="settings-inspector.png" alt="Inspector loop settings">
<figcaption>Settings → Inspector loop.</figcaption></figure> <figcaption>Settings → Inspector loop.</figcaption></figure>
<table> <table>
@@ -223,15 +394,33 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr> <tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
<tr><td>Event history</td><td>How many days of events to keep.</td></tr> <tr><td>Event history</td><td>How many days of events to keep.</td></tr>
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr> <tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr> <tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00, in your <a href="#settings">time zone</a>) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr> <tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#portainer">section 7</a>.</td></tr>
</table> </table>
<p>At the bottom of every settings page is a strip showing where the config file lives and <p>At the bottom of every settings page is a strip showing where the config file lives,
whether Inspector can write to it.</p> whether Inspector can write to it, and which time zone it is using.</p>
<h3>Time zone</h3>
<p>Inspector uses the <code>TZ</code> setting of its Portainer stack for every time it shows or acts on:</p>
<ul>
<li><b>The maintenance window runs in this zone.</b> A 02:00–05:00 window with the wrong zone
runs hours off, so Inspector could restart containers during your backups and ignore
problems at other times.</li>
<li>Alert times (with the zone, e.g. <code>03:15 AM CET</code>), dashboard times, the Events
page's days (midnight to midnight), and logs.</li>
</ul>
<p>To change it: in Portainer open <b>Stacks → inspector → Editor</b>, set the <code>TZ</code>
environment variable to your zone's name, e.g. <code>America/New_York</code>,
<code>Europe/London</code> or <code>Australia/Sydney</code> (names from the
<a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">list of time zones</a>;
abbreviations like <code>EST</code> don't work), then click <b>Update the stack</b>. Without
<code>TZ</code>, Inspector uses UTC.</p>
</section> </section>
<section id="restart"> <section id="restart">
<h2>8. "Requires restart" fields</h2> <h2>10. "Requires restart" fields</h2>
<p>Most changes apply on the next check. Fields marked <p>Most changes apply on the next check. Fields marked
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the <span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
dashboard on or off) only take effect after Inspector restarts:</p> dashboard on or off) only take effect after Inspector restarts:</p>
@@ -243,7 +432,7 @@ dashboard on or off) only take effect after Inspector restarts:</p>
</section> </section>
<section id="trouble"> <section id="trouble">
<h2>9. Troubleshooting</h2> <h2>11. Troubleshooting</h2>
<table> <table>
<tr><th>Problem</th><th>Fix</th></tr> <tr><th>Problem</th><th>Fix</th></tr>
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td> <tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
Binary file not shown.

After

Width:  |  Height:  |  Size: 42 KiB

+34 -4
View File
@@ -1,10 +1,12 @@
from __future__ import annotations from __future__ import annotations
import io import io
import logging
import os import os
import shutil import shutil
import threading import threading
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import time
from pathlib import Path from pathlib import Path
import yaml import yaml
@@ -23,6 +25,30 @@ VALID_STATUSES = {
"UNREACHABLE", "UNREACHABLE",
} }
log = logging.getLogger("inspector.config")
# Names logging.setLevel() accepts; anything else would crash at startup.
VALID_LOG_LEVELS = {"CRITICAL", "FATAL", "ERROR", "WARN", "WARNING", "INFO", "DEBUG", "NOTSET"}
def valid_hhmm(value: str) -> bool:
"""True if `value` parses the way the maintenance tracker parses it ("HH:MM")."""
try:
hour, minute = str(value).split(":")
time(int(hour), int(minute))
return True
except (TypeError, ValueError):
return False
def _startup_safe(value, default, ok: bool, key: str):
"""Return `value`, or `default` with a warning when it would crash Inspector."""
if ok:
return value
log.warning("config: invalid %s %r; using %r instead", key, value, default)
return default
VALID_RECOVERY_ACTIONS = { VALID_RECOVERY_ACTIONS = {
"restart_container", "restart_container",
"start_container", "start_container",
@@ -190,9 +216,11 @@ def build_config_from_raw(raw: dict) -> AppConfig:
to validate an edited raw dict before writing it back to disk. to validate an edited raw dict before writing it back to disk.
""" """
insp_raw = raw.get("inspector", {}) or {} insp_raw = raw.get("inspector", {}) or {}
interval = int(insp_raw.get("interval_seconds", 60))
log_level = str(insp_raw.get("log_level", "INFO")).upper()
inspector = InspectorConfig( inspector = InspectorConfig(
interval_seconds=int(insp_raw.get("interval_seconds", 60)), interval_seconds=_startup_safe(interval, 1, interval >= 1, "inspector.interval_seconds"),
log_level=str(insp_raw.get("log_level", "INFO")).upper(), log_level=_startup_safe(log_level, "INFO", log_level in VALID_LOG_LEVELS, "inspector.log_level"),
dry_run=bool(insp_raw.get("dry_run", False)), dry_run=bool(insp_raw.get("dry_run", False)),
log_dir=str(insp_raw.get("log_dir", "/app/logs")), log_dir=str(insp_raw.get("log_dir", "/app/logs")),
log_filename=str(insp_raw.get("log_filename", "inspector.log")), log_filename=str(insp_raw.get("log_filename", "inspector.log")),
@@ -237,10 +265,12 @@ def build_config_from_raw(raw: dict) -> AppConfig:
) )
maint_raw = raw.get("maintenance") or {} maint_raw = raw.get("maintenance") or {}
maint_start = str(maint_raw.get("start", "00:00"))
maint_end = str(maint_raw.get("end", "06:00"))
maintenance = MaintenanceConfig( maintenance = MaintenanceConfig(
enabled=bool(maint_raw.get("enabled", False)), enabled=bool(maint_raw.get("enabled", False)),
start=str(maint_raw.get("start", "00:00")), start=_startup_safe(maint_start, "00:00", valid_hhmm(maint_start), "maintenance.start"),
end=str(maint_raw.get("end", "06:00")), end=_startup_safe(maint_end, "06:00", valid_hhmm(maint_end), "maintenance.end"),
dir=str(maint_raw.get("dir", "/app/data/maintenance")), dir=str(maint_raw.get("dir", "/app/data/maintenance")),
max_hours=int(maint_raw.get("max_hours", 8)), max_hours=int(maint_raw.get("max_hours", 8)),
) )
File diff suppressed because it is too large. Load diff
+45 -6
View File
@@ -1,14 +1,49 @@
from __future__ import annotations from __future__ import annotations
from datetime import datetime, timezone import logging
from zoneinfo import ZoneInfo import os
from datetime import datetime, timezone, tzinfo
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
PACIFIC = ZoneInfo("America/Los_Angeles") log = logging.getLogger("inspector.formatting")
def format_timestamp(value: datetime | str | None) -> str: def _resolve_local_tz(env: dict | None = None) -> tuple[tzinfo, str]:
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'. """The zone every displayed time uses, plus its name.
Follows the same rules as the C library (which drives log timestamps and
the maintenance window), so all times Inspector shows agree:
TZ set to a zone name -> that zone; TZ unset -> the container's
/etc/localtime; anything else -> UTC.
"""
env = os.environ if env is None else env
name = (env.get("TZ") or "").strip().lstrip(":")
if name:
try:
return ZoneInfo(name), name
except (ZoneInfoNotFoundError, ValueError):
log.warning(
"TZ=%r is not a known time zone name (e.g. Europe/Berlin); showing times in UTC",
name,
)
return timezone.utc, "UTC"
try:
with open("/etc/localtime", "rb") as fh:
zone = ZoneInfo.from_file(fh, key="localtime")
label = datetime.now(zone).tzname() or "localtime"
return zone, label
except (OSError, ValueError):
return timezone.utc, "UTC"
# Resolved once at import; the container's TZ doesn't change while it runs.
LOCAL_TZ, LOCAL_TZ_NAME = _resolve_local_tz()
def format_timestamp(value: datetime | str | None, *, with_zone: bool = False) -> str:
"""Render a timestamp in the configured local zone, e.g. '06-14 03:45 PM'
(with_zone adds the abbreviation: '06-14 03:45 PM CET').
Accepts a datetime, an ISO 8601 string (as stored by EventStore / Accepts a datetime, an ISO 8601 string (as stored by EventStore /
ContainerState), or None/"" (returns ""). Naive datetimes/strings are ContainerState), or None/"" (returns ""). Naive datetimes/strings are
@@ -24,4 +59,8 @@ def format_timestamp(value: datetime | str | None) -> str:
return value return value
if value.tzinfo is None: if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc) value = value.replace(tzinfo=timezone.utc)
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p") local = value.astimezone(LOCAL_TZ)
text = local.strftime("%m-%d %I:%M %p")
if with_zone:
text += f" {local.tzname()}"
return text
+20 -7
View File
@@ -170,7 +170,7 @@ def _check_one(
message=( message=(
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n" f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
f"- Errors before recovery: **{state.error_count}**\n" f"- Errors before recovery: **{state.error_count}**\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=3, priority=3,
) )
@@ -196,7 +196,7 @@ def _check_one(
f"**{name}** is missing from Docker.\n\n" f"**{name}** is missing from Docker.\n\n"
+ (f"- Detail: {result.message}\n" if result.message else "") + (f"- Detail: {result.message}\n" if result.message else "")
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n" + f"- Last known status: *{state.previous_status or 'unknown'}*\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=7, priority=7,
) )
@@ -240,7 +240,7 @@ def _check_one(
+ ".\n\n" + ".\n\n"
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n" f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
f"- Previous status: *{state.previous_status or 'unknown'}*\n" f"- Previous status: *{state.previous_status or 'unknown'}*\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=5, priority=5,
) )
@@ -276,7 +276,7 @@ def _check_one(
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n" f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
f"- Current status: **{result.status}**\n" f"- Current status: **{result.status}**\n"
f"- Restarts paused until the hourly window clears\n" f"- Restarts paused until the hourly window clears\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=9, priority=9,
) )
@@ -293,8 +293,10 @@ def _check_one(
if ok: if ok:
dep = container_config.depends_on dep = container_config.depends_on
if dep: # Only wait on a dependency Inspector is watching. A disabled one has
dep_status = states[dep].current_status if dep in states else None # no state, and waiting on it would block recovery forever.
if dep and dep in states:
dep_status = states[dep].current_status
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK): if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
if container_config.vpn_routed: if container_config.vpn_routed:
log.info( log.info(
@@ -417,6 +419,7 @@ def run_loop(config_ref: ConfigRef) -> int:
check_order = _topological_order(enabled_configs) check_order = _topological_order(enabled_configs)
portainer_client = PortainerClient(app.integrations.portainer) portainer_client = PortainerClient(app.integrations.portainer)
integrations_cfg = app.integrations
stack_cooldowns: dict[str, datetime] = {} stack_cooldowns: dict[str, datetime] = {}
dashboard = DashboardServer( dashboard = DashboardServer(
@@ -453,8 +456,18 @@ def run_loop(config_ref: ConfigRef) -> int:
for gone in states.keys() - current_enabled.keys(): for gone in states.keys() - current_enabled.keys():
states.pop(gone, None) states.pop(gone, None)
log.info("Stopped watching container (removed or disabled): %s", gone) log.info("Stopped watching container (removed or disabled): %s", gone)
if current_enabled != enabled_configs:
# Only re-order (and re-log ordering warnings) when the config changed.
check_order = _topological_order(current_enabled)
enabled_configs = current_enabled enabled_configs = current_enabled
check_order = _topological_order(enabled_configs)
# Integration clients hold their config; rebuild them when the
# settings page changes Uptime Kuma / Portainer.
if app.integrations != integrations_cfg:
integrations_cfg = app.integrations
uptime_kuma_client = UptimeKumaClient(integrations_cfg.uptime_kuma)
portainer_client = PortainerClient(integrations_cfg.portainer)
log.info("Integration settings changed; reloaded Uptime Kuma / Portainer clients")
uptime_kuma_snapshot = uptime_kuma_client.fetch() uptime_kuma_snapshot = uptime_kuma_client.fetch()
for name in check_order: for name in check_order:
+2 -1
View File
@@ -33,9 +33,10 @@ class GotifyNotifier:
}, },
} }
try: try:
# Header, not ?token=, so the token stays out of access logs.
resp = requests.post( resp = requests.post(
url, url,
params={"token": self._config.token}, headers={"X-Gotify-Key": self._config.token},
json=payload, json=payload,
timeout=self._timeout, timeout=self._timeout,
) )
+15 -15
View File
@@ -228,7 +228,7 @@ def _perform_restart_then_redeploy(
f"**{result.status}** check(s).\n\n" f"**{result.status}** check(s).\n\n"
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n" f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n" f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=6, priority=6,
) )
@@ -242,7 +242,7 @@ def _perform_restart_then_redeploy(
state.restart_limit_notified = False state.restart_limit_notified = False
log.info( log.info(
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)", "%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
width, name, state.cooldown_until.isoformat(), width, name, format_timestamp(state.cooldown_until, with_zone=True),
) )
eventstore.record( eventstore.record(
container=name, container=name,
@@ -250,15 +250,15 @@ def _perform_restart_then_redeploy(
old_status=state.previous_status, old_status=state.previous_status,
new_status=result.status, new_status=result.status,
error_count=state.error_count, error_count=state.error_count,
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}", message=f"restart succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
) )
notifier.notify( notifier.notify(
title=f"✅ Inspector: {name} restart completed", title=f"✅ Inspector: {name} restart completed",
message=( message=(
f"**{name}** was successfully restarted by Docker.\n\n" f"**{name}** was successfully restarted by Docker.\n\n"
f"- Previous status: *{result.status}*\n" f"- Previous status: *{result.status}*\n"
f"- Cooldown until: {state.cooldown_until.isoformat()}\n" f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=3, priority=3,
) )
@@ -321,7 +321,7 @@ def _perform_docker_action(
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive " f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
f"**{result.status}** check(s).\n\n" f"**{result.status}** check(s).\n\n"
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n" f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=6, priority=6,
) )
@@ -335,22 +335,22 @@ def _perform_docker_action(
state.record_restart(now, container_config.cooldown_seconds) state.record_restart(now, container_config.cooldown_seconds)
state.last_notification_state = "RESTARTED" state.last_notification_state = "RESTARTED"
state.restart_limit_notified = False state.restart_limit_notified = False
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat()) log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, format_timestamp(state.cooldown_until, with_zone=True))
eventstore.record( eventstore.record(
container=name, container=name,
event_type=EVENT_RESTART, event_type=EVENT_RESTART,
old_status=state.previous_status, old_status=state.previous_status,
new_status=result.status, new_status=result.status,
error_count=state.error_count, error_count=state.error_count,
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}", message=f"{verb} succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
) )
notifier.notify( notifier.notify(
title=f"✅ Inspector: {name} {verb} completed", title=f"✅ Inspector: {name} {verb} completed",
message=( message=(
f"**{name}** was successfully {verb}ed by Docker.\n\n" f"**{name}** was successfully {verb}ed by Docker.\n\n"
f"- Previous status: *{result.status}*\n" f"- Previous status: *{result.status}*\n"
f"- Cooldown until: {state.cooldown_until.isoformat()}\n" f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=3, priority=3,
) )
@@ -362,7 +362,7 @@ def _perform_docker_action(
f"{verb.capitalize()} of **{name}** failed.\n\n" f"{verb.capitalize()} of **{name}** failed.\n\n"
f"- Status: **{result.status}**\n" f"- Status: **{result.status}**\n"
f"- Error count: **{state.error_count}**\n" f"- Error count: **{state.error_count}**\n"
f"- Time: {format_timestamp(now)}\n\n" f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
f"```\n{err}\n```" f"```\n{err}\n```"
), ),
priority=9, priority=9,
@@ -469,7 +469,7 @@ def _perform_portainer_redeploy(
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) " f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
f"to recover **{name}**.\n\n" f"to recover **{name}**.\n\n"
f"- Status: **{result.status}**\n" f"- Status: **{result.status}**\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=6, priority=6,
) )
@@ -483,7 +483,7 @@ def _perform_portainer_redeploy(
message=( message=(
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n" f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
f"- Triggered by: **{name}** ({result.status})\n" f"- Triggered by: **{name}** ({result.status})\n"
f"- Time: {format_timestamp(now)}\n\n" f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
f"```\n{redeploy_result.message}\n```" f"```\n{redeploy_result.message}\n```"
), ),
priority=9, priority=9,
@@ -516,7 +516,7 @@ def _perform_portainer_redeploy(
f"successfully redeployed to recover **{name}**.\n\n" f"successfully redeployed to recover **{name}**.\n\n"
f"- Triggered by: *{result.status}*\n" f"- Triggered by: *{result.status}*\n"
f"- Post-redeploy wait: {wait_seconds}s\n" f"- Post-redeploy wait: {wait_seconds}s\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=4, priority=4,
) )
@@ -536,7 +536,7 @@ def _perform_portainer_redeploy(
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck " f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
f"for **{name}** still reports a failure.\n\n" f"for **{name}** still reports a failure.\n\n"
f"- Recheck result: *{recheck.message}*\n" f"- Recheck result: *{recheck.message}*\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=6, priority=6,
) )
+12 -6
View File
@@ -41,20 +41,26 @@ class UptimeKumaClient:
and bool(self._config.status_page_slug) and bool(self._config.status_page_slug)
) )
def _heartbeat_url(self) -> str:
return f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
def fetch(self) -> HeartbeatSnapshot: def fetch(self) -> HeartbeatSnapshot:
if not self.enabled: if not self.enabled:
return _EMPTY return _EMPTY
url = f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
try: try:
resp = requests.get(url, timeout=self._timeout) return self.fetch_or_raise()
resp.raise_for_status()
data = resp.json()
return self._parse(data)
except Exception as exc: except Exception as exc:
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", url, exc) log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", self._heartbeat_url(), exc)
return _EMPTY return _EMPTY
def fetch_or_raise(self) -> HeartbeatSnapshot:
"""Like fetch(), but lets request/parse errors propagate (used by the
settings page's "Test connection" button)."""
resp = requests.get(self._heartbeat_url(), timeout=self._timeout)
resp.raise_for_status()
return self._parse(resp.json())
def _parse(self, data: dict) -> HeartbeatSnapshot: def _parse(self, data: dict) -> HeartbeatSnapshot:
# Expected shape: # Expected shape:
# {"publicGroupList": [{"monitorList": [{"id": 1, "name": "..."}]}], # {"publicGroupList": [{"monitorList": [{"id": 1, "name": "..."}]}],