5 Commits
15 changed files with 754 additions and 164 deletions

No files matched your search

+128
View File
@@ -7,6 +7,134 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
--- ---
## [0.9.5] — 2026-10-28
### Fixed
- **A stack redeploy no longer freezes Inspector.** After a successful Portainer
redeploy, Inspector used to stop everything for `post_redeploy_wait_seconds`
(default 90 s): no other container was checked, dashboard times stood still, and
stopping Inspector during the wait wasn't clean. Now the redeployed app is left to
settle while every other container keeps being checked, and its one-time HTTP
re-check (with the "recheck failed" alert) happens on its first check after the wait.
- The redeployed app's cooldown now starts after the settle time. Before, the wait
ate into it (with the defaults, 300 s cooldown became 210 s).
---
## [0.9.4] — 2026-10-28
### Fixed
- **Containers without a Docker health check now "recover" properly.** Many images
don't define a health check, so Docker reports them as `NO HEALTH CHECK` even when
they work. Inspector only treated `HEALTHY` as back to normal, so for these
containers:
- the error count never reset, and one-off failures days apart added up to a restart;
- no "recovered" alert was sent;
- with *Restart, then redeploy*, the next unrelated problem skipped the restart and
redeployed the whole stack.
Now `NO HEALTH CHECK` counts as working (unless you made it a restart trigger): the
count resets, the "recovered" alert is sent ("back to running (no health check)"),
and the next problem starts with a plain restart. Containers with a health check
behave exactly as before.
---
## [0.9.3] — 2026-10-28
### Changed
- **Every time follows the `TZ` of the Inspector stack.** Before, the dashboard,
Events page and alerts were always in US Pacific time while the maintenance window
and logs used `TZ`, so anyone outside Pacific saw times hours off (e.g. a 02:00
Berlin window shown as 5:00 PM). Now the dashboard, the Events page (including
where each day starts), alert times, cooldown times and logs all use `TZ`.
- Alert times include the zone abbreviation (`10-28 09:27 AM CET`), and "Cooldown
until" in alerts, event messages and logs is a local time instead of a UTC
timestamp.
- The active time zone is shown at the bottom of every Settings page, on the Events
page and on the Maintenance page.
- **If you never set `TZ`**, times are now shown in UTC instead of Pacific. Set `TZ`
in the stack (e.g. `America/New_York`). An unrecognized name logs a warning and
falls back to UTC.
### Docs
- INSTALL: `TZ` is now marked as needed, with a "Why the time zone matters" section
(maintenance window, alerts, dashboard, logs), how to find your zone name, and a
troubleshooting row. Guide: a Time zone section and a first-login check. Compose
and README updated.
---
## [0.9.2] — 2026-10-28
Fixes from a code and security review. None of them change how containers are
monitored or recovered.
### Fixed
- **A bad value saved from Settings can no longer stop Inspector from starting.**
The forms now reject invalid times (maintenance start/end), unknown log levels and
out-of-range numbers (for example `interval_seconds: 0`, which made the loop spin).
If `config.yml` already has such a value, Inspector starts with the default and
logs a warning instead of crashing in a restart loop.
- **The dashboard password can no longer be cleared.** Saving an empty password (or
username) is refused, and an empty password never signs anyone in. Before, clearing
it left the running dashboard open to anyone.
- **Passwords with non-ASCII characters (é, ü, £ …) now work.** Before, every login
attempt with one crashed.
- The Events page no longer errors for a moment after a container is removed.
- The HTTP check hint no longer suggests `302` as an expected status; redirects are
followed, so the final page's code is what counts.
### Security
- **Sign-in throttling:** after 5 failed sign-ins (form or Basic Auth) from one address
within 5 minutes, that address is refused for the rest of the window.
- **Changing the dashboard username or password signs out existing sessions.**
Expired sessions are cleaned up.
- Request bodies are limited to 1 MB; an invalid `Content-Length` gets a 400.
- The Gotify token is sent in the `X-Gotify-Key` header instead of the URL, so it no
longer appears in Gotify or reverse-proxy access logs.
- Portainer token environment variables must be named `PORTAINER_…` (a name already in
your `config.yml` keeps working), so Test connection can't be used to send other
container secrets to an arbitrary URL.
- Single quotes are escaped in HTML, and the delete confirmations build their text safely.
- Two settings saves at the same moment can no longer overwrite each other.
- `requests` updated to 2.32.5 (CVE-2024-47081).
---
## [0.9.1] — 2026-10-28
### Fixed
- **A dependency on a disabled container no longer blocks recovery forever.** If
`depends_on` named a container that is disabled in Inspector, the dependent app was
never restarted. Inspector now only waits for dependencies it is watching, and the
container page warns when the selected dependency is disabled.
- **`PORTAINER_API_TOKEN` now reaches the container.** The public `compose.yml` didn't
pass it in, so setting it as a Portainer stack variable had no effect. Existing
stacks need the line `- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}` added under
`environment:` (Stacks → inspector → Editor).
- The "depends_on … is disabled or not found" warning was logged on every check
(since 0.8.0). It is now logged once, and again only when the container settings
change.
### Changed
- Guide: new sections **6. Apps behind a VPN container** (why depends_on matters, how to
spot a VPN-routed app, setup, what Inspector does) and **7. Portainer stack redeploy**
(why it's needed, what a redeploy does to the whole stack, requirements, where to find
every value, adding the token, the other settings, Test connection errors).
- Clearer form hints: `vpn_routed` only affects logging; `depends_on` works for any
dependency; where to find the endpoint ID.
---
## [0.9.0] — 2026-10-28 ## [0.9.0] — 2026-10-28
### Added ### Added
+30 -1
View File
@@ -52,10 +52,38 @@ TZ=America/Los_Angeles
|---|---|---| |---|---|---|
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes | | `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended | | `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
| `TZ` | Your [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), so times on the dashboard are local. | Recommended | | `TZ` | **Your time zone**, e.g. `America/New_York` or `Europe/Berlin`. Without it Inspector uses UTC, so the maintenance window, alerts and dashboard times are off by your UTC offset. See [Why the time zone matters](#why-the-time-zone-matters). | **Yes** (default `UTC`) |
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) | | `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) | | `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) | | `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
| `PORTAINER_API_TOKEN` | A Portainer access token. Only needed if you use **Portainer stack redeploy** for VPN-routed apps; see the in-app guide (Help → Portainer). | No |
### Why the time zone matters
Inspector uses `TZ` for every time it shows or acts on. If it's wrong, things
still work, but at the wrong hour or with confusing times:
- **The maintenance window runs in this time zone.** A window of `02:00–05:00`
with `TZ` left at UTC runs at 02:00 UTC, which is the middle of the evening in
the Americas or mid-morning in Asia. Inspector would then restart containers
during your nightly backups, and ignore real problems during the day.
- **Alerts** (Gotify / Pushover) show the time of the problem in this zone, with
its abbreviation, e.g. `10-28 03:15 AM CET`.
- **The dashboard and Events page** show times in this zone, and the Events
page's days run midnight to midnight in this zone.
- **Logs** are stamped in this zone.
**How to set it:** use your zone's name from the
[list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
("TZ identifier" column), e.g. `America/New_York`, `America/Chicago`,
`Europe/London`, `Europe/Berlin`, `Australia/Sydney`. Abbreviations like `EST` or
`CET` don't work. On Unraid, **Settings → Date and Time** shows the zone your
server uses.
**How to check it:** the bottom of every Settings page shows
`time zone: <name>`. If it says `UTC` and you're not on UTC, set `TZ` in the
stack and click **Update the stack**. If the name you set isn't recognized,
Inspector logs a warning and uses UTC.
## 3. Deploy and sign in ## 3. Deploy and sign in
@@ -115,6 +143,7 @@ starts and restarts the ones you add to it.
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. | | Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. | | Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. | | Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
| Times are a few hours off, or the maintenance window runs at the wrong time | `TZ` is missing or wrong. Check `time zone:` at the bottom of Settings, set `TZ` to your zone name (e.g. `Europe/London`), and update the stack. |
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. | | A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
## Security notes ## Security notes
+4 -1
View File
@@ -35,6 +35,8 @@ restarts them when they break, and tells your phone about it.</p>
- **Stays out of the way when you're working.** Put a container in maintenance or - **Stays out of the way when you're working.** Put a container in maintenance or
mute its alerts with one click, or set a nightly maintenance window for backups mute its alerts with one click, or set a nightly maintenance window for backups
and updates. and updates.
- **Works in your time zone.** Set `TZ` in the stack and the maintenance window,
alerts, dashboard and logs all follow it.
## Install ## Install
@@ -91,7 +93,8 @@ and that socket is effectively root on your server. So:
- Keep the dashboard on your home network, or put it behind a VPN or a reverse - Keep the dashboard on your home network, or put it behind a VPN or a reverse
proxy with authentication. Don't expose port 8080 to the internet. proxy with authentication. Don't expose port 8080 to the internet.
- Use a real dashboard password. - Use a real dashboard password. Repeated failed sign-ins from one address are
blocked for a few minutes.
- Inspector itself runs as an unprivileged user; it starts as root only to detect - Inspector itself runs as an unprivileged user; it starts as root only to detect
the socket's permissions and fix ownership of its own folders. the socket's permissions and fix ownership of its own folders.
+6 -1
View File
@@ -7,13 +7,17 @@
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector # Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
# INSPECTOR_PASSWORD Dashboard password (username is "admin"). # INSPECTOR_PASSWORD Dashboard password (username is "admin").
# Only used on first start; change it later in Settings. # Only used on first start; change it later in Settings.
# TZ Your timezone, e.g. America/Los_Angeles # TZ Your time zone name, e.g. America/New_York, Europe/Berlin.
# Important: the maintenance window, alerts, dashboard and
# logs all use it. Left unset, everything is in UTC.
# #
# Optional: # Optional:
# INSPECTOR_PORT Port for the dashboard on your server (default 8080). # INSPECTOR_PORT Port for the dashboard on your server (default 8080).
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin). # INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
# PUID / PGID User/group Inspector runs as (default 1000/1000). # PUID / PGID User/group Inspector runs as (default 1000/1000).
# Unraid users often prefer 99 / 100. # Unraid users often prefer 99 / 100.
# PORTAINER_API_TOKEN Portainer access token, only if you use stack redeploy
# (Settings -> Portainer in the dashboard).
# #
# No config file to create, no docker group ID to look up: Inspector creates # No config file to create, no docker group ID to look up: Inspector creates
# its config on first start and detects Docker socket permissions by itself. # its config on first start and detects Docker socket permissions by itself.
@@ -36,3 +40,4 @@ services:
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-} - INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
- PUID=${PUID:-1000} - PUID=${PUID:-1000}
- PGID=${PGID:-1000} - PGID=${PGID:-1000}
- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}
+1 -1
View File
@@ -1,5 +1,5 @@
docker==7.1.0 docker==7.1.0
PyYAML==6.0.2 PyYAML==6.0.2
requests==2.32.3 requests==2.32.5
colorlog==6.9.0 colorlog==6.9.0
ruamel.yaml==0.18.6 ruamel.yaml==0.18.6
+1 -1
View File
@@ -1 +1 @@
__version__ = "0.9.0" __version__ = "0.9.5"
+177 -29
View File
@@ -78,10 +78,12 @@
<a href="#status">3. The Status page</a> <a href="#status">3. The Status page</a>
<a href="#notifications">4. Notifications</a> <a href="#notifications">4. Notifications</a>
<a href="#containers">5. Managing containers</a> <a href="#containers">5. Managing containers</a>
<a href="#events">6. Events history</a> <a href="#vpn">6. Apps behind a VPN</a>
<a href="#settings">7. Other settings</a> <a href="#portainer">7. Portainer stack redeploy</a>
<a href="#restart">8. "Requires restart" fields</a> <a href="#events">8. Events history</a>
<a href="#trouble">9. Troubleshooting</a> <a href="#settings">9. Other settings</a>
<a href="#restart">10. "Requires restart" fields</a>
<a href="#trouble">11. Troubleshooting</a>
</nav> </nav>
<main> <main>
@@ -101,6 +103,8 @@ dashboard; you never need to touch a config file.</p>
<i>Inspector dashboard login</i> with the username and password.</li> <i>Inspector dashboard login</i> with the username and password.</li>
<li>Open <code>http://&lt;your-server&gt;:8080</code> and sign in.</li> <li>Open <code>http://&lt;your-server&gt;:8080</code> and sign in.</li>
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li> <li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
<li>Check the <b>time zone</b>: at the bottom of any Settings page it says
<code>time zone: …</code>. If that's not your zone, see <a href="#settings">Time zone</a>.</li>
</ol> </ol>
</section> </section>
@@ -175,7 +179,7 @@ Each card shows:</p>
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li> under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails, <li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
the reason appears next to the button.</li> the reason appears next to the button.</li>
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 8</a>), <li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 10</a>),
because turning a channel on or off is a "requires restart" field.</li> because turning a channel on or off is a "requires restart" field.</li>
</ol> </ol>
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it <p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
@@ -220,32 +224,160 @@ Inspector to act on it.</p>
<figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups"> <figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups">
<figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure> <figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure>
<h3>Apps behind a VPN container</h3> </section>
<p>If an app shares a VPN container's network (<code>network_mode: container:vpn</code> or similar), set these on the app's page:</p>
<section id="vpn">
<h2>6. Apps behind a VPN container</h2>
<h3>Why these apps need special handling</h3>
<p>A common setup routes download and indexer apps through a VPN container. Those apps don't get a
network of their own: their compose file says <code>network_mode: container:&lt;vpn&gt;</code> (or
<code>service:&lt;vpn&gt;</code>) and all their traffic goes through the VPN container. That causes
two problems for a watchdog:</p>
<ul>
<li><b>When the VPN breaks, every app behind it breaks at the same time.</b> Restarting the apps
does nothing while the tunnel is down. Without help, Inspector would restart each app over and
over, use up their hourly restart limits, and flood you with alerts, all for a problem in a
different container.</li>
<li><b>When the VPN container is recreated</b> (an image update, or someone redeploys its stack),
it gets a new container ID. The apps still point at the old one, so a normal restart fails with
errors like <i>"No such container"</i> or <i>"joining network namespace"</i>. The only fix is
to recreate the apps too, which is what a Portainer stack redeploy does
(<a href="#portainer">section 7</a>).</li>
</ul>
<h3>How to tell if an app is behind a VPN</h3>
<ul>
<li>In Portainer open <b>Stacks → your stack → Editor</b> and look for
<code>network_mode: container:…</code> or <code>network_mode: service:…</code> on the app.</li>
<li>Or open <b>Containers → the app</b>: under <b>Network</b> it shows the VPN container instead
of a normal network, and it has no ports of its own (they're published on the VPN container).</li>
</ul>
<h3>Setting it up</h3>
<ol class="steps"> <ol class="steps">
<li><b>Dependencies → depends_on</b>: pick the VPN container, and turn on <b>vpn_routed</b>. <li><b>Add the VPN container to Inspector and keep it enabled</b> (restart on UNHEALTHY and NOT
Inspector checks the VPN first and waits for it to recover before touching the app.</li> RUNNING is a good start). Inspector can only wait for containers it is watching: if the VPN
<li><b>Recovery → recovery_action</b>: choose <b>Restart, then redeploy stack if that fails</b> container is disabled, its apps are recovered normally, as if there were no dependency, and the
and enter the <b>stack name</b> exactly as Portainer shows it. When the VPN container gets container page shows a warning.</li>
recreated, a plain restart of the app fails; redeploying the stack reconnects it.</li> <li>On each app behind it, set <b>Dependencies → depends_on</b> to the VPN container.</li>
<li>Turn on <b>vpn_routed</b>. This only changes the log: Inspector writes down when it holds off
on the app because the VPN is down. The waiting itself comes from depends_on.</li>
<li>Add an <b>HTTP check</b> (<a href="#containers">section 5</a>) using your server's IP and the
port published on the VPN container, and tick <b>UNREACHABLE</b>. Docker often still reports
these apps as healthy when the tunnel is down; the HTTP check notices.</li>
<li>Under <b>Recovery</b>, choose <b>Restart, then redeploy stack if that fails</b> and enter the
stack name. This needs the Portainer connection from <a href="#portainer">section 7</a>.</li>
</ol> </ol>
<p>Stack redeploys need the Portainer connection in <b>Settings → Portainer</b>:</p> <h3>What Inspector then does</h3>
<p>Every check, it looks at the VPN container first. While the VPN is anything other than healthy
(or starting, or running without a health check), the apps that depend on it are not restarted;
Inspector fixes the VPN first. Once the VPN is healthy again, the apps are handled normally: a
restart first, and a stack redeploy if the restart fails or doesn't help.</p>
<p class="note"><b>depends_on isn't only for VPNs.</b> Use it for any "B needs A" pair, such as an
app and its database: Inspector won't restart the app while its database is down.</p>
</section>
<section id="portainer">
<h2>7. Portainer stack redeploy</h2>
<h3>Why Inspector needs Portainer</h3>
<p>Inspector talks to Docker directly, so it can restart or start a container. It can't
<i>recreate</i> one: that needs the container's compose definition, which lives in Portainer. A
stack redeploy asks Portainer to do exactly what its <b>Update the stack</b> button does. That's
how VPN-routed apps get reattached to a recreated VPN container. You only need this if you use one
of the two "redeploy" recovery actions.</p>
<p class="note warn"><b>A redeploy affects the whole stack.</b> Portainer pulls the latest images
and recreates <i>every</i> container in that stack, not just the failing app, so all of them blink
off for a moment. That's why <b>Restart, then redeploy stack if that fails</b> is the recommended
choice: most problems are fixed by the plain restart, and the redeploy is only used when that
doesn't work.</p>
<h3>Requirements</h3>
<ul>
<li>The app must belong to a stack that was <b>created in Portainer</b>. In <b>Stacks</b>, stacks
created elsewhere (for example with <code>docker compose</code> on the command line) are marked
<b>Limited</b>, and Portainer can't redeploy them.</li>
<li>An access token for a Portainer user who can manage that stack. An administrator's token is
the simplest.</li>
</ul>
<h3>Step 1: find the values</h3>
<table>
<tr><th>Setting</th><th>Where to find it</th></tr>
<tr><td>base_url</td><td>The address you open Portainer at, without anything after the port:
usually <code>http://&lt;server-ip&gt;:9000</code> or <code>https://&lt;server-ip&gt;:9443</code>.
Use the server's IP, not <code>localhost</code>: Inspector runs in its own container.</td></tr>
<tr><td>endpoint_id</td><td>On Portainer's <b>Home</b> page, click your environment (often
<b>local</b>). The browser address then ends like <code>#!/2/docker/dashboard</code>: the
number after <code>#!/</code> is the endpoint ID (here <code>2</code>).</td></tr>
<tr><td>stack name</td><td><b>Stacks</b> list, <b>Name</b> column, spelled exactly the same.
This goes on each app's container page, not on the Portainer page.</td></tr>
<tr><td>verify_ssl</td><td>Leave off for <code>http://</code> and for Portainer's built-in
self-signed certificate on 9443. Turn on only if Portainer has a real certificate.</td></tr>
<tr><td>API token</td><td>Click your username (top right) → <b>My account</b> →
<b>Access tokens</b> → <b>Add access token</b>. Name it <code>inspector</code> and copy the
token straight away: Portainer shows it only once.</td></tr>
</table>
<h3>Step 2: give Inspector the token</h3>
<p><b>Recommended: an environment variable.</b> The token then never lands in
<code>config.yml</code> or its backups.</p>
<ol class="steps"> <ol class="steps">
<li>In Portainer, open <b>My account → Access tokens</b> and create a token.</li> <li>In Portainer open <b>Stacks → inspector → Editor</b>.</li>
<li>Easiest and safest: add it to the Inspector stack as an environment variable (for example <li>Check that the compose file has this line under <code>environment:</code> (it's in the
<code>PORTAINER_API_TOKEN</code>) and redeploy Inspector. Or choose <b>Saved in config.yml</b> compose file from version 0.9.1 on; add it if your stack is older):
and paste it in.</li> <code>- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}</code></li>
<li>In <b>Settings → Portainer</b>, add an environment with Portainer's URL and endpoint ID <li>Under <b>Environment variables</b> click <b>Add an environment variable</b>: name
(usually 1 or 2; it's the number in the URL when you open the environment in Portainer). <code>PORTAINER_API_TOKEN</code>, value the token.</li>
Click <b>Test connection</b>: it should list your stacks.</li> <li>Click <b>Update the stack</b>. Inspector restarts with the token available.</li>
<li>Turn Portainer on and save.</li> </ol>
<p><b>Alternative: save it in Inspector.</b> In the next step choose <b>Saved in config.yml</b>
and paste the token. It is masked in the dashboard, but anyone who can read Inspector's config
folder (or a backup of it) can read the token.</p>
<h3>Step 3: connect Inspector</h3>
<ol class="steps">
<li>Open <b>Settings → Portainer</b>. Under <b>Add environment</b>, give it a name (your own
label, e.g. <code>home</code>) and fill in base_url, endpoint_id and verify_ssl.</li>
<li><b>API token stored as</b>: <i>Environment variable</i> with the name
<code>PORTAINER_API_TOKEN</code>, or <i>Saved in config.yml</i> with the token.</li>
<li>Click <b>Test connection</b>. It should list the stacks on that endpoint; check that your
app's stack is there. Then <b>Save new environment</b>.</li>
<li>At the top of the page turn <b>enabled</b> on, pick the <b>default_environment</b>, and
<b>Save Portainer</b>.</li>
<li>On each app's container page, set the recovery action and stack name
(<a href="#vpn">section 6</a>). Leave <b>portainer_environment</b> on <i>Default</i> unless
you connected more than one Portainer.</li>
</ol> </ol>
<figure><img src="settings-portainer.png" alt="Portainer settings"> <figure><img src="settings-portainer.png" alt="Portainer settings">
<figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure> <figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure>
<h3>The other settings on the page</h3>
<table>
<tr><th>Setting</th><th>What it does</th></tr>
<tr><td>default_environment</td><td>Which connection apps use when they don't pick one. With a
single environment you can leave it on Automatic.</td></tr>
<tr><td>stack_redeploy_cooldown_seconds</td><td>Minimum time between two redeploys of the same
stack (default 300). Several apps often share a stack; this stops them triggering a redeploy
each.</td></tr>
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector gives the stack to come up after
a redeploy before judging that app again (default 90). Everything else keeps being checked
meanwhile.</td></tr>
</table>
<h3>If Test connection fails</h3>
<table>
<tr><th>Message</th><th>Fix</th></tr>
<tr><td>environment variable … is not set</td><td>The compose line from step 2 is missing, or
the stack wasn't updated after adding the variable.</td></tr>
<tr><td>the API token was rejected</td><td>Token copied wrong, deleted, or from a user without
access. Create a new one.</td></tr>
<tr><td>no stacks found on endpoint N</td><td>Wrong endpoint ID; check the number in the
Portainer address again.</td></tr>
<tr><td>request failed …</td><td>Wrong address or port, <code>http</code> vs <code>https</code>
mixed up, or <code>localhost</code> used instead of the server IP. With https and a
self-signed certificate, turn verify_ssl off.</td></tr>
</table>
</section> </section>
<section id="events"> <section id="events">
<h2>6. Events history</h2> <h2>8. Events history</h2>
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and <p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p> maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
<figure><img src="events.png" alt="Events timeline"> <figure><img src="events.png" alt="Events timeline">
@@ -253,7 +385,7 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
</section> </section>
<section id="settings"> <section id="settings">
<h2>7. Other settings</h2> <h2>9. Other settings</h2>
<figure><img src="settings-inspector.png" alt="Inspector loop settings"> <figure><img src="settings-inspector.png" alt="Inspector loop settings">
<figcaption>Settings → Inspector loop.</figcaption></figure> <figcaption>Settings → Inspector loop.</figcaption></figure>
<table> <table>
@@ -263,17 +395,33 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr> <tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
<tr><td>Event history</td><td>How many days of events to keep.</td></tr> <tr><td>Event history</td><td>How many days of events to keep.</td></tr>
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr> <tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr> <tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00, in your <a href="#settings">time zone</a>) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr> <tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr> <tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#containers">Apps behind a VPN container</a>.</td></tr> <tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#portainer">section 7</a>.</td></tr>
</table> </table>
<p>At the bottom of every settings page is a strip showing where the config file lives and <p>At the bottom of every settings page is a strip showing where the config file lives,
whether Inspector can write to it.</p> whether Inspector can write to it, and which time zone it is using.</p>
<h3>Time zone</h3>
<p>Inspector uses the <code>TZ</code> setting of its Portainer stack for every time it shows or acts on:</p>
<ul>
<li><b>The maintenance window runs in this zone.</b> A 02:00–05:00 window with the wrong zone
runs hours off, so Inspector could restart containers during your backups and ignore
problems at other times.</li>
<li>Alert times (with the zone, e.g. <code>03:15 AM CET</code>), dashboard times, the Events
page's days (midnight to midnight), and logs.</li>
</ul>
<p>To change it: in Portainer open <b>Stacks → inspector → Editor</b>, set the <code>TZ</code>
environment variable to your zone's name, e.g. <code>America/New_York</code>,
<code>Europe/London</code> or <code>Australia/Sydney</code> (names from the
<a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">list of time zones</a>;
abbreviations like <code>EST</code> don't work), then click <b>Update the stack</b>. Without
<code>TZ</code>, Inspector uses UTC.</p>
</section> </section>
<section id="restart"> <section id="restart">
<h2>8. "Requires restart" fields</h2> <h2>10. "Requires restart" fields</h2>
<p>Most changes apply on the next check. Fields marked <p>Most changes apply on the next check. Fields marked
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the <span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
dashboard on or off) only take effect after Inspector restarts:</p> dashboard on or off) only take effect after Inspector restarts:</p>
@@ -285,7 +433,7 @@ dashboard on or off) only take effect after Inspector restarts:</p>
</section> </section>
<section id="trouble"> <section id="trouble">
<h2>9. Troubleshooting</h2> <h2>11. Troubleshooting</h2>
<table> <table>
<tr><th>Problem</th><th>Fix</th></tr> <tr><th>Problem</th><th>Fix</th></tr>
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td> <tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
Binary file not shown.

Before

Width:  |  Height:  |  Size: 41 KiB

After

Width:  |  Height:  |  Size: 42 KiB

+34 -4
View File
@@ -1,10 +1,12 @@
from __future__ import annotations from __future__ import annotations
import io import io
import logging
import os import os
import shutil import shutil
import threading import threading
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import time
from pathlib import Path from pathlib import Path
import yaml import yaml
@@ -23,6 +25,30 @@ VALID_STATUSES = {
"UNREACHABLE", "UNREACHABLE",
} }
log = logging.getLogger("inspector.config")
# Names logging.setLevel() accepts; anything else would crash at startup.
VALID_LOG_LEVELS = {"CRITICAL", "FATAL", "ERROR", "WARN", "WARNING", "INFO", "DEBUG", "NOTSET"}
def valid_hhmm(value: str) -> bool:
"""True if `value` parses the way the maintenance tracker parses it ("HH:MM")."""
try:
hour, minute = str(value).split(":")
time(int(hour), int(minute))
return True
except (TypeError, ValueError):
return False
def _startup_safe(value, default, ok: bool, key: str):
"""Return `value`, or `default` with a warning when it would crash Inspector."""
if ok:
return value
log.warning("config: invalid %s %r; using %r instead", key, value, default)
return default
VALID_RECOVERY_ACTIONS = { VALID_RECOVERY_ACTIONS = {
"restart_container", "restart_container",
"start_container", "start_container",
@@ -190,9 +216,11 @@ def build_config_from_raw(raw: dict) -> AppConfig:
to validate an edited raw dict before writing it back to disk. to validate an edited raw dict before writing it back to disk.
""" """
insp_raw = raw.get("inspector", {}) or {} insp_raw = raw.get("inspector", {}) or {}
interval = int(insp_raw.get("interval_seconds", 60))
log_level = str(insp_raw.get("log_level", "INFO")).upper()
inspector = InspectorConfig( inspector = InspectorConfig(
interval_seconds=int(insp_raw.get("interval_seconds", 60)), interval_seconds=_startup_safe(interval, 1, interval >= 1, "inspector.interval_seconds"),
log_level=str(insp_raw.get("log_level", "INFO")).upper(), log_level=_startup_safe(log_level, "INFO", log_level in VALID_LOG_LEVELS, "inspector.log_level"),
dry_run=bool(insp_raw.get("dry_run", False)), dry_run=bool(insp_raw.get("dry_run", False)),
log_dir=str(insp_raw.get("log_dir", "/app/logs")), log_dir=str(insp_raw.get("log_dir", "/app/logs")),
log_filename=str(insp_raw.get("log_filename", "inspector.log")), log_filename=str(insp_raw.get("log_filename", "inspector.log")),
@@ -237,10 +265,12 @@ def build_config_from_raw(raw: dict) -> AppConfig:
) )
maint_raw = raw.get("maintenance") or {} maint_raw = raw.get("maintenance") or {}
maint_start = str(maint_raw.get("start", "00:00"))
maint_end = str(maint_raw.get("end", "06:00"))
maintenance = MaintenanceConfig( maintenance = MaintenanceConfig(
enabled=bool(maint_raw.get("enabled", False)), enabled=bool(maint_raw.get("enabled", False)),
start=str(maint_raw.get("start", "00:00")), start=_startup_safe(maint_start, "00:00", valid_hhmm(maint_start), "maintenance.start"),
end=str(maint_raw.get("end", "06:00")), end=_startup_safe(maint_end, "06:00", valid_hhmm(maint_end), "maintenance.end"),
dir=str(maint_raw.get("dir", "/app/data/maintenance")), dir=str(maint_raw.get("dir", "/app/data/maintenance")),
max_hours=int(maint_raw.get("max_hours", 8)), max_hours=int(maint_raw.get("max_hours", 8)),
) )
+224 -69
View File
@@ -1,5 +1,7 @@
from __future__ import annotations from __future__ import annotations
import functools
import hashlib
import json import json
import logging import logging
import os import os
@@ -21,12 +23,15 @@ from .config import (
AppConfig, AppConfig,
ConfigRef, ConfigRef,
ContainerConfig, ContainerConfig,
VALID_LOG_LEVELS,
HttpCheckConfig, HttpCheckConfig,
UptimeKumaMonitorConfig, UptimeKumaMonitorConfig,
build_config_from_raw, build_config_from_raw,
valid_hhmm,
write_config_raw, write_config_raw,
) )
from .formatting import PACIFIC, format_timestamp from . import formatting
from .formatting import format_timestamp
from .health import CONTAINER_NOT_FOUND, ERROR, HEALTHY, NOT_RUNNING, UNHEALTHY, UNREACHABLE from .health import CONTAINER_NOT_FOUND, ERROR, HEALTHY, NOT_RUNNING, UNHEALTHY, UNREACHABLE
from .maintenance import MaintenanceTracker from .maintenance import MaintenanceTracker
from .notification_mute import NotificationMuteTracker from .notification_mute import NotificationMuteTracker
@@ -36,6 +41,9 @@ from .state import ContainerState, utcnow
log = logging.getLogger("inspector.dashboard") log = logging.getLogger("inspector.dashboard")
_SESSION_MAX_AGE = 86400 # seconds a login cookie stays valid _SESSION_MAX_AGE = 86400 # seconds a login cookie stays valid
_MAX_BODY_BYTES = 1024 * 1024 # largest request body accepted
_LOGIN_MAX_FAILURES = 5 # failed logins per client before it is throttled
_LOGIN_FAILURE_WINDOW = 300 # seconds a failed login counts toward the limit
_UNCHANGED_SECRET_PLACEHOLDER = "__UNCHANGED__" # UI sentinel: keep existing secret value _UNCHANGED_SECRET_PLACEHOLDER = "__UNCHANGED__" # UI sentinel: keep existing secret value
# Config keys that require an Inspector container restart to take effect. # Config keys that require an Inspector container restart to take effect.
@@ -84,7 +92,13 @@ def _color_map(
states: dict[str, ContainerState], states: dict[str, ContainerState],
containers: dict[str, ContainerConfig], containers: dict[str, ContainerConfig],
) -> dict[str, str]: ) -> dict[str, str]:
return {name: _resolve_color(containers[name], i) for i, name in enumerate(states)} # list() snapshot + skip: the inspector loop adds/removes states from another
# thread, and a container removed in the UI keeps its state until the next check.
return {
name: _resolve_color(containers[name], i)
for i, name in enumerate(list(states))
if name in containers
}
def _compute_phase( def _compute_phase(
@@ -167,20 +181,20 @@ def _status_snapshot(
def _parse_date_param(value: str | None) -> date: def _parse_date_param(value: str | None) -> date:
"""Parse a ?date=YYYY-MM-DD query param; falls back to "today" in """Parse a ?date=YYYY-MM-DD query param; falls back to "today" in the
America/Los_Angeles on missing/invalid input.""" configured time zone on missing/invalid input."""
if value: if value:
try: try:
return date.fromisoformat(value) return date.fromisoformat(value)
except ValueError: except ValueError:
pass pass
return datetime.now(PACIFIC).date() return datetime.now(formatting.LOCAL_TZ).date()
def _day_bounds_utc(day: date) -> tuple[datetime, datetime]: def _day_bounds_utc(day: date) -> tuple[datetime, datetime]:
"""UTC [start, end) bounds for a Pacific calendar day, extended one hour """UTC [start, end) bounds for a local calendar day, extended one hour
into the previous day for the lookback-overlap.""" into the previous day for the lookback-overlap."""
midnight = datetime.combine(day, datetime.min.time(), tzinfo=PACIFIC) midnight = datetime.combine(day, datetime.min.time(), tzinfo=formatting.LOCAL_TZ)
start = (midnight - timedelta(hours=1)).astimezone(timezone.utc) start = (midnight - timedelta(hours=1)).astimezone(timezone.utc)
end = (midnight + timedelta(days=1)).astimezone(timezone.utc) end = (midnight + timedelta(days=1)).astimezone(timezone.utc)
return start, end return start, end
@@ -205,8 +219,8 @@ def _read_events_range(db_path: str, start: datetime, end: datetime) -> list[dic
def _hour_markers(day: date) -> list[datetime]: def _hour_markers(day: date) -> list[datetime]:
"""24 Pacific-local hour boundaries for `day`, 00:00..23:00 ascending.""" """24 local hour boundaries for `day`, 00:00..23:00 ascending."""
midnight = datetime.combine(day, datetime.min.time(), tzinfo=PACIFIC) midnight = datetime.combine(day, datetime.min.time(), tzinfo=formatting.LOCAL_TZ)
return [midnight + timedelta(hours=h) for h in range(24)] return [midnight + timedelta(hours=h) for h in range(24)]
@@ -226,7 +240,7 @@ def _group_events_by_hour(
for ev in events: for ev in events:
try: try:
ts = datetime.fromisoformat(ev["ts"]).astimezone(PACIFIC) ts = datetime.fromisoformat(ev["ts"]).astimezone(formatting.LOCAL_TZ)
except (ValueError, KeyError): except (ValueError, KeyError):
continue continue
if prev_hour_start <= ts < markers[0]: if prev_hour_start <= ts < markers[0]:
@@ -246,9 +260,31 @@ def _escape(value: object) -> str:
.replace("<", "&lt;") .replace("<", "&lt;")
.replace(">", "&gt;") .replace(">", "&gt;")
.replace('"', "&quot;") .replace('"', "&quot;")
.replace("'", "&#x27;")
) )
def _js_string_attr(text: str) -> str:
"""A JS string literal safe to put inside a double-quoted HTML attribute."""
return _escape(json.dumps(text))
def _creds_match(username: str, password: str, cfg) -> bool:
"""Constant-time credential check. Never accepts an empty configured
password, and works for non-ASCII input (compare_digest on str doesn't)."""
if not cfg.password:
return False
user_ok = secrets.compare_digest(username.encode("utf-8"), cfg.username.encode("utf-8"))
pass_ok = secrets.compare_digest(password.encode("utf-8"), cfg.password.encode("utf-8"))
return user_ok and pass_ok
def _cred_fingerprint(cfg) -> str:
"""Changes whenever the dashboard username or password changes, which
invalidates every session created with the old credentials."""
return hashlib.sha256(f"{cfg.username}\0{cfg.password}".encode("utf-8")).hexdigest()
_STATUS_BADGE_CLASS = { _STATUS_BADGE_CLASS = {
HEALTHY: "healthy", HEALTHY: "healthy",
UNHEALTHY: "unhealthy", UNHEALTHY: "unhealthy",
@@ -1076,12 +1112,15 @@ def _render_events_page(
prev_day = (selected_day - timedelta(days=1)).isoformat() prev_day = (selected_day - timedelta(days=1)).isoformat()
next_day = (selected_day + timedelta(days=1)).isoformat() next_day = (selected_day + timedelta(days=1)).isoformat()
today = datetime.now(PACIFIC).date() today = datetime.now(formatting.LOCAL_TZ).date()
day_label = selected_day.strftime("%A, %B %-d, %Y") day_label = selected_day.strftime("%A, %B %-d, %Y")
parts.append('<div class="events-nav">\n') parts.append('<div class="events-nav">\n')
parts.append(f'<a class="day-nav" href="/events?date={prev_day}">&larr; Prev day</a>\n') parts.append(f'<a class="day-nav" href="/events?date={prev_day}">&larr; Prev day</a>\n')
parts.append(f'<span class="day-label">{_escape(day_label)}</span>\n') parts.append(
f'<span class="day-label">{_escape(day_label)}'
f' <span style="color:var(--muted); font-weight:400; font-size:0.8em;">· times in {_escape(formatting.LOCAL_TZ_NAME)}</span></span>\n'
)
if selected_day >= today: if selected_day >= today:
parts.append('<span class="day-nav disabled">Next day &rarr;</span>\n') parts.append('<span class="day-nav disabled">Next day &rarr;</span>\n')
else: else:
@@ -1146,10 +1185,13 @@ def _render_event_item(ev: dict, container_colors: dict[str, str] | None) -> str
) )
def _render_login_page(error: bool = False) -> bytes: def _render_login_page(error: bool = False, throttled: bool = False) -> bytes:
error_html = ( if throttled:
'<div class="login-error">Incorrect username or password.</div>\n' if error else "" error_html = '<div class="login-error">Too many failed sign-ins. Try again in a few minutes.</div>\n'
) elif error:
error_html = '<div class="login-error">Incorrect username or password.</div>\n'
else:
error_html = ""
head = ( head = (
"<!DOCTYPE html>\n" "<!DOCTYPE html>\n"
'<html lang="en">\n<head>\n' '<html lang="en">\n<head>\n'
@@ -1243,6 +1285,14 @@ def _form_float(form: dict, key: str, default: float = 0.0) -> float:
return default return default
def _int_in(form: dict, key: str, default: int, lo: int, hi: int) -> int:
"""Read an int form field and reject values outside lo..hi."""
value = _form_int(form, key, default)
if not lo <= value <= hi:
raise _SettingsError(f"{key} must be between {lo} and {hi} (got {value})")
return value
def _set_or_drop(section: dict, key: str, value) -> None: def _set_or_drop(section: dict, key: str, value) -> None:
"""Store value, or remove the key when it is blank/False so config.yml """Store value, or remove the key when it is blank/False so config.yml
doesn't fill up with empty defaults.""" doesn't fill up with empty defaults."""
@@ -1270,8 +1320,10 @@ def _apply_container_advanced(raw: dict, name: str, cfg: dict, form: dict) -> No
hc["enabled"] = hc_enabled hc["enabled"] = hc_enabled
hc["url"] = hc_url hc["url"] = hc_url
timeout = _form_float(form, "http_check_timeout_seconds", 5.0) timeout = _form_float(form, "http_check_timeout_seconds", 5.0)
if not 0.5 <= timeout <= 120:
raise _SettingsError(f"HTTP check timeout must be between 0.5 and 120 seconds (got {timeout:g})")
hc["timeout_seconds"] = int(timeout) if timeout.is_integer() else timeout hc["timeout_seconds"] = int(timeout) if timeout.is_integer() else timeout
hc["expected_status"] = _form_int(form, "http_check_expected_status", 200) hc["expected_status"] = _int_in(form, "http_check_expected_status", 200, 100, 599)
cfg["http_check"] = hc cfg["http_check"] = hc
else: else:
cfg.pop("http_check", None) cfg.pop("http_check", None)
@@ -1314,6 +1366,15 @@ def _apply_container_advanced(raw: dict, name: str, cfg: dict, form: dict) -> No
cfg.pop("portainer_stack_name", None) cfg.pop("portainer_stack_name", None)
_PORTAINER_ENV_NAME_RE = re.compile(r"^PORTAINER_[A-Z0-9_]+$")
def _token_env_allowed(name: str, saved: str = "") -> bool:
"""Only PORTAINER_* variables (or the name already in config.yml) may be read
as a token, so the dashboard can't be used to send other container secrets out."""
return bool(_PORTAINER_ENV_NAME_RE.match(name)) or (bool(saved) and name == saved)
def _apply_portainer_env(raw: dict, form: dict) -> None: def _apply_portainer_env(raw: dict, form: dict) -> None:
"""Add, edit, rename or delete one integrations.portainer.environments entry.""" """Add, edit, rename or delete one integrations.portainer.environments entry."""
pt = raw.setdefault("integrations", {}).setdefault("portainer", {}) pt = raw.setdefault("integrations", {}).setdefault("portainer", {})
@@ -1350,7 +1411,7 @@ def _apply_portainer_env(raw: dict, form: dict) -> None:
env = envs.get(original) or {} env = envs.get(original) or {}
env["base_url"] = base_url env["base_url"] = base_url
env["endpoint_id"] = _form_int(form, "endpoint_id", 1) env["endpoint_id"] = _int_in(form, "endpoint_id", 1, 1, 1_000_000)
env["verify_ssl"] = _form_bool(form, "verify_ssl") env["verify_ssl"] = _form_bool(form, "verify_ssl")
if _form_str(form, "token_source", "env") == "inline": if _form_str(form, "token_source", "env") == "inline":
_apply_secret(form, env, "api_token", "api_token") _apply_secret(form, env, "api_token", "api_token")
@@ -1361,6 +1422,8 @@ def _apply_portainer_env(raw: dict, form: dict) -> None:
token_env = _form_str(form, "api_token_env", "").strip() token_env = _form_str(form, "api_token_env", "").strip()
if not token_env: if not token_env:
raise _SettingsError("enter the name of the environment variable holding the API token") raise _SettingsError("enter the name of the environment variable holding the API token")
if not _token_env_allowed(token_env, env.get("api_token_env", "")):
raise _SettingsError("the environment variable name must start with PORTAINER_ (e.g. PORTAINER_API_TOKEN)")
env["api_token_env"] = token_env env["api_token_env"] = token_env
env.pop("api_token", None) env.pop("api_token", None)
@@ -1376,57 +1439,81 @@ def _apply_portainer_env(raw: dict, form: dict) -> None:
pt["environments"] = envs pt["environments"] = envs
_SAVE_LOCK = threading.Lock()
def _serialized(fn):
"""Run config read-modify-write functions one at a time, so two saves at
the same moment can't overwrite each other's changes."""
@functools.wraps(fn)
def wrapper(*args, **kwargs):
with _SAVE_LOCK:
return fn(*args, **kwargs)
return wrapper
@_serialized
def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> None: def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> None:
"""Validate the form, merge into the raw config, write atomically, swap ref.""" """Validate the form, merge into the raw config, write atomically, swap ref."""
raw = config_ref.get_raw() raw = config_ref.get_raw()
if section == "inspector": if section == "inspector":
insp = raw.setdefault("inspector", {}) insp = raw.setdefault("inspector", {})
insp["interval_seconds"] = _form_int(form, "interval_seconds", 60) insp["interval_seconds"] = _int_in(form, "interval_seconds", 60, 1, 86400)
insp["log_level"] = _form_str(form, "log_level", "INFO").upper() level = _form_str(form, "log_level", "INFO").strip().upper()
if level not in VALID_LOG_LEVELS:
raise _SettingsError("log_level must be one of DEBUG, INFO, WARNING, ERROR, CRITICAL")
insp["log_level"] = level
insp["dry_run"] = _form_bool(form, "dry_run") insp["dry_run"] = _form_bool(form, "dry_run")
insp["heartbeat"] = _form_bool(form, "heartbeat") insp["heartbeat"] = _form_bool(form, "heartbeat")
insp["log_retention_days"] = _form_int(form, "log_retention_days", 30) insp["log_retention_days"] = _int_in(form, "log_retention_days", 30, 1, 3650)
elif section == "gotify": elif section == "gotify":
notif = raw.setdefault("notifications", {}) notif = raw.setdefault("notifications", {})
gotify = notif.setdefault("gotify", {}) gotify = notif.setdefault("gotify", {})
gotify["enabled"] = _form_bool(form, "enabled") gotify["enabled"] = _form_bool(form, "enabled")
gotify["url"] = _form_str(form, "url", "") gotify["url"] = _form_str(form, "url", "")
_apply_secret(form, gotify, "token", "token") _apply_secret(form, gotify, "token", "token")
gotify["priority"] = _form_int(form, "priority", 5) gotify["priority"] = _int_in(form, "priority", 5, 0, 10)
elif section == "pushover": elif section == "pushover":
notif = raw.setdefault("notifications", {}) notif = raw.setdefault("notifications", {})
pushover = notif.setdefault("pushover", {}) pushover = notif.setdefault("pushover", {})
pushover["enabled"] = _form_bool(form, "enabled") pushover["enabled"] = _form_bool(form, "enabled")
_apply_secret(form, pushover, "token", "token") _apply_secret(form, pushover, "token", "token")
_apply_secret(form, pushover, "user_key", "user_key") _apply_secret(form, pushover, "user_key", "user_key")
pushover["priority"] = _form_int(form, "priority", 5) pushover["priority"] = _int_in(form, "priority", 5, 0, 10)
pushover["device"] = _form_str(form, "device", "") pushover["device"] = _form_str(form, "device", "")
pushover["sound"] = _form_str(form, "sound", "") pushover["sound"] = _form_str(form, "sound", "")
elif section == "dashboard": elif section == "dashboard":
dash = raw.setdefault("dashboard", {}) dash = raw.setdefault("dashboard", {})
dash["enabled"] = _form_bool(form, "enabled") dash["enabled"] = _form_bool(form, "enabled")
dash["username"] = _form_str(form, "username", "admin") dash["username"] = _form_str(form, "username", "admin").strip()
if not dash["username"]:
raise _SettingsError("the dashboard username can't be empty")
_apply_secret(form, dash, "password", "password") _apply_secret(form, dash, "password", "password")
if not dash.get("password"):
raise _SettingsError("the dashboard password can't be empty")
elif section == "database": elif section == "database":
db = raw.setdefault("database", {}) db = raw.setdefault("database", {})
db["enabled"] = _form_bool(form, "enabled") db["enabled"] = _form_bool(form, "enabled")
db["retention_days"] = _form_int(form, "retention_days", 30) db["retention_days"] = _int_in(form, "retention_days", 30, 1, 3650)
elif section == "health_summary": elif section == "health_summary":
hs = raw.setdefault("health_summary", {}) hs = raw.setdefault("health_summary", {})
hs["enabled"] = _form_bool(form, "enabled") hs["enabled"] = _form_bool(form, "enabled")
hs["interval_hours"] = _form_int(form, "interval_hours", 24) hs["interval_hours"] = _int_in(form, "interval_hours", 24, 1, 8760)
hs["priority"] = _form_int(form, "priority", 3) hs["priority"] = _int_in(form, "priority", 3, 0, 10)
elif section == "maintenance": elif section == "maintenance":
m = raw.setdefault("maintenance", {}) m = raw.setdefault("maintenance", {})
m["enabled"] = _form_bool(form, "enabled") m["enabled"] = _form_bool(form, "enabled")
m["start"] = _form_str(form, "start", "00:00") for key, default in (("start", "00:00"), ("end", "06:00")):
m["end"] = _form_str(form, "end", "06:00") value = _form_str(form, key, default).strip()
m["max_hours"] = _form_int(form, "max_hours", 8) if not valid_hhmm(value):
raise _SettingsError(f"{key} must be a 24-hour time like 02:30 (got {value!r})")
m[key] = value
m["max_hours"] = _int_in(form, "max_hours", 8, 1, 720)
elif section == "notification_mute": elif section == "notification_mute":
nm = raw.setdefault("notification_mute", {}) nm = raw.setdefault("notification_mute", {})
nm["enabled"] = _form_bool(form, "enabled") nm["enabled"] = _form_bool(form, "enabled")
nm["default_hours"] = _form_int(form, "default_hours", 12) nm["default_hours"] = _int_in(form, "default_hours", 12, 1, 720)
elif section.startswith("container:"): elif section.startswith("container:"):
name = section.split(":", 1)[1] name = section.split(":", 1)[1]
containers = raw.get("containers") or {} containers = raw.get("containers") or {}
@@ -1434,10 +1521,10 @@ def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> N
raise _SettingsError(f"unknown container {name!r}") raise _SettingsError(f"unknown container {name!r}")
cfg = containers[name] or {} cfg = containers[name] or {}
cfg["enabled"] = _form_bool(form, "enabled") cfg["enabled"] = _form_bool(form, "enabled")
cfg["max_error_count"] = _form_int(form, "max_error_count", 5) cfg["max_error_count"] = _int_in(form, "max_error_count", 5, 1, 1000)
cfg["cooldown_seconds"] = _form_int(form, "cooldown_seconds", 300) cfg["cooldown_seconds"] = _int_in(form, "cooldown_seconds", 300, 0, 86400)
cfg["grace_period_seconds"] = _form_int(form, "grace_period_seconds", 120) cfg["grace_period_seconds"] = _int_in(form, "grace_period_seconds", 120, 0, 86400)
cfg["max_restarts_per_hour"] = _form_int(form, "max_restarts_per_hour", 3) cfg["max_restarts_per_hour"] = _int_in(form, "max_restarts_per_hour", 3, 1, 100)
_set_or_drop(cfg, "color", _form_str(form, "color", "").strip()) _set_or_drop(cfg, "color", _form_str(form, "color", "").strip())
if form.get("restart_on_present"): # only our own form carries this field if form.get("restart_on_present"): # only our own form carries this field
_set_or_drop(cfg, "maintenance_window", _form_bool(form, "maintenance_window")) _set_or_drop(cfg, "maintenance_window", _form_bool(form, "maintenance_window"))
@@ -1467,8 +1554,8 @@ def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> N
if default_env and default_env not in (pt.get("environments") or {}): if default_env and default_env not in (pt.get("environments") or {}):
raise _SettingsError(f"unknown Portainer environment {default_env!r}") raise _SettingsError(f"unknown Portainer environment {default_env!r}")
_set_or_drop(pt, "default_environment", default_env) _set_or_drop(pt, "default_environment", default_env)
pt["stack_redeploy_cooldown_seconds"] = _form_int(form, "stack_redeploy_cooldown_seconds", 300) pt["stack_redeploy_cooldown_seconds"] = _int_in(form, "stack_redeploy_cooldown_seconds", 300, 0, 86400)
pt["post_redeploy_wait_seconds"] = _form_int(form, "post_redeploy_wait_seconds", 90) pt["post_redeploy_wait_seconds"] = _int_in(form, "post_redeploy_wait_seconds", 90, 0, 3600)
elif section == "portainer_env": elif section == "portainer_env":
_apply_portainer_env(raw, form) _apply_portainer_env(raw, form)
else: else:
@@ -1492,6 +1579,7 @@ def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> N
_CONTAINER_NAME_RE = re.compile(r"^[a-zA-Z0-9][a-zA-Z0-9_-]{0,62}$") _CONTAINER_NAME_RE = re.compile(r"^[a-zA-Z0-9][a-zA-Z0-9_-]{0,62}$")
@_serialized
def _apply_container_create(config_ref: ConfigRef, form: dict) -> str: def _apply_container_create(config_ref: ConfigRef, form: dict) -> str:
"""Insert a new container under raw['containers']. Returns the new name.""" """Insert a new container under raw['containers']. Returns the new name."""
name = _form_str(form, "name", "").strip() name = _form_str(form, "name", "").strip()
@@ -1507,10 +1595,10 @@ def _apply_container_create(config_ref: ConfigRef, form: dict) -> str:
cfg = { cfg = {
"enabled": _form_bool(form, "enabled"), "enabled": _form_bool(form, "enabled"),
"max_error_count": _form_int(form, "max_error_count", 5), "max_error_count": _int_in(form, "max_error_count", 5, 1, 1000),
"cooldown_seconds": _form_int(form, "cooldown_seconds", 300), "cooldown_seconds": _int_in(form, "cooldown_seconds", 300, 0, 86400),
"grace_period_seconds": _form_int(form, "grace_period_seconds", 120), "grace_period_seconds": _int_in(form, "grace_period_seconds", 120, 0, 86400),
"max_restarts_per_hour": _form_int(form, "max_restarts_per_hour", 3), "max_restarts_per_hour": _int_in(form, "max_restarts_per_hour", 3, 1, 100),
"color": _form_str(form, "color", ""), "color": _form_str(form, "color", ""),
"maintenance_window": _form_bool(form, "maintenance_window"), "maintenance_window": _form_bool(form, "maintenance_window"),
} }
@@ -1536,6 +1624,7 @@ def _apply_container_create(config_ref: ConfigRef, form: dict) -> str:
return name return name
@_serialized
def _apply_container_delete(config_ref: ConfigRef, name: str) -> None: def _apply_container_delete(config_ref: ConfigRef, name: str) -> None:
raw = config_ref.get_raw() raw = config_ref.get_raw()
containers = raw.get("containers") or {} containers = raw.get("containers") or {}
@@ -1583,7 +1672,7 @@ def _run_test_notification(channel: str, form: dict, app: AppConfig) -> dict:
try: try:
resp = requests.post( resp = requests.post(
f"{url}/message", f"{url}/message",
params={"token": token}, headers={"X-Gotify-Key": token},
json={"title": title, "message": message, "priority": priority}, json={"title": title, "message": message, "priority": priority},
timeout=5.0, timeout=5.0,
) )
@@ -1665,6 +1754,9 @@ def _run_test_integration(kind: str, form: dict, app: AppConfig) -> dict:
return {"ok": False, "error": "API token required"} return {"ok": False, "error": "API token required"}
else: else:
var = _form_str(form, "api_token_env", "").strip() var = _form_str(form, "api_token_env", "").strip()
saved = app.integrations.portainer.environments.get(_form_str(form, "env_original", ""))
if var and not _token_env_allowed(var, saved.api_token_env if saved else ""):
return {"ok": False, "error": "the environment variable name must start with PORTAINER_"}
token = os.environ.get(var, "") if var else "" token = os.environ.get(var, "") if var else ""
if not token: if not token:
return {"ok": False, "error": f"environment variable {var!r} is not set in the Inspector container"} return {"ok": False, "error": f"environment variable {var!r} is not set in the Inspector container"}
@@ -1974,6 +2066,7 @@ def _render_diag_strip(config_ref: ConfigRef) -> str:
return ( return (
'<div class="diag-strip">' '<div class="diag-strip">'
f'<span>config: <code>{_escape(str(path))}</code></span>' f'<span>config: <code>{_escape(str(path))}</code></span>'
f'<span>time zone: <code>{_escape(formatting.LOCAL_TZ_NAME)}</code></span>'
f"<span>{badge}{hint}</span>" f"<span>{badge}{hint}</span>"
f'<span style="margin-left:auto;">Inspector v{__version__}</span>' f'<span style="margin-left:auto;">Inspector v{__version__}</span>'
"</div>\n" "</div>\n"
@@ -2113,6 +2206,8 @@ def _settings_body_for(section: str, app: AppConfig, csrf_token: str) -> str:
return ( return (
'<h3 class="pane-title">Scheduled maintenance window</h3>' '<h3 class="pane-title">Scheduled maintenance window</h3>'
'<p class="pane-sub">Daily quiet hours. Containers that opt in (<code>maintenance_window: true</code> in their config) are not acted on during the window.</p>' '<p class="pane-sub">Daily quiet hours. Containers that opt in (<code>maintenance_window: true</code> in their config) are not acted on during the window.</p>'
f'<p class="pane-sub" style="margin-top:-1rem;">Times are in <b>{_escape(formatting.LOCAL_TZ_NAME)}</b>, '
"from the <code>TZ</code> setting of the Inspector stack. If that's not your time zone, the window runs at the wrong hours.</p>"
+ _settings_form("maintenance", "Maintenance", body, csrf_token) + _settings_form("maintenance", "Maintenance", body, csrf_token)
) )
@@ -2177,7 +2272,7 @@ def _portainer_env_form(env_name: str, env, csrf_token: str, *, suggest_name: st
+ _settings_field("name", "env_name", env_name or suggest_name, + _settings_field("name", "env_name", env_name or suggest_name,
hint="Your own label, referenced by containers") hint="Your own label, referenced by containers")
+ _settings_field("endpoint_id", "endpoint_id", str(env.endpoint_id if env else 1), kind="number", + _settings_field("endpoint_id", "endpoint_id", str(env.endpoint_id if env else 1), kind="number",
hint="Portainer → Environments; the number in the URL (usually 1 or 2)") hint="Open the environment from Portainer's Home page; the number after #!/ in the address (often 1 or 2)")
+ "</div>" + "</div>"
+ _settings_field("base_url", "base_url", env.base_url if env else "", + _settings_field("base_url", "base_url", env.base_url if env else "",
hint="e.g. http://192.168.1.10:9000 or https://portainer.example.com") hint="e.g. http://192.168.1.10:9000 or https://portainer.example.com")
@@ -2190,7 +2285,7 @@ def _portainer_env_form(env_name: str, env, csrf_token: str, *, suggest_name: st
+ f'<div class="token-src" data-src="env"{" hidden" if inline else ""}>' + f'<div class="token-src" data-src="env"{" hidden" if inline else ""}>'
+ _settings_field("environment variable name", "api_token_env", + _settings_field("environment variable name", "api_token_env",
(env.api_token_env if env else "") or "PORTAINER_API_TOKEN", (env.api_token_env if env else "") or "PORTAINER_API_TOKEN",
hint="Set this variable on the Inspector container (Portainer → stack → Environment variables), then redeploy Inspector.") hint="Add it to the Inspector stack's environment variables; the compose file must pass it in. Step by step: Help → Portainer stack redeploy, step 2.")
+ "</div>" + "</div>"
+ f'<div class="token-src" data-src="inline"{"" if inline else " hidden"}>' + f'<div class="token-src" data-src="inline"{"" if inline else " hidden"}>'
+ _settings_secret_field("api_token", "api_token", bool(env and env.api_token)) + _settings_secret_field("api_token", "api_token", bool(env and env.api_token))
@@ -2203,7 +2298,7 @@ def _portainer_env_form(env_name: str, env, csrf_token: str, *, suggest_name: st
if env is not None: if env is not None:
delete = ( delete = (
'<button type="submit" name="delete" value="1" class="btn-danger" ' '<button type="submit" name="delete" value="1" class="btn-danger" '
f"onclick=\"return confirm('Delete Portainer environment {_escape(env_name)}?');\">Delete</button>" f'onclick="return confirm({_js_string_attr(f"Delete Portainer environment {env_name}?")});">Delete</button>'
) )
label = "environment" if env is not None else "new environment" label = "environment" if env is not None else "new environment"
return ( return (
@@ -2440,7 +2535,7 @@ def _render_container_advanced(name: str, cfg, app: AppConfig) -> str:
+ '<div class="grid-2">' + '<div class="grid-2">'
+ _settings_field("timeout_seconds", "http_check_timeout_seconds", timeout_str, kind="number") + _settings_field("timeout_seconds", "http_check_timeout_seconds", timeout_str, kind="number")
+ _settings_field("expected_status", "http_check_expected_status", str(hc.expected_status), kind="number", + _settings_field("expected_status", "http_check_expected_status", str(hc.expected_status), kind="number",
hint="Exact HTTP code a healthy service returns (often 200; some apps return 401 or 302)") hint="Exact HTTP code a healthy service returns (often 200; some apps return 401). Redirects are followed, so use the final page's code.")
+ "</div></div>" + "</div></div>"
'<div class="form-group"><h4>Uptime Kuma monitor</h4>' '<div class="form-group"><h4>Uptime Kuma monitor</h4>'
+ _settings_field("enabled", "uptime_kuma_enabled", "on" if ukm.enabled else "", kind="switch") + _settings_field("enabled", "uptime_kuma_enabled", "on" if ukm.enabled else "", kind="switch")
@@ -2451,18 +2546,25 @@ def _render_container_advanced(name: str, cfg, app: AppConfig) -> str:
) )
others = [c.name for c in app.containers if c.name != name] others = [c.name for c in app.containers if c.name != name]
disabled = {c.name for c in app.containers if not c.enabled}
dep_value = getattr(cfg, "depends_on", "") or "" dep_value = getattr(cfg, "depends_on", "") or ""
dep_options = [("", "— none —")] + [(n, n) for n in others] dep_options = [("", "— none —")] + [(n, n + (" (disabled)" if n in disabled else "")) for n in others]
if dep_value and dep_value not in others: if dep_value and dep_value not in others:
dep_options.append((dep_value, f"{dep_value} (not configured)")) dep_options.append((dep_value, f"{dep_value} (not configured)"))
dep_warn = ""
if dep_value in disabled:
dep_warn = _warn(f"{_escape(dep_value)} is disabled, so Inspector can't see its status and won't wait for it. "
"Enable it to make this dependency count.")
deps = ( deps = (
'<div class="form-group"><h4>Dependencies</h4>' '<div class="form-group"><h4>Dependencies</h4>'
'<p class="group-sub">Check this container after the one it relies on, and hold off recovering it ' '<p class="group-sub">Check this container after the one it relies on, and hold off recovering it '
"while that one is still failing.</p>" "while that one is still failing.</p>"
+ _settings_select("depends_on", "depends_on", dep_value, dep_options, + _settings_select("depends_on", "depends_on", dep_value, dep_options,
hint="e.g. the VPN container, for apps that use network_mode: container:<vpn>") hint="e.g. the VPN container, for apps that use network_mode: container:<vpn>. "
"Also works for any other dependency, like an app and its database.")
+ dep_warn
+ _settings_field("vpn_routed", "vpn_routed", "on" if getattr(cfg, "vpn_routed", False) else "", kind="switch", + _settings_field("vpn_routed", "vpn_routed", "on" if getattr(cfg, "vpn_routed", False) else "", kind="switch",
hint="Routes through the container above. Logs why recovery is deferred while it is down.") hint="Only affects logging: says in the log when recovery waits on the VPN container. The waiting itself comes from depends_on.")
+ "</div>" + "</div>"
) )
@@ -2573,7 +2675,7 @@ def _render_container_form(
delete_btn = ( delete_btn = (
f'<form method="post" action="/containers/{_escape(name)}/delete" ' f'<form method="post" action="/containers/{_escape(name)}/delete" '
'style="display:inline;" ' 'style="display:inline;" '
f"onsubmit=\"return confirm('Delete container {_escape(name)} from config.yml?');\">" f'onsubmit="return confirm({_js_string_attr(f"Delete container {name} from config.yml?")});">'
f'<input type="hidden" name="csrf" value="{_escape(csrf_token)}">' f'<input type="hidden" name="csrf" value="{_escape(csrf_token)}">'
'<button type="submit" class="btn-danger">Delete container</button>' '<button type="submit" class="btn-danger">Delete container</button>'
"</form>" "</form>"
@@ -2677,8 +2779,39 @@ class _DashboardHTTPServer(ThreadingHTTPServer):
self.states = states self.states = states
self.maintenance = maintenance self.maintenance = maintenance
self.notification_mute = notification_mute self.notification_mute = notification_mute
# token -> (expiry_timestamp, csrf_token) # token -> (expiry_timestamp, csrf_token, credential fingerprint)
self._sessions: dict[str, tuple[float, str]] = {} self._sessions: dict[str, tuple[float, str, str]] = {}
# client address -> timestamps of recent failed logins
self._failed_logins: dict[str, list[float]] = {}
self._failed_lock = threading.Lock()
def login_throttled(self, client: str) -> bool:
cutoff = time.time() - _LOGIN_FAILURE_WINDOW
with self._failed_lock:
recent = [t for t in self._failed_logins.get(client, []) if t > cutoff]
if recent:
self._failed_logins[client] = recent
else:
self._failed_logins.pop(client, None)
return len(recent) >= _LOGIN_MAX_FAILURES
def note_login_failure(self, client: str) -> None:
with self._failed_lock:
self._failed_logins.setdefault(client, []).append(time.time())
def clear_login_failures(self, client: str) -> None:
with self._failed_lock:
self._failed_logins.pop(client, None)
def new_session(self) -> str:
now = time.time()
for tok, entry in list(self._sessions.items()): # drop expired sessions
if entry[0] <= now:
self._sessions.pop(tok, None)
token = secrets.token_urlsafe(32)
fp = _cred_fingerprint(self.current_app().dashboard)
self._sessions[token] = (now + _SESSION_MAX_AGE, secrets.token_urlsafe(32), fp)
return token
def current_app(self) -> AppConfig: def current_app(self) -> AppConfig:
return self.config_ref.get() return self.config_ref.get()
@@ -2703,29 +2836,47 @@ class _Handler(BaseHTTPRequestHandler):
def _is_authenticated(self) -> bool: def _is_authenticated(self) -> bool:
# Session cookie # Session cookie
now = time.time() now = time.time()
cfg = self.server.current_app().dashboard
token = self._session_token() token = self._session_token()
if token: if token:
entry = self.server._sessions.get(token) entry = self.server._sessions.get(token)
if entry and now < entry[0]: if entry and now < entry[0] and entry[2] == _cred_fingerprint(cfg):
return True return True
self.server._sessions.pop(token, None) self.server._sessions.pop(token, None)
# Basic Auth # Basic Auth
header = self.headers.get("Authorization", "") header = self.headers.get("Authorization", "")
if header.startswith("Basic "): if header.startswith("Basic "):
client = self.client_address[0]
if self.server.login_throttled(client):
return False
try: try:
decoded = b64decode(header[len("Basic "):]).decode("utf-8") decoded = b64decode(header[len("Basic "):]).decode("utf-8")
username, _, password = decoded.partition(":") username, _, password = decoded.partition(":")
except Exception: except Exception:
username, password = "", "" username, password = "", ""
cfg = self.server.current_app().dashboard if _creds_match(username, password, cfg):
if secrets.compare_digest(username, cfg.username) and secrets.compare_digest( self.server.clear_login_failures(client)
password, cfg.password
):
return True return True
self.server.note_login_failure(client)
return False return False
def _read_form(self) -> dict | None:
"""Parse the urlencoded request body, or send 400/413 and return None."""
try:
length = int(self.headers.get("Content-Length", 0))
except ValueError:
length = -1
if length < 0:
self._send_bytes(HTTPStatus.BAD_REQUEST, "text/plain; charset=utf-8", b"bad Content-Length")
return None
if length > _MAX_BODY_BYTES:
self._send_bytes(HTTPStatus.REQUEST_ENTITY_TOO_LARGE, "text/plain; charset=utf-8", b"request too large")
return None
raw = self.rfile.read(length) if length > 0 else b""
return parse_qs(raw.decode("utf-8", errors="replace"))
def _csrf_token_for_session(self) -> str: def _csrf_token_for_session(self) -> str:
"""Return the CSRF token bound to the current session, or '' if none.""" """Return the CSRF token bound to the current session, or '' if none."""
token = self._session_token() token = self._session_token()
@@ -2883,18 +3034,21 @@ class _Handler(BaseHTTPRequestHandler):
server = self.server server = self.server
if path == "/login": if path == "/login":
length = int(self.headers.get("Content-Length", 0)) data = self._read_form()
raw = self.rfile.read(length) if length > 0 else b"" if data is None:
data = parse_qs(raw.decode("utf-8", errors="replace")) return
client = self.client_address[0]
if server.login_throttled(client):
self._send_bytes(
HTTPStatus.TOO_MANY_REQUESTS, "text/html; charset=utf-8", _render_login_page(throttled=True)
)
return
username = data.get("username", [""])[0] username = data.get("username", [""])[0]
password = data.get("password", [""])[0] password = data.get("password", [""])[0]
cfg = server.current_app().dashboard cfg = server.current_app().dashboard
if secrets.compare_digest(username, cfg.username) and secrets.compare_digest( if _creds_match(username, password, cfg):
password, cfg.password server.clear_login_failures(client)
): token = server.new_session()
token = secrets.token_urlsafe(32)
csrf = secrets.token_urlsafe(32)
server._sessions[token] = (time.time() + _SESSION_MAX_AGE, csrf)
self.send_response(HTTPStatus.SEE_OTHER) self.send_response(HTTPStatus.SEE_OTHER)
self.send_header("Location", "/") self.send_header("Location", "/")
self.send_header( self.send_header(
@@ -2904,6 +3058,7 @@ class _Handler(BaseHTTPRequestHandler):
self.send_header("Content-Length", "0") self.send_header("Content-Length", "0")
self.end_headers() self.end_headers()
else: else:
server.note_login_failure(client)
self._send_bytes( self._send_bytes(
HTTPStatus.UNAUTHORIZED, "text/html; charset=utf-8", _render_login_page(error=True) HTTPStatus.UNAUTHORIZED, "text/html; charset=utf-8", _render_login_page(error=True)
) )
@@ -2912,9 +3067,9 @@ class _Handler(BaseHTTPRequestHandler):
if not self._check_auth(): if not self._check_auth():
return return
length = int(self.headers.get("Content-Length", 0)) form = self._read_form()
raw_body = self.rfile.read(length) if length > 0 else b"" if form is None:
form = parse_qs(raw_body.decode("utf-8", errors="replace")) return
# Only cookie-authenticated requests need a CSRF token. Basic Auth # Only cookie-authenticated requests need a CSRF token. Basic Auth
# clients prove identity on every request so they're not subject to # clients prove identity on every request so they're not subject to
+45 -6
View File
@@ -1,14 +1,49 @@
from __future__ import annotations from __future__ import annotations
from datetime import datetime, timezone import logging
from zoneinfo import ZoneInfo import os
from datetime import datetime, timezone, tzinfo
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
PACIFIC = ZoneInfo("America/Los_Angeles") log = logging.getLogger("inspector.formatting")
def format_timestamp(value: datetime | str | None) -> str: def _resolve_local_tz(env: dict | None = None) -> tuple[tzinfo, str]:
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'. """The zone every displayed time uses, plus its name.
Follows the same rules as the C library (which drives log timestamps and
the maintenance window), so all times Inspector shows agree:
TZ set to a zone name -> that zone; TZ unset -> the container's
/etc/localtime; anything else -> UTC.
"""
env = os.environ if env is None else env
name = (env.get("TZ") or "").strip().lstrip(":")
if name:
try:
return ZoneInfo(name), name
except (ZoneInfoNotFoundError, ValueError):
log.warning(
"TZ=%r is not a known time zone name (e.g. Europe/Berlin); showing times in UTC",
name,
)
return timezone.utc, "UTC"
try:
with open("/etc/localtime", "rb") as fh:
zone = ZoneInfo.from_file(fh, key="localtime")
label = datetime.now(zone).tzname() or "localtime"
return zone, label
except (OSError, ValueError):
return timezone.utc, "UTC"
# Resolved once at import; the container's TZ doesn't change while it runs.
LOCAL_TZ, LOCAL_TZ_NAME = _resolve_local_tz()
def format_timestamp(value: datetime | str | None, *, with_zone: bool = False) -> str:
"""Render a timestamp in the configured local zone, e.g. '06-14 03:45 PM'
(with_zone adds the abbreviation: '06-14 03:45 PM CET').
Accepts a datetime, an ISO 8601 string (as stored by EventStore / Accepts a datetime, an ISO 8601 string (as stored by EventStore /
ContainerState), or None/"" (returns ""). Naive datetimes/strings are ContainerState), or None/"" (returns ""). Naive datetimes/strings are
@@ -24,4 +59,8 @@ def format_timestamp(value: datetime | str | None) -> str:
return value return value
if value.tzinfo is None: if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc) value = value.replace(tzinfo=timezone.utc)
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p") local = value.astimezone(LOCAL_TZ)
text = local.strftime("%m-%d %I:%M %p")
if with_zone:
text += f" {local.tzname()}"
return text
+43 -15
View File
@@ -41,7 +41,7 @@ from .maintenance import MaintenanceTracker
from .notification_mute import NotificationMuteTracker from .notification_mute import NotificationMuteTracker
from .notifier import GotifyNotifier, MultiNotifier, NullNotifier, PushoverNotifier from .notifier import GotifyNotifier, MultiNotifier, NullNotifier, PushoverNotifier
from .portainer_client import PortainerClient from .portainer_client import PortainerClient
from .recovery import perform_recovery from .recovery import perform_recovery, run_post_redeploy_recheck
from .uptime_kuma import HeartbeatSnapshot, UptimeKumaClient from .uptime_kuma import HeartbeatSnapshot, UptimeKumaClient
from .state import ( from .state import (
ContainerState, ContainerState,
@@ -63,6 +63,19 @@ def _request_stop(signum, _frame):
_stop = True _stop = True
def _is_normal(status: str | None, container_config: ContainerConfig) -> bool:
"""True when `status` means the container is working.
HEALTHY always counts. NO HEALTH CHECK (running, but the image defines no
health check) counts too, unless the user made it a restart trigger.
Many images have no health check, and without this their error count and
restart episode would never reset.
"""
if status == HEALTHY:
return True
return status == NO_HEALTH_CHECK and NO_HEALTH_CHECK not in container_config.restart_on
def _apply_reachability_override( def _apply_reachability_override(
container_config: ContainerConfig, container_config: ContainerConfig,
result: HealthResult, result: HealthResult,
@@ -140,6 +153,11 @@ def _check_one(
if notification_mute.is_muted(name): if notification_mute.is_muted(name):
notifier = NullNotifier() notifier = NullNotifier()
if state.redeploy_recheck_at is not None and now >= state.redeploy_recheck_at:
run_post_redeploy_recheck(
container_config=container_config, state=state, notifier=notifier, now=now,
)
if state.previous_status != state.current_status: if state.previous_status != state.current_status:
log.info( log.info(
"%-*s status: %s -> %s%s", "%-*s status: %s -> %s%s",
@@ -158,19 +176,25 @@ def _check_one(
message=result.message, message=result.message,
) )
is_normal = _is_normal(state.current_status, container_config)
if ( if (
state.previous_status is not None state.previous_status is not None
and state.previous_status != HEALTHY and not _is_normal(state.previous_status, container_config)
and state.current_status == HEALTHY and is_normal
# HEALTHY here means "a recovery was already announced", for either status.
and state.last_notification_state != HEALTHY and state.last_notification_state != HEALTHY
): ):
log.info("%-*s recovered: %s -> HEALTHY", width, name, state.previous_status) log.info("%-*s recovered: %s -> %s", width, name, state.previous_status, state.current_status)
back_to = (
"**HEALTHY**" if state.current_status == HEALTHY
else "**running** (no health check)"
)
notifier.notify( notifier.notify(
title=f"✅ Inspector: {name} recovered", title=f"✅ Inspector: {name} recovered",
message=( message=(
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n" f"**{name}** is back to {back_to} (was *{state.previous_status}*).\n\n"
f"- Errors before recovery: **{state.error_count}**\n" f"- Errors before recovery: **{state.error_count}**\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=3, priority=3,
) )
@@ -178,7 +202,7 @@ def _check_one(
container=name, container=name,
event_type=EVENT_RECOVERY, event_type=EVENT_RECOVERY,
old_status=state.previous_status, old_status=state.previous_status,
new_status=HEALTHY, new_status=state.current_status,
error_count=state.error_count, error_count=state.error_count,
) )
state.last_notification_state = HEALTHY state.last_notification_state = HEALTHY
@@ -196,7 +220,7 @@ def _check_one(
f"**{name}** is missing from Docker.\n\n" f"**{name}** is missing from Docker.\n\n"
+ (f"- Detail: {result.message}\n" if result.message else "") + (f"- Detail: {result.message}\n" if result.message else "")
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n" + f"- Last known status: *{state.previous_status or 'unknown'}*\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=7, priority=7,
) )
@@ -240,7 +264,7 @@ def _check_one(
+ ".\n\n" + ".\n\n"
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n" f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
f"- Previous status: *{state.previous_status or 'unknown'}*\n" f"- Previous status: *{state.previous_status or 'unknown'}*\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=5, priority=5,
) )
@@ -253,9 +277,9 @@ def _check_one(
message=result.message, message=result.message,
) )
state.last_notification_state = result.status state.last_notification_state = result.status
elif result.status == HEALTHY: elif is_normal:
if state.error_count != 0: if state.error_count != 0:
log.info("%-*s healthy again; resetting error count", width, name) log.info("%-*s back to normal; resetting error count", width, name)
state.error_count = 0 state.error_count = 0
state.restarted_this_episode = False state.restarted_this_episode = False
@@ -276,7 +300,7 @@ def _check_one(
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n" f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
f"- Current status: **{result.status}**\n" f"- Current status: **{result.status}**\n"
f"- Restarts paused until the hourly window clears\n" f"- Restarts paused until the hourly window clears\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=9, priority=9,
) )
@@ -293,8 +317,10 @@ def _check_one(
if ok: if ok:
dep = container_config.depends_on dep = container_config.depends_on
if dep: # Only wait on a dependency Inspector is watching. A disabled one has
dep_status = states[dep].current_status if dep in states else None # no state, and waiting on it would block recovery forever.
if dep and dep in states:
dep_status = states[dep].current_status
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK): if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
if container_config.vpn_routed: if container_config.vpn_routed:
log.info( log.info(
@@ -454,8 +480,10 @@ def run_loop(config_ref: ConfigRef) -> int:
for gone in states.keys() - current_enabled.keys(): for gone in states.keys() - current_enabled.keys():
states.pop(gone, None) states.pop(gone, None)
log.info("Stopped watching container (removed or disabled): %s", gone) log.info("Stopped watching container (removed or disabled): %s", gone)
if current_enabled != enabled_configs:
# Only re-order (and re-log ordering warnings) when the config changed.
check_order = _topological_order(current_enabled)
enabled_configs = current_enabled enabled_configs = current_enabled
check_order = _topological_order(enabled_configs)
# Integration clients hold their config; rebuild them when the # Integration clients hold their config; rebuild them when the
# settings page changes Uptime Kuma / Portainer. # settings page changes Uptime Kuma / Portainer.
+2 -1
View File
@@ -33,9 +33,10 @@ class GotifyNotifier:
}, },
} }
try: try:
# Header, not ?token=, so the token stays out of access logs.
resp = requests.post( resp = requests.post(
url, url,
params={"token": self._config.token}, headers={"X-Gotify-Key": self._config.token},
json=payload, json=payload,
timeout=self._timeout, timeout=self._timeout,
) )
+57 -35
View File
@@ -1,7 +1,6 @@
from __future__ import annotations from __future__ import annotations
import logging import logging
import time
from datetime import datetime, timedelta from datetime import datetime, timedelta
from .config import AppConfig, ContainerConfig, PortainerConfig from .config import AppConfig, ContainerConfig, PortainerConfig
@@ -175,7 +174,8 @@ def _perform_restart_then_redeploy(
- the container is still failing on a later recovery cycle - the container is still failing on a later recovery cycle
(`state.restarted_this_episode` is already set), meaning the restart didn't stick. (`state.restarted_this_episode` is already set), meaning the restart didn't stick.
The episode flag resets when the container is next seen HEALTHY (see main._check_one). The episode flag resets when the container is next seen working: HEALTHY, or
NO HEALTH CHECK when that isn't a restart trigger (see main._is_normal).
""" """
name = container_config.name name = container_config.name
@@ -228,7 +228,7 @@ def _perform_restart_then_redeploy(
f"**{result.status}** check(s).\n\n" f"**{result.status}** check(s).\n\n"
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n" f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n" f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=6, priority=6,
) )
@@ -242,7 +242,7 @@ def _perform_restart_then_redeploy(
state.restart_limit_notified = False state.restart_limit_notified = False
log.info( log.info(
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)", "%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
width, name, state.cooldown_until.isoformat(), width, name, format_timestamp(state.cooldown_until, with_zone=True),
) )
eventstore.record( eventstore.record(
container=name, container=name,
@@ -250,15 +250,15 @@ def _perform_restart_then_redeploy(
old_status=state.previous_status, old_status=state.previous_status,
new_status=result.status, new_status=result.status,
error_count=state.error_count, error_count=state.error_count,
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}", message=f"restart succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
) )
notifier.notify( notifier.notify(
title=f"✅ Inspector: {name} restart completed", title=f"✅ Inspector: {name} restart completed",
message=( message=(
f"**{name}** was successfully restarted by Docker.\n\n" f"**{name}** was successfully restarted by Docker.\n\n"
f"- Previous status: *{result.status}*\n" f"- Previous status: *{result.status}*\n"
f"- Cooldown until: {state.cooldown_until.isoformat()}\n" f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=3, priority=3,
) )
@@ -321,7 +321,7 @@ def _perform_docker_action(
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive " f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
f"**{result.status}** check(s).\n\n" f"**{result.status}** check(s).\n\n"
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n" f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=6, priority=6,
) )
@@ -335,22 +335,22 @@ def _perform_docker_action(
state.record_restart(now, container_config.cooldown_seconds) state.record_restart(now, container_config.cooldown_seconds)
state.last_notification_state = "RESTARTED" state.last_notification_state = "RESTARTED"
state.restart_limit_notified = False state.restart_limit_notified = False
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat()) log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, format_timestamp(state.cooldown_until, with_zone=True))
eventstore.record( eventstore.record(
container=name, container=name,
event_type=EVENT_RESTART, event_type=EVENT_RESTART,
old_status=state.previous_status, old_status=state.previous_status,
new_status=result.status, new_status=result.status,
error_count=state.error_count, error_count=state.error_count,
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}", message=f"{verb} succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
) )
notifier.notify( notifier.notify(
title=f"✅ Inspector: {name} {verb} completed", title=f"✅ Inspector: {name} {verb} completed",
message=( message=(
f"**{name}** was successfully {verb}ed by Docker.\n\n" f"**{name}** was successfully {verb}ed by Docker.\n\n"
f"- Previous status: *{result.status}*\n" f"- Previous status: *{result.status}*\n"
f"- Cooldown until: {state.cooldown_until.isoformat()}\n" f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=3, priority=3,
) )
@@ -362,7 +362,7 @@ def _perform_docker_action(
f"{verb.capitalize()} of **{name}** failed.\n\n" f"{verb.capitalize()} of **{name}** failed.\n\n"
f"- Status: **{result.status}**\n" f"- Status: **{result.status}**\n"
f"- Error count: **{state.error_count}**\n" f"- Error count: **{state.error_count}**\n"
f"- Time: {format_timestamp(now)}\n\n" f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
f"```\n{err}\n```" f"```\n{err}\n```"
), ),
priority=9, priority=9,
@@ -469,7 +469,7 @@ def _perform_portainer_redeploy(
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) " f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
f"to recover **{name}**.\n\n" f"to recover **{name}**.\n\n"
f"- Status: **{result.status}**\n" f"- Status: **{result.status}**\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=6, priority=6,
) )
@@ -483,7 +483,7 @@ def _perform_portainer_redeploy(
message=( message=(
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n" f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
f"- Triggered by: **{name}** ({result.status})\n" f"- Triggered by: **{name}** ({result.status})\n"
f"- Time: {format_timestamp(now)}\n\n" f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
f"```\n{redeploy_result.message}\n```" f"```\n{redeploy_result.message}\n```"
), ),
priority=9, priority=9,
@@ -516,29 +516,51 @@ def _perform_portainer_redeploy(
f"successfully redeployed to recover **{name}**.\n\n" f"successfully redeployed to recover **{name}**.\n\n"
f"- Triggered by: *{result.status}*\n" f"- Triggered by: *{result.status}*\n"
f"- Post-redeploy wait: {wait_seconds}s\n" f"- Post-redeploy wait: {wait_seconds}s\n"
f"- Time: {format_timestamp(now)}" f"- Time: {format_timestamp(now, with_zone=True)}"
), ),
priority=4, priority=4,
) )
log.info("waiting %d seconds after stack redeploy", wait_seconds) # Don't block the loop while the stack settles: keep this container in
time.sleep(wait_seconds) # cooldown for the wait plus its normal cooldown, and do the one-time HTTP
# re-check on its first regular check after the wait (main._check_one).
log.info(
"%-*s letting stack %s settle for %ds; other containers keep being checked",
width, name, key, wait_seconds,
)
state.record_restart(now, wait_seconds + container_config.cooldown_seconds)
if container_config.http_check.enabled: if container_config.http_check.enabled:
recheck = check_http(container_config.http_check) state.redeploy_recheck_at = now + timedelta(seconds=wait_seconds)
if recheck.ok:
log.info("recheck succeeded for %s", name)
else:
log.warning("recheck failed for %s: %s", name, recheck.message)
notifier.notify(
title=f"⚠️ Inspector: {name} recheck failed after redeploy",
message=(
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
f"for **{name}** still reports a failure.\n\n"
f"- Recheck result: *{recheck.message}*\n"
f"- Time: {format_timestamp(now)}"
),
priority=6,
)
state.record_restart(now, container_config.cooldown_seconds)
def run_post_redeploy_recheck(
*,
container_config: ContainerConfig,
state: ContainerState,
notifier: GotifyNotifier,
now: datetime,
) -> None:
"""One HTTP re-check after a stack redeploy has had time to settle.
Called by main._check_one once `state.redeploy_recheck_at` has passed;
alerts if the app still doesn't answer.
"""
state.redeploy_recheck_at = None
name = container_config.name
if not container_config.http_check.enabled:
return
recheck = check_http(container_config.http_check)
if recheck.ok:
log.info("recheck succeeded for %s", name)
return
log.warning("recheck failed for %s: %s", name, recheck.message)
notifier.notify(
title=f"⚠️ Inspector: {name} recheck failed after redeploy",
message=(
f"Portainer stack **{container_config.portainer_stack_name}** was redeployed, but the HTTP recheck "
f"for **{name}** still reports a failure.\n\n"
f"- Recheck result: *{recheck.message}*\n"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
+2
View File
@@ -29,6 +29,8 @@ class ContainerState:
restart_limit_notified: bool = False restart_limit_notified: bool = False
restarted_this_episode: bool = False restarted_this_episode: bool = False
last_check_at: datetime | None = None last_check_at: datetime | None = None
# After a stack redeploy: when to do the one-time HTTP re-check.
redeploy_recheck_at: datetime | None = None
def in_cooldown(self, now: datetime) -> bool: def in_cooldown(self, now: datetime) -> bool:
return self.cooldown_until is not None and now < self.cooldown_until return self.cooldown_until is not None and now < self.cooldown_until