Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2a9c57b7d7 | ||
|
|
6aed0cf9a7 | ||
|
|
57100d0a7b | ||
|
|
b4da527732 | ||
|
|
284ba2ea15 |
No files matched your search
+128
@@ -7,6 +7,134 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## [0.9.5] — 2026-10-28
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **A stack redeploy no longer freezes Inspector.** After a successful Portainer
|
||||||
|
redeploy, Inspector used to stop everything for `post_redeploy_wait_seconds`
|
||||||
|
(default 90 s): no other container was checked, dashboard times stood still, and
|
||||||
|
stopping Inspector during the wait wasn't clean. Now the redeployed app is left to
|
||||||
|
settle while every other container keeps being checked, and its one-time HTTP
|
||||||
|
re-check (with the "recheck failed" alert) happens on its first check after the wait.
|
||||||
|
- The redeployed app's cooldown now starts after the settle time. Before, the wait
|
||||||
|
ate into it (with the defaults, 300 s cooldown became 210 s).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [0.9.4] — 2026-10-28
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Containers without a Docker health check now "recover" properly.** Many images
|
||||||
|
don't define a health check, so Docker reports them as `NO HEALTH CHECK` even when
|
||||||
|
they work. Inspector only treated `HEALTHY` as back to normal, so for these
|
||||||
|
containers:
|
||||||
|
- the error count never reset, and one-off failures days apart added up to a restart;
|
||||||
|
- no "recovered" alert was sent;
|
||||||
|
- with *Restart, then redeploy*, the next unrelated problem skipped the restart and
|
||||||
|
redeployed the whole stack.
|
||||||
|
|
||||||
|
Now `NO HEALTH CHECK` counts as working (unless you made it a restart trigger): the
|
||||||
|
count resets, the "recovered" alert is sent ("back to running (no health check)"),
|
||||||
|
and the next problem starts with a plain restart. Containers with a health check
|
||||||
|
behave exactly as before.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [0.9.3] — 2026-10-28
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Every time follows the `TZ` of the Inspector stack.** Before, the dashboard,
|
||||||
|
Events page and alerts were always in US Pacific time while the maintenance window
|
||||||
|
and logs used `TZ`, so anyone outside Pacific saw times hours off (e.g. a 02:00
|
||||||
|
Berlin window shown as 5:00 PM). Now the dashboard, the Events page (including
|
||||||
|
where each day starts), alert times, cooldown times and logs all use `TZ`.
|
||||||
|
- Alert times include the zone abbreviation (`10-28 09:27 AM CET`), and "Cooldown
|
||||||
|
until" in alerts, event messages and logs is a local time instead of a UTC
|
||||||
|
timestamp.
|
||||||
|
- The active time zone is shown at the bottom of every Settings page, on the Events
|
||||||
|
page and on the Maintenance page.
|
||||||
|
- **If you never set `TZ`**, times are now shown in UTC instead of Pacific. Set `TZ`
|
||||||
|
in the stack (e.g. `America/New_York`). An unrecognized name logs a warning and
|
||||||
|
falls back to UTC.
|
||||||
|
|
||||||
|
### Docs
|
||||||
|
|
||||||
|
- INSTALL: `TZ` is now marked as needed, with a "Why the time zone matters" section
|
||||||
|
(maintenance window, alerts, dashboard, logs), how to find your zone name, and a
|
||||||
|
troubleshooting row. Guide: a Time zone section and a first-login check. Compose
|
||||||
|
and README updated.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [0.9.2] — 2026-10-28
|
||||||
|
|
||||||
|
Fixes from a code and security review. None of them change how containers are
|
||||||
|
monitored or recovered.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **A bad value saved from Settings can no longer stop Inspector from starting.**
|
||||||
|
The forms now reject invalid times (maintenance start/end), unknown log levels and
|
||||||
|
out-of-range numbers (for example `interval_seconds: 0`, which made the loop spin).
|
||||||
|
If `config.yml` already has such a value, Inspector starts with the default and
|
||||||
|
logs a warning instead of crashing in a restart loop.
|
||||||
|
- **The dashboard password can no longer be cleared.** Saving an empty password (or
|
||||||
|
username) is refused, and an empty password never signs anyone in. Before, clearing
|
||||||
|
it left the running dashboard open to anyone.
|
||||||
|
- **Passwords with non-ASCII characters (é, ü, £ …) now work.** Before, every login
|
||||||
|
attempt with one crashed.
|
||||||
|
- The Events page no longer errors for a moment after a container is removed.
|
||||||
|
- The HTTP check hint no longer suggests `302` as an expected status; redirects are
|
||||||
|
followed, so the final page's code is what counts.
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
- **Sign-in throttling:** after 5 failed sign-ins (form or Basic Auth) from one address
|
||||||
|
within 5 minutes, that address is refused for the rest of the window.
|
||||||
|
- **Changing the dashboard username or password signs out existing sessions.**
|
||||||
|
Expired sessions are cleaned up.
|
||||||
|
- Request bodies are limited to 1 MB; an invalid `Content-Length` gets a 400.
|
||||||
|
- The Gotify token is sent in the `X-Gotify-Key` header instead of the URL, so it no
|
||||||
|
longer appears in Gotify or reverse-proxy access logs.
|
||||||
|
- Portainer token environment variables must be named `PORTAINER_…` (a name already in
|
||||||
|
your `config.yml` keeps working), so Test connection can't be used to send other
|
||||||
|
container secrets to an arbitrary URL.
|
||||||
|
- Single quotes are escaped in HTML, and the delete confirmations build their text safely.
|
||||||
|
- Two settings saves at the same moment can no longer overwrite each other.
|
||||||
|
- `requests` updated to 2.32.5 (CVE-2024-47081).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [0.9.1] — 2026-10-28
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **A dependency on a disabled container no longer blocks recovery forever.** If
|
||||||
|
`depends_on` named a container that is disabled in Inspector, the dependent app was
|
||||||
|
never restarted. Inspector now only waits for dependencies it is watching, and the
|
||||||
|
container page warns when the selected dependency is disabled.
|
||||||
|
- **`PORTAINER_API_TOKEN` now reaches the container.** The public `compose.yml` didn't
|
||||||
|
pass it in, so setting it as a Portainer stack variable had no effect. Existing
|
||||||
|
stacks need the line `- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}` added under
|
||||||
|
`environment:` (Stacks → inspector → Editor).
|
||||||
|
- The "depends_on … is disabled or not found" warning was logged on every check
|
||||||
|
(since 0.8.0). It is now logged once, and again only when the container settings
|
||||||
|
change.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Guide: new sections **6. Apps behind a VPN container** (why depends_on matters, how to
|
||||||
|
spot a VPN-routed app, setup, what Inspector does) and **7. Portainer stack redeploy**
|
||||||
|
(why it's needed, what a redeploy does to the whole stack, requirements, where to find
|
||||||
|
every value, adding the token, the other settings, Test connection errors).
|
||||||
|
- Clearer form hints: `vpn_routed` only affects logging; `depends_on` works for any
|
||||||
|
dependency; where to find the endpoint ID.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## [0.9.0] — 2026-10-28
|
## [0.9.0] — 2026-10-28
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
+30
-1
@@ -52,10 +52,38 @@ TZ=America/Los_Angeles
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
|
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
|
||||||
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
|
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
|
||||||
| `TZ` | Your [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), so times on the dashboard are local. | Recommended |
|
| `TZ` | **Your time zone**, e.g. `America/New_York` or `Europe/Berlin`. Without it Inspector uses UTC, so the maintenance window, alerts and dashboard times are off by your UTC offset. See [Why the time zone matters](#why-the-time-zone-matters). | **Yes** (default `UTC`) |
|
||||||
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
|
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
|
||||||
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
|
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
|
||||||
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
|
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
|
||||||
|
| `PORTAINER_API_TOKEN` | A Portainer access token. Only needed if you use **Portainer stack redeploy** for VPN-routed apps; see the in-app guide (Help → Portainer). | No |
|
||||||
|
|
||||||
|
### Why the time zone matters
|
||||||
|
|
||||||
|
Inspector uses `TZ` for every time it shows or acts on. If it's wrong, things
|
||||||
|
still work, but at the wrong hour or with confusing times:
|
||||||
|
|
||||||
|
- **The maintenance window runs in this time zone.** A window of `02:00–05:00`
|
||||||
|
with `TZ` left at UTC runs at 02:00 UTC, which is the middle of the evening in
|
||||||
|
the Americas or mid-morning in Asia. Inspector would then restart containers
|
||||||
|
during your nightly backups, and ignore real problems during the day.
|
||||||
|
- **Alerts** (Gotify / Pushover) show the time of the problem in this zone, with
|
||||||
|
its abbreviation, e.g. `10-28 03:15 AM CET`.
|
||||||
|
- **The dashboard and Events page** show times in this zone, and the Events
|
||||||
|
page's days run midnight to midnight in this zone.
|
||||||
|
- **Logs** are stamped in this zone.
|
||||||
|
|
||||||
|
**How to set it:** use your zone's name from the
|
||||||
|
[list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
|
||||||
|
("TZ identifier" column), e.g. `America/New_York`, `America/Chicago`,
|
||||||
|
`Europe/London`, `Europe/Berlin`, `Australia/Sydney`. Abbreviations like `EST` or
|
||||||
|
`CET` don't work. On Unraid, **Settings → Date and Time** shows the zone your
|
||||||
|
server uses.
|
||||||
|
|
||||||
|
**How to check it:** the bottom of every Settings page shows
|
||||||
|
`time zone: <name>`. If it says `UTC` and you're not on UTC, set `TZ` in the
|
||||||
|
stack and click **Update the stack**. If the name you set isn't recognized,
|
||||||
|
Inspector logs a warning and uses UTC.
|
||||||
|
|
||||||
## 3. Deploy and sign in
|
## 3. Deploy and sign in
|
||||||
|
|
||||||
@@ -115,6 +143,7 @@ starts and restarts the ones you add to it.
|
|||||||
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
|
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
|
||||||
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
|
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
|
||||||
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
|
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
|
||||||
|
| Times are a few hours off, or the maintenance window runs at the wrong time | `TZ` is missing or wrong. Check `time zone:` at the bottom of Settings, set `TZ` to your zone name (e.g. `Europe/London`), and update the stack. |
|
||||||
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
|
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
|
||||||
|
|
||||||
## Security notes
|
## Security notes
|
||||||
|
|||||||
@@ -35,6 +35,8 @@ restarts them when they break, and tells your phone about it.</p>
|
|||||||
- **Stays out of the way when you're working.** Put a container in maintenance or
|
- **Stays out of the way when you're working.** Put a container in maintenance or
|
||||||
mute its alerts with one click, or set a nightly maintenance window for backups
|
mute its alerts with one click, or set a nightly maintenance window for backups
|
||||||
and updates.
|
and updates.
|
||||||
|
- **Works in your time zone.** Set `TZ` in the stack and the maintenance window,
|
||||||
|
alerts, dashboard and logs all follow it.
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
@@ -91,7 +93,8 @@ and that socket is effectively root on your server. So:
|
|||||||
|
|
||||||
- Keep the dashboard on your home network, or put it behind a VPN or a reverse
|
- Keep the dashboard on your home network, or put it behind a VPN or a reverse
|
||||||
proxy with authentication. Don't expose port 8080 to the internet.
|
proxy with authentication. Don't expose port 8080 to the internet.
|
||||||
- Use a real dashboard password.
|
- Use a real dashboard password. Repeated failed sign-ins from one address are
|
||||||
|
blocked for a few minutes.
|
||||||
- Inspector itself runs as an unprivileged user; it starts as root only to detect
|
- Inspector itself runs as an unprivileged user; it starts as root only to detect
|
||||||
the socket's permissions and fix ownership of its own folders.
|
the socket's permissions and fix ownership of its own folders.
|
||||||
|
|
||||||
|
|||||||
+6
-1
@@ -7,13 +7,17 @@
|
|||||||
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
|
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
|
||||||
# INSPECTOR_PASSWORD Dashboard password (username is "admin").
|
# INSPECTOR_PASSWORD Dashboard password (username is "admin").
|
||||||
# Only used on first start; change it later in Settings.
|
# Only used on first start; change it later in Settings.
|
||||||
# TZ Your timezone, e.g. America/Los_Angeles
|
# TZ Your time zone name, e.g. America/New_York, Europe/Berlin.
|
||||||
|
# Important: the maintenance window, alerts, dashboard and
|
||||||
|
# logs all use it. Left unset, everything is in UTC.
|
||||||
#
|
#
|
||||||
# Optional:
|
# Optional:
|
||||||
# INSPECTOR_PORT Port for the dashboard on your server (default 8080).
|
# INSPECTOR_PORT Port for the dashboard on your server (default 8080).
|
||||||
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
|
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
|
||||||
# PUID / PGID User/group Inspector runs as (default 1000/1000).
|
# PUID / PGID User/group Inspector runs as (default 1000/1000).
|
||||||
# Unraid users often prefer 99 / 100.
|
# Unraid users often prefer 99 / 100.
|
||||||
|
# PORTAINER_API_TOKEN Portainer access token, only if you use stack redeploy
|
||||||
|
# (Settings -> Portainer in the dashboard).
|
||||||
#
|
#
|
||||||
# No config file to create, no docker group ID to look up: Inspector creates
|
# No config file to create, no docker group ID to look up: Inspector creates
|
||||||
# its config on first start and detects Docker socket permissions by itself.
|
# its config on first start and detects Docker socket permissions by itself.
|
||||||
@@ -36,3 +40,4 @@ services:
|
|||||||
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
|
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
|
||||||
- PUID=${PUID:-1000}
|
- PUID=${PUID:-1000}
|
||||||
- PGID=${PGID:-1000}
|
- PGID=${PGID:-1000}
|
||||||
|
- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}
|
||||||
+1
-1
@@ -1,5 +1,5 @@
|
|||||||
docker==7.1.0
|
docker==7.1.0
|
||||||
PyYAML==6.0.2
|
PyYAML==6.0.2
|
||||||
requests==2.32.3
|
requests==2.32.5
|
||||||
colorlog==6.9.0
|
colorlog==6.9.0
|
||||||
ruamel.yaml==0.18.6
|
ruamel.yaml==0.18.6
|
||||||
@@ -1 +1 @@
|
|||||||
__version__ = "0.9.0"
|
__version__ = "0.9.5"
|
||||||
@@ -78,10 +78,12 @@
|
|||||||
<a href="#status">3. The Status page</a>
|
<a href="#status">3. The Status page</a>
|
||||||
<a href="#notifications">4. Notifications</a>
|
<a href="#notifications">4. Notifications</a>
|
||||||
<a href="#containers">5. Managing containers</a>
|
<a href="#containers">5. Managing containers</a>
|
||||||
<a href="#events">6. Events history</a>
|
<a href="#vpn">6. Apps behind a VPN</a>
|
||||||
<a href="#settings">7. Other settings</a>
|
<a href="#portainer">7. Portainer stack redeploy</a>
|
||||||
<a href="#restart">8. "Requires restart" fields</a>
|
<a href="#events">8. Events history</a>
|
||||||
<a href="#trouble">9. Troubleshooting</a>
|
<a href="#settings">9. Other settings</a>
|
||||||
|
<a href="#restart">10. "Requires restart" fields</a>
|
||||||
|
<a href="#trouble">11. Troubleshooting</a>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
<main>
|
<main>
|
||||||
@@ -101,6 +103,8 @@ dashboard; you never need to touch a config file.</p>
|
|||||||
<i>Inspector dashboard login</i> with the username and password.</li>
|
<i>Inspector dashboard login</i> with the username and password.</li>
|
||||||
<li>Open <code>http://<your-server>:8080</code> and sign in.</li>
|
<li>Open <code>http://<your-server>:8080</code> and sign in.</li>
|
||||||
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
|
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
|
||||||
|
<li>Check the <b>time zone</b>: at the bottom of any Settings page it says
|
||||||
|
<code>time zone: …</code>. If that's not your zone, see <a href="#settings">Time zone</a>.</li>
|
||||||
</ol>
|
</ol>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
@@ -175,7 +179,7 @@ Each card shows:</p>
|
|||||||
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
|
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
|
||||||
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
|
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
|
||||||
the reason appears next to the button.</li>
|
the reason appears next to the button.</li>
|
||||||
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 8</a>),
|
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 10</a>),
|
||||||
because turning a channel on or off is a "requires restart" field.</li>
|
because turning a channel on or off is a "requires restart" field.</li>
|
||||||
</ol>
|
</ol>
|
||||||
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
|
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
|
||||||
@@ -220,32 +224,160 @@ Inspector to act on it.</p>
|
|||||||
<figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups">
|
<figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups">
|
||||||
<figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure>
|
<figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure>
|
||||||
|
|
||||||
<h3>Apps behind a VPN container</h3>
|
</section>
|
||||||
<p>If an app shares a VPN container's network (<code>network_mode: container:vpn</code> or similar), set these on the app's page:</p>
|
|
||||||
|
<section id="vpn">
|
||||||
|
<h2>6. Apps behind a VPN container</h2>
|
||||||
|
<h3>Why these apps need special handling</h3>
|
||||||
|
<p>A common setup routes download and indexer apps through a VPN container. Those apps don't get a
|
||||||
|
network of their own: their compose file says <code>network_mode: container:<vpn></code> (or
|
||||||
|
<code>service:<vpn></code>) and all their traffic goes through the VPN container. That causes
|
||||||
|
two problems for a watchdog:</p>
|
||||||
|
<ul>
|
||||||
|
<li><b>When the VPN breaks, every app behind it breaks at the same time.</b> Restarting the apps
|
||||||
|
does nothing while the tunnel is down. Without help, Inspector would restart each app over and
|
||||||
|
over, use up their hourly restart limits, and flood you with alerts, all for a problem in a
|
||||||
|
different container.</li>
|
||||||
|
<li><b>When the VPN container is recreated</b> (an image update, or someone redeploys its stack),
|
||||||
|
it gets a new container ID. The apps still point at the old one, so a normal restart fails with
|
||||||
|
errors like <i>"No such container"</i> or <i>"joining network namespace"</i>. The only fix is
|
||||||
|
to recreate the apps too, which is what a Portainer stack redeploy does
|
||||||
|
(<a href="#portainer">section 7</a>).</li>
|
||||||
|
</ul>
|
||||||
|
<h3>How to tell if an app is behind a VPN</h3>
|
||||||
|
<ul>
|
||||||
|
<li>In Portainer open <b>Stacks → your stack → Editor</b> and look for
|
||||||
|
<code>network_mode: container:…</code> or <code>network_mode: service:…</code> on the app.</li>
|
||||||
|
<li>Or open <b>Containers → the app</b>: under <b>Network</b> it shows the VPN container instead
|
||||||
|
of a normal network, and it has no ports of its own (they're published on the VPN container).</li>
|
||||||
|
</ul>
|
||||||
|
<h3>Setting it up</h3>
|
||||||
<ol class="steps">
|
<ol class="steps">
|
||||||
<li><b>Dependencies → depends_on</b>: pick the VPN container, and turn on <b>vpn_routed</b>.
|
<li><b>Add the VPN container to Inspector and keep it enabled</b> (restart on UNHEALTHY and NOT
|
||||||
Inspector checks the VPN first and waits for it to recover before touching the app.</li>
|
RUNNING is a good start). Inspector can only wait for containers it is watching: if the VPN
|
||||||
<li><b>Recovery → recovery_action</b>: choose <b>Restart, then redeploy stack if that fails</b>
|
container is disabled, its apps are recovered normally, as if there were no dependency, and the
|
||||||
and enter the <b>stack name</b> exactly as Portainer shows it. When the VPN container gets
|
container page shows a warning.</li>
|
||||||
recreated, a plain restart of the app fails; redeploying the stack reconnects it.</li>
|
<li>On each app behind it, set <b>Dependencies → depends_on</b> to the VPN container.</li>
|
||||||
|
<li>Turn on <b>vpn_routed</b>. This only changes the log: Inspector writes down when it holds off
|
||||||
|
on the app because the VPN is down. The waiting itself comes from depends_on.</li>
|
||||||
|
<li>Add an <b>HTTP check</b> (<a href="#containers">section 5</a>) using your server's IP and the
|
||||||
|
port published on the VPN container, and tick <b>UNREACHABLE</b>. Docker often still reports
|
||||||
|
these apps as healthy when the tunnel is down; the HTTP check notices.</li>
|
||||||
|
<li>Under <b>Recovery</b>, choose <b>Restart, then redeploy stack if that fails</b> and enter the
|
||||||
|
stack name. This needs the Portainer connection from <a href="#portainer">section 7</a>.</li>
|
||||||
</ol>
|
</ol>
|
||||||
<p>Stack redeploys need the Portainer connection in <b>Settings → Portainer</b>:</p>
|
<h3>What Inspector then does</h3>
|
||||||
|
<p>Every check, it looks at the VPN container first. While the VPN is anything other than healthy
|
||||||
|
(or starting, or running without a health check), the apps that depend on it are not restarted;
|
||||||
|
Inspector fixes the VPN first. Once the VPN is healthy again, the apps are handled normally: a
|
||||||
|
restart first, and a stack redeploy if the restart fails or doesn't help.</p>
|
||||||
|
<p class="note"><b>depends_on isn't only for VPNs.</b> Use it for any "B needs A" pair, such as an
|
||||||
|
app and its database: Inspector won't restart the app while its database is down.</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id="portainer">
|
||||||
|
<h2>7. Portainer stack redeploy</h2>
|
||||||
|
<h3>Why Inspector needs Portainer</h3>
|
||||||
|
<p>Inspector talks to Docker directly, so it can restart or start a container. It can't
|
||||||
|
<i>recreate</i> one: that needs the container's compose definition, which lives in Portainer. A
|
||||||
|
stack redeploy asks Portainer to do exactly what its <b>Update the stack</b> button does. That's
|
||||||
|
how VPN-routed apps get reattached to a recreated VPN container. You only need this if you use one
|
||||||
|
of the two "redeploy" recovery actions.</p>
|
||||||
|
<p class="note warn"><b>A redeploy affects the whole stack.</b> Portainer pulls the latest images
|
||||||
|
and recreates <i>every</i> container in that stack, not just the failing app, so all of them blink
|
||||||
|
off for a moment. That's why <b>Restart, then redeploy stack if that fails</b> is the recommended
|
||||||
|
choice: most problems are fixed by the plain restart, and the redeploy is only used when that
|
||||||
|
doesn't work.</p>
|
||||||
|
<h3>Requirements</h3>
|
||||||
|
<ul>
|
||||||
|
<li>The app must belong to a stack that was <b>created in Portainer</b>. In <b>Stacks</b>, stacks
|
||||||
|
created elsewhere (for example with <code>docker compose</code> on the command line) are marked
|
||||||
|
<b>Limited</b>, and Portainer can't redeploy them.</li>
|
||||||
|
<li>An access token for a Portainer user who can manage that stack. An administrator's token is
|
||||||
|
the simplest.</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h3>Step 1: find the values</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Setting</th><th>Where to find it</th></tr>
|
||||||
|
<tr><td>base_url</td><td>The address you open Portainer at, without anything after the port:
|
||||||
|
usually <code>http://<server-ip>:9000</code> or <code>https://<server-ip>:9443</code>.
|
||||||
|
Use the server's IP, not <code>localhost</code>: Inspector runs in its own container.</td></tr>
|
||||||
|
<tr><td>endpoint_id</td><td>On Portainer's <b>Home</b> page, click your environment (often
|
||||||
|
<b>local</b>). The browser address then ends like <code>#!/2/docker/dashboard</code>: the
|
||||||
|
number after <code>#!/</code> is the endpoint ID (here <code>2</code>).</td></tr>
|
||||||
|
<tr><td>stack name</td><td><b>Stacks</b> list, <b>Name</b> column, spelled exactly the same.
|
||||||
|
This goes on each app's container page, not on the Portainer page.</td></tr>
|
||||||
|
<tr><td>verify_ssl</td><td>Leave off for <code>http://</code> and for Portainer's built-in
|
||||||
|
self-signed certificate on 9443. Turn on only if Portainer has a real certificate.</td></tr>
|
||||||
|
<tr><td>API token</td><td>Click your username (top right) → <b>My account</b> →
|
||||||
|
<b>Access tokens</b> → <b>Add access token</b>. Name it <code>inspector</code> and copy the
|
||||||
|
token straight away: Portainer shows it only once.</td></tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>Step 2: give Inspector the token</h3>
|
||||||
|
<p><b>Recommended: an environment variable.</b> The token then never lands in
|
||||||
|
<code>config.yml</code> or its backups.</p>
|
||||||
<ol class="steps">
|
<ol class="steps">
|
||||||
<li>In Portainer, open <b>My account → Access tokens</b> and create a token.</li>
|
<li>In Portainer open <b>Stacks → inspector → Editor</b>.</li>
|
||||||
<li>Easiest and safest: add it to the Inspector stack as an environment variable (for example
|
<li>Check that the compose file has this line under <code>environment:</code> (it's in the
|
||||||
<code>PORTAINER_API_TOKEN</code>) and redeploy Inspector. Or choose <b>Saved in config.yml</b>
|
compose file from version 0.9.1 on; add it if your stack is older):
|
||||||
and paste it in.</li>
|
<code>- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}</code></li>
|
||||||
<li>In <b>Settings → Portainer</b>, add an environment with Portainer's URL and endpoint ID
|
<li>Under <b>Environment variables</b> click <b>Add an environment variable</b>: name
|
||||||
(usually 1 or 2; it's the number in the URL when you open the environment in Portainer).
|
<code>PORTAINER_API_TOKEN</code>, value the token.</li>
|
||||||
Click <b>Test connection</b>: it should list your stacks.</li>
|
<li>Click <b>Update the stack</b>. Inspector restarts with the token available.</li>
|
||||||
<li>Turn Portainer on and save.</li>
|
</ol>
|
||||||
|
<p><b>Alternative: save it in Inspector.</b> In the next step choose <b>Saved in config.yml</b>
|
||||||
|
and paste the token. It is masked in the dashboard, but anyone who can read Inspector's config
|
||||||
|
folder (or a backup of it) can read the token.</p>
|
||||||
|
|
||||||
|
<h3>Step 3: connect Inspector</h3>
|
||||||
|
<ol class="steps">
|
||||||
|
<li>Open <b>Settings → Portainer</b>. Under <b>Add environment</b>, give it a name (your own
|
||||||
|
label, e.g. <code>home</code>) and fill in base_url, endpoint_id and verify_ssl.</li>
|
||||||
|
<li><b>API token stored as</b>: <i>Environment variable</i> with the name
|
||||||
|
<code>PORTAINER_API_TOKEN</code>, or <i>Saved in config.yml</i> with the token.</li>
|
||||||
|
<li>Click <b>Test connection</b>. It should list the stacks on that endpoint; check that your
|
||||||
|
app's stack is there. Then <b>Save new environment</b>.</li>
|
||||||
|
<li>At the top of the page turn <b>enabled</b> on, pick the <b>default_environment</b>, and
|
||||||
|
<b>Save Portainer</b>.</li>
|
||||||
|
<li>On each app's container page, set the recovery action and stack name
|
||||||
|
(<a href="#vpn">section 6</a>). Leave <b>portainer_environment</b> on <i>Default</i> unless
|
||||||
|
you connected more than one Portainer.</li>
|
||||||
</ol>
|
</ol>
|
||||||
<figure><img src="settings-portainer.png" alt="Portainer settings">
|
<figure><img src="settings-portainer.png" alt="Portainer settings">
|
||||||
<figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure>
|
<figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure>
|
||||||
|
|
||||||
|
<h3>The other settings on the page</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Setting</th><th>What it does</th></tr>
|
||||||
|
<tr><td>default_environment</td><td>Which connection apps use when they don't pick one. With a
|
||||||
|
single environment you can leave it on Automatic.</td></tr>
|
||||||
|
<tr><td>stack_redeploy_cooldown_seconds</td><td>Minimum time between two redeploys of the same
|
||||||
|
stack (default 300). Several apps often share a stack; this stops them triggering a redeploy
|
||||||
|
each.</td></tr>
|
||||||
|
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector gives the stack to come up after
|
||||||
|
a redeploy before judging that app again (default 90). Everything else keeps being checked
|
||||||
|
meanwhile.</td></tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>If Test connection fails</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Message</th><th>Fix</th></tr>
|
||||||
|
<tr><td>environment variable … is not set</td><td>The compose line from step 2 is missing, or
|
||||||
|
the stack wasn't updated after adding the variable.</td></tr>
|
||||||
|
<tr><td>the API token was rejected</td><td>Token copied wrong, deleted, or from a user without
|
||||||
|
access. Create a new one.</td></tr>
|
||||||
|
<tr><td>no stacks found on endpoint N</td><td>Wrong endpoint ID; check the number in the
|
||||||
|
Portainer address again.</td></tr>
|
||||||
|
<tr><td>request failed …</td><td>Wrong address or port, <code>http</code> vs <code>https</code>
|
||||||
|
mixed up, or <code>localhost</code> used instead of the server IP. With https and a
|
||||||
|
self-signed certificate, turn verify_ssl off.</td></tr>
|
||||||
|
</table>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="events">
|
<section id="events">
|
||||||
<h2>6. Events history</h2>
|
<h2>8. Events history</h2>
|
||||||
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
|
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
|
||||||
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
||||||
<figure><img src="events.png" alt="Events timeline">
|
<figure><img src="events.png" alt="Events timeline">
|
||||||
@@ -253,7 +385,7 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
|||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="settings">
|
<section id="settings">
|
||||||
<h2>7. Other settings</h2>
|
<h2>9. Other settings</h2>
|
||||||
<figure><img src="settings-inspector.png" alt="Inspector loop settings">
|
<figure><img src="settings-inspector.png" alt="Inspector loop settings">
|
||||||
<figcaption>Settings → Inspector loop.</figcaption></figure>
|
<figcaption>Settings → Inspector loop.</figcaption></figure>
|
||||||
<table>
|
<table>
|
||||||
@@ -263,17 +395,33 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
|||||||
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
|
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
|
||||||
<tr><td>Event history</td><td>How many days of events to keep.</td></tr>
|
<tr><td>Event history</td><td>How many days of events to keep.</td></tr>
|
||||||
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
|
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
|
||||||
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
|
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00, in your <a href="#settings">time zone</a>) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
|
||||||
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
|
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
|
||||||
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
|
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
|
||||||
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#containers">Apps behind a VPN container</a>.</td></tr>
|
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#portainer">section 7</a>.</td></tr>
|
||||||
</table>
|
</table>
|
||||||
<p>At the bottom of every settings page is a strip showing where the config file lives and
|
<p>At the bottom of every settings page is a strip showing where the config file lives,
|
||||||
whether Inspector can write to it.</p>
|
whether Inspector can write to it, and which time zone it is using.</p>
|
||||||
|
|
||||||
|
<h3>Time zone</h3>
|
||||||
|
<p>Inspector uses the <code>TZ</code> setting of its Portainer stack for every time it shows or acts on:</p>
|
||||||
|
<ul>
|
||||||
|
<li><b>The maintenance window runs in this zone.</b> A 02:00–05:00 window with the wrong zone
|
||||||
|
runs hours off, so Inspector could restart containers during your backups and ignore
|
||||||
|
problems at other times.</li>
|
||||||
|
<li>Alert times (with the zone, e.g. <code>03:15 AM CET</code>), dashboard times, the Events
|
||||||
|
page's days (midnight to midnight), and logs.</li>
|
||||||
|
</ul>
|
||||||
|
<p>To change it: in Portainer open <b>Stacks → inspector → Editor</b>, set the <code>TZ</code>
|
||||||
|
environment variable to your zone's name, e.g. <code>America/New_York</code>,
|
||||||
|
<code>Europe/London</code> or <code>Australia/Sydney</code> (names from the
|
||||||
|
<a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">list of time zones</a>;
|
||||||
|
abbreviations like <code>EST</code> don't work), then click <b>Update the stack</b>. Without
|
||||||
|
<code>TZ</code>, Inspector uses UTC.</p>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="restart">
|
<section id="restart">
|
||||||
<h2>8. "Requires restart" fields</h2>
|
<h2>10. "Requires restart" fields</h2>
|
||||||
<p>Most changes apply on the next check. Fields marked
|
<p>Most changes apply on the next check. Fields marked
|
||||||
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
|
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
|
||||||
dashboard on or off) only take effect after Inspector restarts:</p>
|
dashboard on or off) only take effect after Inspector restarts:</p>
|
||||||
@@ -285,7 +433,7 @@ dashboard on or off) only take effect after Inspector restarts:</p>
|
|||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="trouble">
|
<section id="trouble">
|
||||||
<h2>9. Troubleshooting</h2>
|
<h2>11. Troubleshooting</h2>
|
||||||
<table>
|
<table>
|
||||||
<tr><th>Problem</th><th>Fix</th></tr>
|
<tr><th>Problem</th><th>Fix</th></tr>
|
||||||
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
|
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
|
||||||
|
|||||||
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB After Width: | Height: | Size: 42 KiB |
+34
-4
@@ -1,10 +1,12 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import io
|
import io
|
||||||
|
import logging
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import threading
|
import threading
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import yaml
|
import yaml
|
||||||
@@ -23,6 +25,30 @@ VALID_STATUSES = {
|
|||||||
"UNREACHABLE",
|
"UNREACHABLE",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
log = logging.getLogger("inspector.config")
|
||||||
|
|
||||||
|
# Names logging.setLevel() accepts; anything else would crash at startup.
|
||||||
|
VALID_LOG_LEVELS = {"CRITICAL", "FATAL", "ERROR", "WARN", "WARNING", "INFO", "DEBUG", "NOTSET"}
|
||||||
|
|
||||||
|
|
||||||
|
def valid_hhmm(value: str) -> bool:
|
||||||
|
"""True if `value` parses the way the maintenance tracker parses it ("HH:MM")."""
|
||||||
|
try:
|
||||||
|
hour, minute = str(value).split(":")
|
||||||
|
time(int(hour), int(minute))
|
||||||
|
return True
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _startup_safe(value, default, ok: bool, key: str):
|
||||||
|
"""Return `value`, or `default` with a warning when it would crash Inspector."""
|
||||||
|
if ok:
|
||||||
|
return value
|
||||||
|
log.warning("config: invalid %s %r; using %r instead", key, value, default)
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
VALID_RECOVERY_ACTIONS = {
|
VALID_RECOVERY_ACTIONS = {
|
||||||
"restart_container",
|
"restart_container",
|
||||||
"start_container",
|
"start_container",
|
||||||
@@ -190,9 +216,11 @@ def build_config_from_raw(raw: dict) -> AppConfig:
|
|||||||
to validate an edited raw dict before writing it back to disk.
|
to validate an edited raw dict before writing it back to disk.
|
||||||
"""
|
"""
|
||||||
insp_raw = raw.get("inspector", {}) or {}
|
insp_raw = raw.get("inspector", {}) or {}
|
||||||
|
interval = int(insp_raw.get("interval_seconds", 60))
|
||||||
|
log_level = str(insp_raw.get("log_level", "INFO")).upper()
|
||||||
inspector = InspectorConfig(
|
inspector = InspectorConfig(
|
||||||
interval_seconds=int(insp_raw.get("interval_seconds", 60)),
|
interval_seconds=_startup_safe(interval, 1, interval >= 1, "inspector.interval_seconds"),
|
||||||
log_level=str(insp_raw.get("log_level", "INFO")).upper(),
|
log_level=_startup_safe(log_level, "INFO", log_level in VALID_LOG_LEVELS, "inspector.log_level"),
|
||||||
dry_run=bool(insp_raw.get("dry_run", False)),
|
dry_run=bool(insp_raw.get("dry_run", False)),
|
||||||
log_dir=str(insp_raw.get("log_dir", "/app/logs")),
|
log_dir=str(insp_raw.get("log_dir", "/app/logs")),
|
||||||
log_filename=str(insp_raw.get("log_filename", "inspector.log")),
|
log_filename=str(insp_raw.get("log_filename", "inspector.log")),
|
||||||
@@ -237,10 +265,12 @@ def build_config_from_raw(raw: dict) -> AppConfig:
|
|||||||
)
|
)
|
||||||
|
|
||||||
maint_raw = raw.get("maintenance") or {}
|
maint_raw = raw.get("maintenance") or {}
|
||||||
|
maint_start = str(maint_raw.get("start", "00:00"))
|
||||||
|
maint_end = str(maint_raw.get("end", "06:00"))
|
||||||
maintenance = MaintenanceConfig(
|
maintenance = MaintenanceConfig(
|
||||||
enabled=bool(maint_raw.get("enabled", False)),
|
enabled=bool(maint_raw.get("enabled", False)),
|
||||||
start=str(maint_raw.get("start", "00:00")),
|
start=_startup_safe(maint_start, "00:00", valid_hhmm(maint_start), "maintenance.start"),
|
||||||
end=str(maint_raw.get("end", "06:00")),
|
end=_startup_safe(maint_end, "06:00", valid_hhmm(maint_end), "maintenance.end"),
|
||||||
dir=str(maint_raw.get("dir", "/app/data/maintenance")),
|
dir=str(maint_raw.get("dir", "/app/data/maintenance")),
|
||||||
max_hours=int(maint_raw.get("max_hours", 8)),
|
max_hours=int(maint_raw.get("max_hours", 8)),
|
||||||
)
|
)
|
||||||
|
|||||||
+224
-69
@@ -1,5 +1,7 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import functools
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
@@ -21,12 +23,15 @@ from .config import (
|
|||||||
AppConfig,
|
AppConfig,
|
||||||
ConfigRef,
|
ConfigRef,
|
||||||
ContainerConfig,
|
ContainerConfig,
|
||||||
|
VALID_LOG_LEVELS,
|
||||||
HttpCheckConfig,
|
HttpCheckConfig,
|
||||||
UptimeKumaMonitorConfig,
|
UptimeKumaMonitorConfig,
|
||||||
build_config_from_raw,
|
build_config_from_raw,
|
||||||
|
valid_hhmm,
|
||||||
write_config_raw,
|
write_config_raw,
|
||||||
)
|
)
|
||||||
from .formatting import PACIFIC, format_timestamp
|
from . import formatting
|
||||||
|
from .formatting import format_timestamp
|
||||||
from .health import CONTAINER_NOT_FOUND, ERROR, HEALTHY, NOT_RUNNING, UNHEALTHY, UNREACHABLE
|
from .health import CONTAINER_NOT_FOUND, ERROR, HEALTHY, NOT_RUNNING, UNHEALTHY, UNREACHABLE
|
||||||
from .maintenance import MaintenanceTracker
|
from .maintenance import MaintenanceTracker
|
||||||
from .notification_mute import NotificationMuteTracker
|
from .notification_mute import NotificationMuteTracker
|
||||||
@@ -36,6 +41,9 @@ from .state import ContainerState, utcnow
|
|||||||
log = logging.getLogger("inspector.dashboard")
|
log = logging.getLogger("inspector.dashboard")
|
||||||
|
|
||||||
_SESSION_MAX_AGE = 86400 # seconds a login cookie stays valid
|
_SESSION_MAX_AGE = 86400 # seconds a login cookie stays valid
|
||||||
|
_MAX_BODY_BYTES = 1024 * 1024 # largest request body accepted
|
||||||
|
_LOGIN_MAX_FAILURES = 5 # failed logins per client before it is throttled
|
||||||
|
_LOGIN_FAILURE_WINDOW = 300 # seconds a failed login counts toward the limit
|
||||||
_UNCHANGED_SECRET_PLACEHOLDER = "__UNCHANGED__" # UI sentinel: keep existing secret value
|
_UNCHANGED_SECRET_PLACEHOLDER = "__UNCHANGED__" # UI sentinel: keep existing secret value
|
||||||
|
|
||||||
# Config keys that require an Inspector container restart to take effect.
|
# Config keys that require an Inspector container restart to take effect.
|
||||||
@@ -84,7 +92,13 @@ def _color_map(
|
|||||||
states: dict[str, ContainerState],
|
states: dict[str, ContainerState],
|
||||||
containers: dict[str, ContainerConfig],
|
containers: dict[str, ContainerConfig],
|
||||||
) -> dict[str, str]:
|
) -> dict[str, str]:
|
||||||
return {name: _resolve_color(containers[name], i) for i, name in enumerate(states)}
|
# list() snapshot + skip: the inspector loop adds/removes states from another
|
||||||
|
# thread, and a container removed in the UI keeps its state until the next check.
|
||||||
|
return {
|
||||||
|
name: _resolve_color(containers[name], i)
|
||||||
|
for i, name in enumerate(list(states))
|
||||||
|
if name in containers
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _compute_phase(
|
def _compute_phase(
|
||||||
@@ -167,20 +181,20 @@ def _status_snapshot(
|
|||||||
|
|
||||||
|
|
||||||
def _parse_date_param(value: str | None) -> date:
|
def _parse_date_param(value: str | None) -> date:
|
||||||
"""Parse a ?date=YYYY-MM-DD query param; falls back to "today" in
|
"""Parse a ?date=YYYY-MM-DD query param; falls back to "today" in the
|
||||||
America/Los_Angeles on missing/invalid input."""
|
configured time zone on missing/invalid input."""
|
||||||
if value:
|
if value:
|
||||||
try:
|
try:
|
||||||
return date.fromisoformat(value)
|
return date.fromisoformat(value)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
pass
|
pass
|
||||||
return datetime.now(PACIFIC).date()
|
return datetime.now(formatting.LOCAL_TZ).date()
|
||||||
|
|
||||||
|
|
||||||
def _day_bounds_utc(day: date) -> tuple[datetime, datetime]:
|
def _day_bounds_utc(day: date) -> tuple[datetime, datetime]:
|
||||||
"""UTC [start, end) bounds for a Pacific calendar day, extended one hour
|
"""UTC [start, end) bounds for a local calendar day, extended one hour
|
||||||
into the previous day for the lookback-overlap."""
|
into the previous day for the lookback-overlap."""
|
||||||
midnight = datetime.combine(day, datetime.min.time(), tzinfo=PACIFIC)
|
midnight = datetime.combine(day, datetime.min.time(), tzinfo=formatting.LOCAL_TZ)
|
||||||
start = (midnight - timedelta(hours=1)).astimezone(timezone.utc)
|
start = (midnight - timedelta(hours=1)).astimezone(timezone.utc)
|
||||||
end = (midnight + timedelta(days=1)).astimezone(timezone.utc)
|
end = (midnight + timedelta(days=1)).astimezone(timezone.utc)
|
||||||
return start, end
|
return start, end
|
||||||
@@ -205,8 +219,8 @@ def _read_events_range(db_path: str, start: datetime, end: datetime) -> list[dic
|
|||||||
|
|
||||||
|
|
||||||
def _hour_markers(day: date) -> list[datetime]:
|
def _hour_markers(day: date) -> list[datetime]:
|
||||||
"""24 Pacific-local hour boundaries for `day`, 00:00..23:00 ascending."""
|
"""24 local hour boundaries for `day`, 00:00..23:00 ascending."""
|
||||||
midnight = datetime.combine(day, datetime.min.time(), tzinfo=PACIFIC)
|
midnight = datetime.combine(day, datetime.min.time(), tzinfo=formatting.LOCAL_TZ)
|
||||||
return [midnight + timedelta(hours=h) for h in range(24)]
|
return [midnight + timedelta(hours=h) for h in range(24)]
|
||||||
|
|
||||||
|
|
||||||
@@ -226,7 +240,7 @@ def _group_events_by_hour(
|
|||||||
|
|
||||||
for ev in events:
|
for ev in events:
|
||||||
try:
|
try:
|
||||||
ts = datetime.fromisoformat(ev["ts"]).astimezone(PACIFIC)
|
ts = datetime.fromisoformat(ev["ts"]).astimezone(formatting.LOCAL_TZ)
|
||||||
except (ValueError, KeyError):
|
except (ValueError, KeyError):
|
||||||
continue
|
continue
|
||||||
if prev_hour_start <= ts < markers[0]:
|
if prev_hour_start <= ts < markers[0]:
|
||||||
@@ -246,9 +260,31 @@ def _escape(value: object) -> str:
|
|||||||
.replace("<", "<")
|
.replace("<", "<")
|
||||||
.replace(">", ">")
|
.replace(">", ">")
|
||||||
.replace('"', """)
|
.replace('"', """)
|
||||||
|
.replace("'", "'")
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _js_string_attr(text: str) -> str:
|
||||||
|
"""A JS string literal safe to put inside a double-quoted HTML attribute."""
|
||||||
|
return _escape(json.dumps(text))
|
||||||
|
|
||||||
|
|
||||||
|
def _creds_match(username: str, password: str, cfg) -> bool:
|
||||||
|
"""Constant-time credential check. Never accepts an empty configured
|
||||||
|
password, and works for non-ASCII input (compare_digest on str doesn't)."""
|
||||||
|
if not cfg.password:
|
||||||
|
return False
|
||||||
|
user_ok = secrets.compare_digest(username.encode("utf-8"), cfg.username.encode("utf-8"))
|
||||||
|
pass_ok = secrets.compare_digest(password.encode("utf-8"), cfg.password.encode("utf-8"))
|
||||||
|
return user_ok and pass_ok
|
||||||
|
|
||||||
|
|
||||||
|
def _cred_fingerprint(cfg) -> str:
|
||||||
|
"""Changes whenever the dashboard username or password changes, which
|
||||||
|
invalidates every session created with the old credentials."""
|
||||||
|
return hashlib.sha256(f"{cfg.username}\0{cfg.password}".encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
_STATUS_BADGE_CLASS = {
|
_STATUS_BADGE_CLASS = {
|
||||||
HEALTHY: "healthy",
|
HEALTHY: "healthy",
|
||||||
UNHEALTHY: "unhealthy",
|
UNHEALTHY: "unhealthy",
|
||||||
@@ -1076,12 +1112,15 @@ def _render_events_page(
|
|||||||
|
|
||||||
prev_day = (selected_day - timedelta(days=1)).isoformat()
|
prev_day = (selected_day - timedelta(days=1)).isoformat()
|
||||||
next_day = (selected_day + timedelta(days=1)).isoformat()
|
next_day = (selected_day + timedelta(days=1)).isoformat()
|
||||||
today = datetime.now(PACIFIC).date()
|
today = datetime.now(formatting.LOCAL_TZ).date()
|
||||||
day_label = selected_day.strftime("%A, %B %-d, %Y")
|
day_label = selected_day.strftime("%A, %B %-d, %Y")
|
||||||
|
|
||||||
parts.append('<div class="events-nav">\n')
|
parts.append('<div class="events-nav">\n')
|
||||||
parts.append(f'<a class="day-nav" href="/events?date={prev_day}">← Prev day</a>\n')
|
parts.append(f'<a class="day-nav" href="/events?date={prev_day}">← Prev day</a>\n')
|
||||||
parts.append(f'<span class="day-label">{_escape(day_label)}</span>\n')
|
parts.append(
|
||||||
|
f'<span class="day-label">{_escape(day_label)}'
|
||||||
|
f' <span style="color:var(--muted); font-weight:400; font-size:0.8em;">· times in {_escape(formatting.LOCAL_TZ_NAME)}</span></span>\n'
|
||||||
|
)
|
||||||
if selected_day >= today:
|
if selected_day >= today:
|
||||||
parts.append('<span class="day-nav disabled">Next day →</span>\n')
|
parts.append('<span class="day-nav disabled">Next day →</span>\n')
|
||||||
else:
|
else:
|
||||||
@@ -1146,10 +1185,13 @@ def _render_event_item(ev: dict, container_colors: dict[str, str] | None) -> str
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _render_login_page(error: bool = False) -> bytes:
|
def _render_login_page(error: bool = False, throttled: bool = False) -> bytes:
|
||||||
error_html = (
|
if throttled:
|
||||||
'<div class="login-error">Incorrect username or password.</div>\n' if error else ""
|
error_html = '<div class="login-error">Too many failed sign-ins. Try again in a few minutes.</div>\n'
|
||||||
)
|
elif error:
|
||||||
|
error_html = '<div class="login-error">Incorrect username or password.</div>\n'
|
||||||
|
else:
|
||||||
|
error_html = ""
|
||||||
head = (
|
head = (
|
||||||
"<!DOCTYPE html>\n"
|
"<!DOCTYPE html>\n"
|
||||||
'<html lang="en">\n<head>\n'
|
'<html lang="en">\n<head>\n'
|
||||||
@@ -1243,6 +1285,14 @@ def _form_float(form: dict, key: str, default: float = 0.0) -> float:
|
|||||||
return default
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def _int_in(form: dict, key: str, default: int, lo: int, hi: int) -> int:
|
||||||
|
"""Read an int form field and reject values outside lo..hi."""
|
||||||
|
value = _form_int(form, key, default)
|
||||||
|
if not lo <= value <= hi:
|
||||||
|
raise _SettingsError(f"{key} must be between {lo} and {hi} (got {value})")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
def _set_or_drop(section: dict, key: str, value) -> None:
|
def _set_or_drop(section: dict, key: str, value) -> None:
|
||||||
"""Store value, or remove the key when it is blank/False so config.yml
|
"""Store value, or remove the key when it is blank/False so config.yml
|
||||||
doesn't fill up with empty defaults."""
|
doesn't fill up with empty defaults."""
|
||||||
@@ -1270,8 +1320,10 @@ def _apply_container_advanced(raw: dict, name: str, cfg: dict, form: dict) -> No
|
|||||||
hc["enabled"] = hc_enabled
|
hc["enabled"] = hc_enabled
|
||||||
hc["url"] = hc_url
|
hc["url"] = hc_url
|
||||||
timeout = _form_float(form, "http_check_timeout_seconds", 5.0)
|
timeout = _form_float(form, "http_check_timeout_seconds", 5.0)
|
||||||
|
if not 0.5 <= timeout <= 120:
|
||||||
|
raise _SettingsError(f"HTTP check timeout must be between 0.5 and 120 seconds (got {timeout:g})")
|
||||||
hc["timeout_seconds"] = int(timeout) if timeout.is_integer() else timeout
|
hc["timeout_seconds"] = int(timeout) if timeout.is_integer() else timeout
|
||||||
hc["expected_status"] = _form_int(form, "http_check_expected_status", 200)
|
hc["expected_status"] = _int_in(form, "http_check_expected_status", 200, 100, 599)
|
||||||
cfg["http_check"] = hc
|
cfg["http_check"] = hc
|
||||||
else:
|
else:
|
||||||
cfg.pop("http_check", None)
|
cfg.pop("http_check", None)
|
||||||
@@ -1314,6 +1366,15 @@ def _apply_container_advanced(raw: dict, name: str, cfg: dict, form: dict) -> No
|
|||||||
cfg.pop("portainer_stack_name", None)
|
cfg.pop("portainer_stack_name", None)
|
||||||
|
|
||||||
|
|
||||||
|
_PORTAINER_ENV_NAME_RE = re.compile(r"^PORTAINER_[A-Z0-9_]+$")
|
||||||
|
|
||||||
|
|
||||||
|
def _token_env_allowed(name: str, saved: str = "") -> bool:
|
||||||
|
"""Only PORTAINER_* variables (or the name already in config.yml) may be read
|
||||||
|
as a token, so the dashboard can't be used to send other container secrets out."""
|
||||||
|
return bool(_PORTAINER_ENV_NAME_RE.match(name)) or (bool(saved) and name == saved)
|
||||||
|
|
||||||
|
|
||||||
def _apply_portainer_env(raw: dict, form: dict) -> None:
|
def _apply_portainer_env(raw: dict, form: dict) -> None:
|
||||||
"""Add, edit, rename or delete one integrations.portainer.environments entry."""
|
"""Add, edit, rename or delete one integrations.portainer.environments entry."""
|
||||||
pt = raw.setdefault("integrations", {}).setdefault("portainer", {})
|
pt = raw.setdefault("integrations", {}).setdefault("portainer", {})
|
||||||
@@ -1350,7 +1411,7 @@ def _apply_portainer_env(raw: dict, form: dict) -> None:
|
|||||||
|
|
||||||
env = envs.get(original) or {}
|
env = envs.get(original) or {}
|
||||||
env["base_url"] = base_url
|
env["base_url"] = base_url
|
||||||
env["endpoint_id"] = _form_int(form, "endpoint_id", 1)
|
env["endpoint_id"] = _int_in(form, "endpoint_id", 1, 1, 1_000_000)
|
||||||
env["verify_ssl"] = _form_bool(form, "verify_ssl")
|
env["verify_ssl"] = _form_bool(form, "verify_ssl")
|
||||||
if _form_str(form, "token_source", "env") == "inline":
|
if _form_str(form, "token_source", "env") == "inline":
|
||||||
_apply_secret(form, env, "api_token", "api_token")
|
_apply_secret(form, env, "api_token", "api_token")
|
||||||
@@ -1361,6 +1422,8 @@ def _apply_portainer_env(raw: dict, form: dict) -> None:
|
|||||||
token_env = _form_str(form, "api_token_env", "").strip()
|
token_env = _form_str(form, "api_token_env", "").strip()
|
||||||
if not token_env:
|
if not token_env:
|
||||||
raise _SettingsError("enter the name of the environment variable holding the API token")
|
raise _SettingsError("enter the name of the environment variable holding the API token")
|
||||||
|
if not _token_env_allowed(token_env, env.get("api_token_env", "")):
|
||||||
|
raise _SettingsError("the environment variable name must start with PORTAINER_ (e.g. PORTAINER_API_TOKEN)")
|
||||||
env["api_token_env"] = token_env
|
env["api_token_env"] = token_env
|
||||||
env.pop("api_token", None)
|
env.pop("api_token", None)
|
||||||
|
|
||||||
@@ -1376,57 +1439,81 @@ def _apply_portainer_env(raw: dict, form: dict) -> None:
|
|||||||
pt["environments"] = envs
|
pt["environments"] = envs
|
||||||
|
|
||||||
|
|
||||||
|
_SAVE_LOCK = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def _serialized(fn):
|
||||||
|
"""Run config read-modify-write functions one at a time, so two saves at
|
||||||
|
the same moment can't overwrite each other's changes."""
|
||||||
|
@functools.wraps(fn)
|
||||||
|
def wrapper(*args, **kwargs):
|
||||||
|
with _SAVE_LOCK:
|
||||||
|
return fn(*args, **kwargs)
|
||||||
|
return wrapper
|
||||||
|
|
||||||
|
|
||||||
|
@_serialized
|
||||||
def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> None:
|
def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> None:
|
||||||
"""Validate the form, merge into the raw config, write atomically, swap ref."""
|
"""Validate the form, merge into the raw config, write atomically, swap ref."""
|
||||||
raw = config_ref.get_raw()
|
raw = config_ref.get_raw()
|
||||||
|
|
||||||
if section == "inspector":
|
if section == "inspector":
|
||||||
insp = raw.setdefault("inspector", {})
|
insp = raw.setdefault("inspector", {})
|
||||||
insp["interval_seconds"] = _form_int(form, "interval_seconds", 60)
|
insp["interval_seconds"] = _int_in(form, "interval_seconds", 60, 1, 86400)
|
||||||
insp["log_level"] = _form_str(form, "log_level", "INFO").upper()
|
level = _form_str(form, "log_level", "INFO").strip().upper()
|
||||||
|
if level not in VALID_LOG_LEVELS:
|
||||||
|
raise _SettingsError("log_level must be one of DEBUG, INFO, WARNING, ERROR, CRITICAL")
|
||||||
|
insp["log_level"] = level
|
||||||
insp["dry_run"] = _form_bool(form, "dry_run")
|
insp["dry_run"] = _form_bool(form, "dry_run")
|
||||||
insp["heartbeat"] = _form_bool(form, "heartbeat")
|
insp["heartbeat"] = _form_bool(form, "heartbeat")
|
||||||
insp["log_retention_days"] = _form_int(form, "log_retention_days", 30)
|
insp["log_retention_days"] = _int_in(form, "log_retention_days", 30, 1, 3650)
|
||||||
elif section == "gotify":
|
elif section == "gotify":
|
||||||
notif = raw.setdefault("notifications", {})
|
notif = raw.setdefault("notifications", {})
|
||||||
gotify = notif.setdefault("gotify", {})
|
gotify = notif.setdefault("gotify", {})
|
||||||
gotify["enabled"] = _form_bool(form, "enabled")
|
gotify["enabled"] = _form_bool(form, "enabled")
|
||||||
gotify["url"] = _form_str(form, "url", "")
|
gotify["url"] = _form_str(form, "url", "")
|
||||||
_apply_secret(form, gotify, "token", "token")
|
_apply_secret(form, gotify, "token", "token")
|
||||||
gotify["priority"] = _form_int(form, "priority", 5)
|
gotify["priority"] = _int_in(form, "priority", 5, 0, 10)
|
||||||
elif section == "pushover":
|
elif section == "pushover":
|
||||||
notif = raw.setdefault("notifications", {})
|
notif = raw.setdefault("notifications", {})
|
||||||
pushover = notif.setdefault("pushover", {})
|
pushover = notif.setdefault("pushover", {})
|
||||||
pushover["enabled"] = _form_bool(form, "enabled")
|
pushover["enabled"] = _form_bool(form, "enabled")
|
||||||
_apply_secret(form, pushover, "token", "token")
|
_apply_secret(form, pushover, "token", "token")
|
||||||
_apply_secret(form, pushover, "user_key", "user_key")
|
_apply_secret(form, pushover, "user_key", "user_key")
|
||||||
pushover["priority"] = _form_int(form, "priority", 5)
|
pushover["priority"] = _int_in(form, "priority", 5, 0, 10)
|
||||||
pushover["device"] = _form_str(form, "device", "")
|
pushover["device"] = _form_str(form, "device", "")
|
||||||
pushover["sound"] = _form_str(form, "sound", "")
|
pushover["sound"] = _form_str(form, "sound", "")
|
||||||
elif section == "dashboard":
|
elif section == "dashboard":
|
||||||
dash = raw.setdefault("dashboard", {})
|
dash = raw.setdefault("dashboard", {})
|
||||||
dash["enabled"] = _form_bool(form, "enabled")
|
dash["enabled"] = _form_bool(form, "enabled")
|
||||||
dash["username"] = _form_str(form, "username", "admin")
|
dash["username"] = _form_str(form, "username", "admin").strip()
|
||||||
|
if not dash["username"]:
|
||||||
|
raise _SettingsError("the dashboard username can't be empty")
|
||||||
_apply_secret(form, dash, "password", "password")
|
_apply_secret(form, dash, "password", "password")
|
||||||
|
if not dash.get("password"):
|
||||||
|
raise _SettingsError("the dashboard password can't be empty")
|
||||||
elif section == "database":
|
elif section == "database":
|
||||||
db = raw.setdefault("database", {})
|
db = raw.setdefault("database", {})
|
||||||
db["enabled"] = _form_bool(form, "enabled")
|
db["enabled"] = _form_bool(form, "enabled")
|
||||||
db["retention_days"] = _form_int(form, "retention_days", 30)
|
db["retention_days"] = _int_in(form, "retention_days", 30, 1, 3650)
|
||||||
elif section == "health_summary":
|
elif section == "health_summary":
|
||||||
hs = raw.setdefault("health_summary", {})
|
hs = raw.setdefault("health_summary", {})
|
||||||
hs["enabled"] = _form_bool(form, "enabled")
|
hs["enabled"] = _form_bool(form, "enabled")
|
||||||
hs["interval_hours"] = _form_int(form, "interval_hours", 24)
|
hs["interval_hours"] = _int_in(form, "interval_hours", 24, 1, 8760)
|
||||||
hs["priority"] = _form_int(form, "priority", 3)
|
hs["priority"] = _int_in(form, "priority", 3, 0, 10)
|
||||||
elif section == "maintenance":
|
elif section == "maintenance":
|
||||||
m = raw.setdefault("maintenance", {})
|
m = raw.setdefault("maintenance", {})
|
||||||
m["enabled"] = _form_bool(form, "enabled")
|
m["enabled"] = _form_bool(form, "enabled")
|
||||||
m["start"] = _form_str(form, "start", "00:00")
|
for key, default in (("start", "00:00"), ("end", "06:00")):
|
||||||
m["end"] = _form_str(form, "end", "06:00")
|
value = _form_str(form, key, default).strip()
|
||||||
m["max_hours"] = _form_int(form, "max_hours", 8)
|
if not valid_hhmm(value):
|
||||||
|
raise _SettingsError(f"{key} must be a 24-hour time like 02:30 (got {value!r})")
|
||||||
|
m[key] = value
|
||||||
|
m["max_hours"] = _int_in(form, "max_hours", 8, 1, 720)
|
||||||
elif section == "notification_mute":
|
elif section == "notification_mute":
|
||||||
nm = raw.setdefault("notification_mute", {})
|
nm = raw.setdefault("notification_mute", {})
|
||||||
nm["enabled"] = _form_bool(form, "enabled")
|
nm["enabled"] = _form_bool(form, "enabled")
|
||||||
nm["default_hours"] = _form_int(form, "default_hours", 12)
|
nm["default_hours"] = _int_in(form, "default_hours", 12, 1, 720)
|
||||||
elif section.startswith("container:"):
|
elif section.startswith("container:"):
|
||||||
name = section.split(":", 1)[1]
|
name = section.split(":", 1)[1]
|
||||||
containers = raw.get("containers") or {}
|
containers = raw.get("containers") or {}
|
||||||
@@ -1434,10 +1521,10 @@ def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> N
|
|||||||
raise _SettingsError(f"unknown container {name!r}")
|
raise _SettingsError(f"unknown container {name!r}")
|
||||||
cfg = containers[name] or {}
|
cfg = containers[name] or {}
|
||||||
cfg["enabled"] = _form_bool(form, "enabled")
|
cfg["enabled"] = _form_bool(form, "enabled")
|
||||||
cfg["max_error_count"] = _form_int(form, "max_error_count", 5)
|
cfg["max_error_count"] = _int_in(form, "max_error_count", 5, 1, 1000)
|
||||||
cfg["cooldown_seconds"] = _form_int(form, "cooldown_seconds", 300)
|
cfg["cooldown_seconds"] = _int_in(form, "cooldown_seconds", 300, 0, 86400)
|
||||||
cfg["grace_period_seconds"] = _form_int(form, "grace_period_seconds", 120)
|
cfg["grace_period_seconds"] = _int_in(form, "grace_period_seconds", 120, 0, 86400)
|
||||||
cfg["max_restarts_per_hour"] = _form_int(form, "max_restarts_per_hour", 3)
|
cfg["max_restarts_per_hour"] = _int_in(form, "max_restarts_per_hour", 3, 1, 100)
|
||||||
_set_or_drop(cfg, "color", _form_str(form, "color", "").strip())
|
_set_or_drop(cfg, "color", _form_str(form, "color", "").strip())
|
||||||
if form.get("restart_on_present"): # only our own form carries this field
|
if form.get("restart_on_present"): # only our own form carries this field
|
||||||
_set_or_drop(cfg, "maintenance_window", _form_bool(form, "maintenance_window"))
|
_set_or_drop(cfg, "maintenance_window", _form_bool(form, "maintenance_window"))
|
||||||
@@ -1467,8 +1554,8 @@ def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> N
|
|||||||
if default_env and default_env not in (pt.get("environments") or {}):
|
if default_env and default_env not in (pt.get("environments") or {}):
|
||||||
raise _SettingsError(f"unknown Portainer environment {default_env!r}")
|
raise _SettingsError(f"unknown Portainer environment {default_env!r}")
|
||||||
_set_or_drop(pt, "default_environment", default_env)
|
_set_or_drop(pt, "default_environment", default_env)
|
||||||
pt["stack_redeploy_cooldown_seconds"] = _form_int(form, "stack_redeploy_cooldown_seconds", 300)
|
pt["stack_redeploy_cooldown_seconds"] = _int_in(form, "stack_redeploy_cooldown_seconds", 300, 0, 86400)
|
||||||
pt["post_redeploy_wait_seconds"] = _form_int(form, "post_redeploy_wait_seconds", 90)
|
pt["post_redeploy_wait_seconds"] = _int_in(form, "post_redeploy_wait_seconds", 90, 0, 3600)
|
||||||
elif section == "portainer_env":
|
elif section == "portainer_env":
|
||||||
_apply_portainer_env(raw, form)
|
_apply_portainer_env(raw, form)
|
||||||
else:
|
else:
|
||||||
@@ -1492,6 +1579,7 @@ def _apply_settings_update(config_ref: ConfigRef, section: str, form: dict) -> N
|
|||||||
_CONTAINER_NAME_RE = re.compile(r"^[a-zA-Z0-9][a-zA-Z0-9_-]{0,62}$")
|
_CONTAINER_NAME_RE = re.compile(r"^[a-zA-Z0-9][a-zA-Z0-9_-]{0,62}$")
|
||||||
|
|
||||||
|
|
||||||
|
@_serialized
|
||||||
def _apply_container_create(config_ref: ConfigRef, form: dict) -> str:
|
def _apply_container_create(config_ref: ConfigRef, form: dict) -> str:
|
||||||
"""Insert a new container under raw['containers']. Returns the new name."""
|
"""Insert a new container under raw['containers']. Returns the new name."""
|
||||||
name = _form_str(form, "name", "").strip()
|
name = _form_str(form, "name", "").strip()
|
||||||
@@ -1507,10 +1595,10 @@ def _apply_container_create(config_ref: ConfigRef, form: dict) -> str:
|
|||||||
|
|
||||||
cfg = {
|
cfg = {
|
||||||
"enabled": _form_bool(form, "enabled"),
|
"enabled": _form_bool(form, "enabled"),
|
||||||
"max_error_count": _form_int(form, "max_error_count", 5),
|
"max_error_count": _int_in(form, "max_error_count", 5, 1, 1000),
|
||||||
"cooldown_seconds": _form_int(form, "cooldown_seconds", 300),
|
"cooldown_seconds": _int_in(form, "cooldown_seconds", 300, 0, 86400),
|
||||||
"grace_period_seconds": _form_int(form, "grace_period_seconds", 120),
|
"grace_period_seconds": _int_in(form, "grace_period_seconds", 120, 0, 86400),
|
||||||
"max_restarts_per_hour": _form_int(form, "max_restarts_per_hour", 3),
|
"max_restarts_per_hour": _int_in(form, "max_restarts_per_hour", 3, 1, 100),
|
||||||
"color": _form_str(form, "color", ""),
|
"color": _form_str(form, "color", ""),
|
||||||
"maintenance_window": _form_bool(form, "maintenance_window"),
|
"maintenance_window": _form_bool(form, "maintenance_window"),
|
||||||
}
|
}
|
||||||
@@ -1536,6 +1624,7 @@ def _apply_container_create(config_ref: ConfigRef, form: dict) -> str:
|
|||||||
return name
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
@_serialized
|
||||||
def _apply_container_delete(config_ref: ConfigRef, name: str) -> None:
|
def _apply_container_delete(config_ref: ConfigRef, name: str) -> None:
|
||||||
raw = config_ref.get_raw()
|
raw = config_ref.get_raw()
|
||||||
containers = raw.get("containers") or {}
|
containers = raw.get("containers") or {}
|
||||||
@@ -1583,7 +1672,7 @@ def _run_test_notification(channel: str, form: dict, app: AppConfig) -> dict:
|
|||||||
try:
|
try:
|
||||||
resp = requests.post(
|
resp = requests.post(
|
||||||
f"{url}/message",
|
f"{url}/message",
|
||||||
params={"token": token},
|
headers={"X-Gotify-Key": token},
|
||||||
json={"title": title, "message": message, "priority": priority},
|
json={"title": title, "message": message, "priority": priority},
|
||||||
timeout=5.0,
|
timeout=5.0,
|
||||||
)
|
)
|
||||||
@@ -1665,6 +1754,9 @@ def _run_test_integration(kind: str, form: dict, app: AppConfig) -> dict:
|
|||||||
return {"ok": False, "error": "API token required"}
|
return {"ok": False, "error": "API token required"}
|
||||||
else:
|
else:
|
||||||
var = _form_str(form, "api_token_env", "").strip()
|
var = _form_str(form, "api_token_env", "").strip()
|
||||||
|
saved = app.integrations.portainer.environments.get(_form_str(form, "env_original", ""))
|
||||||
|
if var and not _token_env_allowed(var, saved.api_token_env if saved else ""):
|
||||||
|
return {"ok": False, "error": "the environment variable name must start with PORTAINER_"}
|
||||||
token = os.environ.get(var, "") if var else ""
|
token = os.environ.get(var, "") if var else ""
|
||||||
if not token:
|
if not token:
|
||||||
return {"ok": False, "error": f"environment variable {var!r} is not set in the Inspector container"}
|
return {"ok": False, "error": f"environment variable {var!r} is not set in the Inspector container"}
|
||||||
@@ -1974,6 +2066,7 @@ def _render_diag_strip(config_ref: ConfigRef) -> str:
|
|||||||
return (
|
return (
|
||||||
'<div class="diag-strip">'
|
'<div class="diag-strip">'
|
||||||
f'<span>config: <code>{_escape(str(path))}</code></span>'
|
f'<span>config: <code>{_escape(str(path))}</code></span>'
|
||||||
|
f'<span>time zone: <code>{_escape(formatting.LOCAL_TZ_NAME)}</code></span>'
|
||||||
f"<span>{badge}{hint}</span>"
|
f"<span>{badge}{hint}</span>"
|
||||||
f'<span style="margin-left:auto;">Inspector v{__version__}</span>'
|
f'<span style="margin-left:auto;">Inspector v{__version__}</span>'
|
||||||
"</div>\n"
|
"</div>\n"
|
||||||
@@ -2113,6 +2206,8 @@ def _settings_body_for(section: str, app: AppConfig, csrf_token: str) -> str:
|
|||||||
return (
|
return (
|
||||||
'<h3 class="pane-title">Scheduled maintenance window</h3>'
|
'<h3 class="pane-title">Scheduled maintenance window</h3>'
|
||||||
'<p class="pane-sub">Daily quiet hours. Containers that opt in (<code>maintenance_window: true</code> in their config) are not acted on during the window.</p>'
|
'<p class="pane-sub">Daily quiet hours. Containers that opt in (<code>maintenance_window: true</code> in their config) are not acted on during the window.</p>'
|
||||||
|
f'<p class="pane-sub" style="margin-top:-1rem;">Times are in <b>{_escape(formatting.LOCAL_TZ_NAME)}</b>, '
|
||||||
|
"from the <code>TZ</code> setting of the Inspector stack. If that's not your time zone, the window runs at the wrong hours.</p>"
|
||||||
+ _settings_form("maintenance", "Maintenance", body, csrf_token)
|
+ _settings_form("maintenance", "Maintenance", body, csrf_token)
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -2177,7 +2272,7 @@ def _portainer_env_form(env_name: str, env, csrf_token: str, *, suggest_name: st
|
|||||||
+ _settings_field("name", "env_name", env_name or suggest_name,
|
+ _settings_field("name", "env_name", env_name or suggest_name,
|
||||||
hint="Your own label, referenced by containers")
|
hint="Your own label, referenced by containers")
|
||||||
+ _settings_field("endpoint_id", "endpoint_id", str(env.endpoint_id if env else 1), kind="number",
|
+ _settings_field("endpoint_id", "endpoint_id", str(env.endpoint_id if env else 1), kind="number",
|
||||||
hint="Portainer → Environments; the number in the URL (usually 1 or 2)")
|
hint="Open the environment from Portainer's Home page; the number after #!/ in the address (often 1 or 2)")
|
||||||
+ "</div>"
|
+ "</div>"
|
||||||
+ _settings_field("base_url", "base_url", env.base_url if env else "",
|
+ _settings_field("base_url", "base_url", env.base_url if env else "",
|
||||||
hint="e.g. http://192.168.1.10:9000 or https://portainer.example.com")
|
hint="e.g. http://192.168.1.10:9000 or https://portainer.example.com")
|
||||||
@@ -2190,7 +2285,7 @@ def _portainer_env_form(env_name: str, env, csrf_token: str, *, suggest_name: st
|
|||||||
+ f'<div class="token-src" data-src="env"{" hidden" if inline else ""}>'
|
+ f'<div class="token-src" data-src="env"{" hidden" if inline else ""}>'
|
||||||
+ _settings_field("environment variable name", "api_token_env",
|
+ _settings_field("environment variable name", "api_token_env",
|
||||||
(env.api_token_env if env else "") or "PORTAINER_API_TOKEN",
|
(env.api_token_env if env else "") or "PORTAINER_API_TOKEN",
|
||||||
hint="Set this variable on the Inspector container (Portainer → stack → Environment variables), then redeploy Inspector.")
|
hint="Add it to the Inspector stack's environment variables; the compose file must pass it in. Step by step: Help → Portainer stack redeploy, step 2.")
|
||||||
+ "</div>"
|
+ "</div>"
|
||||||
+ f'<div class="token-src" data-src="inline"{"" if inline else " hidden"}>'
|
+ f'<div class="token-src" data-src="inline"{"" if inline else " hidden"}>'
|
||||||
+ _settings_secret_field("api_token", "api_token", bool(env and env.api_token))
|
+ _settings_secret_field("api_token", "api_token", bool(env and env.api_token))
|
||||||
@@ -2203,7 +2298,7 @@ def _portainer_env_form(env_name: str, env, csrf_token: str, *, suggest_name: st
|
|||||||
if env is not None:
|
if env is not None:
|
||||||
delete = (
|
delete = (
|
||||||
'<button type="submit" name="delete" value="1" class="btn-danger" '
|
'<button type="submit" name="delete" value="1" class="btn-danger" '
|
||||||
f"onclick=\"return confirm('Delete Portainer environment {_escape(env_name)}?');\">Delete</button>"
|
f'onclick="return confirm({_js_string_attr(f"Delete Portainer environment {env_name}?")});">Delete</button>'
|
||||||
)
|
)
|
||||||
label = "environment" if env is not None else "new environment"
|
label = "environment" if env is not None else "new environment"
|
||||||
return (
|
return (
|
||||||
@@ -2440,7 +2535,7 @@ def _render_container_advanced(name: str, cfg, app: AppConfig) -> str:
|
|||||||
+ '<div class="grid-2">'
|
+ '<div class="grid-2">'
|
||||||
+ _settings_field("timeout_seconds", "http_check_timeout_seconds", timeout_str, kind="number")
|
+ _settings_field("timeout_seconds", "http_check_timeout_seconds", timeout_str, kind="number")
|
||||||
+ _settings_field("expected_status", "http_check_expected_status", str(hc.expected_status), kind="number",
|
+ _settings_field("expected_status", "http_check_expected_status", str(hc.expected_status), kind="number",
|
||||||
hint="Exact HTTP code a healthy service returns (often 200; some apps return 401 or 302)")
|
hint="Exact HTTP code a healthy service returns (often 200; some apps return 401). Redirects are followed, so use the final page's code.")
|
||||||
+ "</div></div>"
|
+ "</div></div>"
|
||||||
'<div class="form-group"><h4>Uptime Kuma monitor</h4>'
|
'<div class="form-group"><h4>Uptime Kuma monitor</h4>'
|
||||||
+ _settings_field("enabled", "uptime_kuma_enabled", "on" if ukm.enabled else "", kind="switch")
|
+ _settings_field("enabled", "uptime_kuma_enabled", "on" if ukm.enabled else "", kind="switch")
|
||||||
@@ -2451,18 +2546,25 @@ def _render_container_advanced(name: str, cfg, app: AppConfig) -> str:
|
|||||||
)
|
)
|
||||||
|
|
||||||
others = [c.name for c in app.containers if c.name != name]
|
others = [c.name for c in app.containers if c.name != name]
|
||||||
|
disabled = {c.name for c in app.containers if not c.enabled}
|
||||||
dep_value = getattr(cfg, "depends_on", "") or ""
|
dep_value = getattr(cfg, "depends_on", "") or ""
|
||||||
dep_options = [("", "— none —")] + [(n, n) for n in others]
|
dep_options = [("", "— none —")] + [(n, n + (" (disabled)" if n in disabled else "")) for n in others]
|
||||||
if dep_value and dep_value not in others:
|
if dep_value and dep_value not in others:
|
||||||
dep_options.append((dep_value, f"{dep_value} (not configured)"))
|
dep_options.append((dep_value, f"{dep_value} (not configured)"))
|
||||||
|
dep_warn = ""
|
||||||
|
if dep_value in disabled:
|
||||||
|
dep_warn = _warn(f"{_escape(dep_value)} is disabled, so Inspector can't see its status and won't wait for it. "
|
||||||
|
"Enable it to make this dependency count.")
|
||||||
deps = (
|
deps = (
|
||||||
'<div class="form-group"><h4>Dependencies</h4>'
|
'<div class="form-group"><h4>Dependencies</h4>'
|
||||||
'<p class="group-sub">Check this container after the one it relies on, and hold off recovering it '
|
'<p class="group-sub">Check this container after the one it relies on, and hold off recovering it '
|
||||||
"while that one is still failing.</p>"
|
"while that one is still failing.</p>"
|
||||||
+ _settings_select("depends_on", "depends_on", dep_value, dep_options,
|
+ _settings_select("depends_on", "depends_on", dep_value, dep_options,
|
||||||
hint="e.g. the VPN container, for apps that use network_mode: container:<vpn>")
|
hint="e.g. the VPN container, for apps that use network_mode: container:<vpn>. "
|
||||||
|
"Also works for any other dependency, like an app and its database.")
|
||||||
|
+ dep_warn
|
||||||
+ _settings_field("vpn_routed", "vpn_routed", "on" if getattr(cfg, "vpn_routed", False) else "", kind="switch",
|
+ _settings_field("vpn_routed", "vpn_routed", "on" if getattr(cfg, "vpn_routed", False) else "", kind="switch",
|
||||||
hint="Routes through the container above. Logs why recovery is deferred while it is down.")
|
hint="Only affects logging: says in the log when recovery waits on the VPN container. The waiting itself comes from depends_on.")
|
||||||
+ "</div>"
|
+ "</div>"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -2573,7 +2675,7 @@ def _render_container_form(
|
|||||||
delete_btn = (
|
delete_btn = (
|
||||||
f'<form method="post" action="/containers/{_escape(name)}/delete" '
|
f'<form method="post" action="/containers/{_escape(name)}/delete" '
|
||||||
'style="display:inline;" '
|
'style="display:inline;" '
|
||||||
f"onsubmit=\"return confirm('Delete container {_escape(name)} from config.yml?');\">"
|
f'onsubmit="return confirm({_js_string_attr(f"Delete container {name} from config.yml?")});">'
|
||||||
f'<input type="hidden" name="csrf" value="{_escape(csrf_token)}">'
|
f'<input type="hidden" name="csrf" value="{_escape(csrf_token)}">'
|
||||||
'<button type="submit" class="btn-danger">Delete container</button>'
|
'<button type="submit" class="btn-danger">Delete container</button>'
|
||||||
"</form>"
|
"</form>"
|
||||||
@@ -2677,8 +2779,39 @@ class _DashboardHTTPServer(ThreadingHTTPServer):
|
|||||||
self.states = states
|
self.states = states
|
||||||
self.maintenance = maintenance
|
self.maintenance = maintenance
|
||||||
self.notification_mute = notification_mute
|
self.notification_mute = notification_mute
|
||||||
# token -> (expiry_timestamp, csrf_token)
|
# token -> (expiry_timestamp, csrf_token, credential fingerprint)
|
||||||
self._sessions: dict[str, tuple[float, str]] = {}
|
self._sessions: dict[str, tuple[float, str, str]] = {}
|
||||||
|
# client address -> timestamps of recent failed logins
|
||||||
|
self._failed_logins: dict[str, list[float]] = {}
|
||||||
|
self._failed_lock = threading.Lock()
|
||||||
|
|
||||||
|
def login_throttled(self, client: str) -> bool:
|
||||||
|
cutoff = time.time() - _LOGIN_FAILURE_WINDOW
|
||||||
|
with self._failed_lock:
|
||||||
|
recent = [t for t in self._failed_logins.get(client, []) if t > cutoff]
|
||||||
|
if recent:
|
||||||
|
self._failed_logins[client] = recent
|
||||||
|
else:
|
||||||
|
self._failed_logins.pop(client, None)
|
||||||
|
return len(recent) >= _LOGIN_MAX_FAILURES
|
||||||
|
|
||||||
|
def note_login_failure(self, client: str) -> None:
|
||||||
|
with self._failed_lock:
|
||||||
|
self._failed_logins.setdefault(client, []).append(time.time())
|
||||||
|
|
||||||
|
def clear_login_failures(self, client: str) -> None:
|
||||||
|
with self._failed_lock:
|
||||||
|
self._failed_logins.pop(client, None)
|
||||||
|
|
||||||
|
def new_session(self) -> str:
|
||||||
|
now = time.time()
|
||||||
|
for tok, entry in list(self._sessions.items()): # drop expired sessions
|
||||||
|
if entry[0] <= now:
|
||||||
|
self._sessions.pop(tok, None)
|
||||||
|
token = secrets.token_urlsafe(32)
|
||||||
|
fp = _cred_fingerprint(self.current_app().dashboard)
|
||||||
|
self._sessions[token] = (now + _SESSION_MAX_AGE, secrets.token_urlsafe(32), fp)
|
||||||
|
return token
|
||||||
|
|
||||||
def current_app(self) -> AppConfig:
|
def current_app(self) -> AppConfig:
|
||||||
return self.config_ref.get()
|
return self.config_ref.get()
|
||||||
@@ -2703,29 +2836,47 @@ class _Handler(BaseHTTPRequestHandler):
|
|||||||
def _is_authenticated(self) -> bool:
|
def _is_authenticated(self) -> bool:
|
||||||
# Session cookie
|
# Session cookie
|
||||||
now = time.time()
|
now = time.time()
|
||||||
|
cfg = self.server.current_app().dashboard
|
||||||
token = self._session_token()
|
token = self._session_token()
|
||||||
if token:
|
if token:
|
||||||
entry = self.server._sessions.get(token)
|
entry = self.server._sessions.get(token)
|
||||||
if entry and now < entry[0]:
|
if entry and now < entry[0] and entry[2] == _cred_fingerprint(cfg):
|
||||||
return True
|
return True
|
||||||
self.server._sessions.pop(token, None)
|
self.server._sessions.pop(token, None)
|
||||||
|
|
||||||
# Basic Auth
|
# Basic Auth
|
||||||
header = self.headers.get("Authorization", "")
|
header = self.headers.get("Authorization", "")
|
||||||
if header.startswith("Basic "):
|
if header.startswith("Basic "):
|
||||||
|
client = self.client_address[0]
|
||||||
|
if self.server.login_throttled(client):
|
||||||
|
return False
|
||||||
try:
|
try:
|
||||||
decoded = b64decode(header[len("Basic "):]).decode("utf-8")
|
decoded = b64decode(header[len("Basic "):]).decode("utf-8")
|
||||||
username, _, password = decoded.partition(":")
|
username, _, password = decoded.partition(":")
|
||||||
except Exception:
|
except Exception:
|
||||||
username, password = "", ""
|
username, password = "", ""
|
||||||
cfg = self.server.current_app().dashboard
|
if _creds_match(username, password, cfg):
|
||||||
if secrets.compare_digest(username, cfg.username) and secrets.compare_digest(
|
self.server.clear_login_failures(client)
|
||||||
password, cfg.password
|
|
||||||
):
|
|
||||||
return True
|
return True
|
||||||
|
self.server.note_login_failure(client)
|
||||||
|
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def _read_form(self) -> dict | None:
|
||||||
|
"""Parse the urlencoded request body, or send 400/413 and return None."""
|
||||||
|
try:
|
||||||
|
length = int(self.headers.get("Content-Length", 0))
|
||||||
|
except ValueError:
|
||||||
|
length = -1
|
||||||
|
if length < 0:
|
||||||
|
self._send_bytes(HTTPStatus.BAD_REQUEST, "text/plain; charset=utf-8", b"bad Content-Length")
|
||||||
|
return None
|
||||||
|
if length > _MAX_BODY_BYTES:
|
||||||
|
self._send_bytes(HTTPStatus.REQUEST_ENTITY_TOO_LARGE, "text/plain; charset=utf-8", b"request too large")
|
||||||
|
return None
|
||||||
|
raw = self.rfile.read(length) if length > 0 else b""
|
||||||
|
return parse_qs(raw.decode("utf-8", errors="replace"))
|
||||||
|
|
||||||
def _csrf_token_for_session(self) -> str:
|
def _csrf_token_for_session(self) -> str:
|
||||||
"""Return the CSRF token bound to the current session, or '' if none."""
|
"""Return the CSRF token bound to the current session, or '' if none."""
|
||||||
token = self._session_token()
|
token = self._session_token()
|
||||||
@@ -2883,18 +3034,21 @@ class _Handler(BaseHTTPRequestHandler):
|
|||||||
server = self.server
|
server = self.server
|
||||||
|
|
||||||
if path == "/login":
|
if path == "/login":
|
||||||
length = int(self.headers.get("Content-Length", 0))
|
data = self._read_form()
|
||||||
raw = self.rfile.read(length) if length > 0 else b""
|
if data is None:
|
||||||
data = parse_qs(raw.decode("utf-8", errors="replace"))
|
return
|
||||||
|
client = self.client_address[0]
|
||||||
|
if server.login_throttled(client):
|
||||||
|
self._send_bytes(
|
||||||
|
HTTPStatus.TOO_MANY_REQUESTS, "text/html; charset=utf-8", _render_login_page(throttled=True)
|
||||||
|
)
|
||||||
|
return
|
||||||
username = data.get("username", [""])[0]
|
username = data.get("username", [""])[0]
|
||||||
password = data.get("password", [""])[0]
|
password = data.get("password", [""])[0]
|
||||||
cfg = server.current_app().dashboard
|
cfg = server.current_app().dashboard
|
||||||
if secrets.compare_digest(username, cfg.username) and secrets.compare_digest(
|
if _creds_match(username, password, cfg):
|
||||||
password, cfg.password
|
server.clear_login_failures(client)
|
||||||
):
|
token = server.new_session()
|
||||||
token = secrets.token_urlsafe(32)
|
|
||||||
csrf = secrets.token_urlsafe(32)
|
|
||||||
server._sessions[token] = (time.time() + _SESSION_MAX_AGE, csrf)
|
|
||||||
self.send_response(HTTPStatus.SEE_OTHER)
|
self.send_response(HTTPStatus.SEE_OTHER)
|
||||||
self.send_header("Location", "/")
|
self.send_header("Location", "/")
|
||||||
self.send_header(
|
self.send_header(
|
||||||
@@ -2904,6 +3058,7 @@ class _Handler(BaseHTTPRequestHandler):
|
|||||||
self.send_header("Content-Length", "0")
|
self.send_header("Content-Length", "0")
|
||||||
self.end_headers()
|
self.end_headers()
|
||||||
else:
|
else:
|
||||||
|
server.note_login_failure(client)
|
||||||
self._send_bytes(
|
self._send_bytes(
|
||||||
HTTPStatus.UNAUTHORIZED, "text/html; charset=utf-8", _render_login_page(error=True)
|
HTTPStatus.UNAUTHORIZED, "text/html; charset=utf-8", _render_login_page(error=True)
|
||||||
)
|
)
|
||||||
@@ -2912,9 +3067,9 @@ class _Handler(BaseHTTPRequestHandler):
|
|||||||
if not self._check_auth():
|
if not self._check_auth():
|
||||||
return
|
return
|
||||||
|
|
||||||
length = int(self.headers.get("Content-Length", 0))
|
form = self._read_form()
|
||||||
raw_body = self.rfile.read(length) if length > 0 else b""
|
if form is None:
|
||||||
form = parse_qs(raw_body.decode("utf-8", errors="replace"))
|
return
|
||||||
|
|
||||||
# Only cookie-authenticated requests need a CSRF token. Basic Auth
|
# Only cookie-authenticated requests need a CSRF token. Basic Auth
|
||||||
# clients prove identity on every request so they're not subject to
|
# clients prove identity on every request so they're not subject to
|
||||||
|
|||||||
@@ -1,14 +1,49 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
import logging
|
||||||
from zoneinfo import ZoneInfo
|
import os
|
||||||
|
from datetime import datetime, timezone, tzinfo
|
||||||
|
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||||
|
|
||||||
|
|
||||||
PACIFIC = ZoneInfo("America/Los_Angeles")
|
log = logging.getLogger("inspector.formatting")
|
||||||
|
|
||||||
|
|
||||||
def format_timestamp(value: datetime | str | None) -> str:
|
def _resolve_local_tz(env: dict | None = None) -> tuple[tzinfo, str]:
|
||||||
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'.
|
"""The zone every displayed time uses, plus its name.
|
||||||
|
|
||||||
|
Follows the same rules as the C library (which drives log timestamps and
|
||||||
|
the maintenance window), so all times Inspector shows agree:
|
||||||
|
TZ set to a zone name -> that zone; TZ unset -> the container's
|
||||||
|
/etc/localtime; anything else -> UTC.
|
||||||
|
"""
|
||||||
|
env = os.environ if env is None else env
|
||||||
|
name = (env.get("TZ") or "").strip().lstrip(":")
|
||||||
|
if name:
|
||||||
|
try:
|
||||||
|
return ZoneInfo(name), name
|
||||||
|
except (ZoneInfoNotFoundError, ValueError):
|
||||||
|
log.warning(
|
||||||
|
"TZ=%r is not a known time zone name (e.g. Europe/Berlin); showing times in UTC",
|
||||||
|
name,
|
||||||
|
)
|
||||||
|
return timezone.utc, "UTC"
|
||||||
|
try:
|
||||||
|
with open("/etc/localtime", "rb") as fh:
|
||||||
|
zone = ZoneInfo.from_file(fh, key="localtime")
|
||||||
|
label = datetime.now(zone).tzname() or "localtime"
|
||||||
|
return zone, label
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return timezone.utc, "UTC"
|
||||||
|
|
||||||
|
|
||||||
|
# Resolved once at import; the container's TZ doesn't change while it runs.
|
||||||
|
LOCAL_TZ, LOCAL_TZ_NAME = _resolve_local_tz()
|
||||||
|
|
||||||
|
|
||||||
|
def format_timestamp(value: datetime | str | None, *, with_zone: bool = False) -> str:
|
||||||
|
"""Render a timestamp in the configured local zone, e.g. '06-14 03:45 PM'
|
||||||
|
(with_zone adds the abbreviation: '06-14 03:45 PM CET').
|
||||||
|
|
||||||
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
|
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
|
||||||
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
|
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
|
||||||
@@ -24,4 +59,8 @@ def format_timestamp(value: datetime | str | None) -> str:
|
|||||||
return value
|
return value
|
||||||
if value.tzinfo is None:
|
if value.tzinfo is None:
|
||||||
value = value.replace(tzinfo=timezone.utc)
|
value = value.replace(tzinfo=timezone.utc)
|
||||||
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p")
|
local = value.astimezone(LOCAL_TZ)
|
||||||
|
text = local.strftime("%m-%d %I:%M %p")
|
||||||
|
if with_zone:
|
||||||
|
text += f" {local.tzname()}"
|
||||||
|
return text
|
||||||
+43
-15
@@ -41,7 +41,7 @@ from .maintenance import MaintenanceTracker
|
|||||||
from .notification_mute import NotificationMuteTracker
|
from .notification_mute import NotificationMuteTracker
|
||||||
from .notifier import GotifyNotifier, MultiNotifier, NullNotifier, PushoverNotifier
|
from .notifier import GotifyNotifier, MultiNotifier, NullNotifier, PushoverNotifier
|
||||||
from .portainer_client import PortainerClient
|
from .portainer_client import PortainerClient
|
||||||
from .recovery import perform_recovery
|
from .recovery import perform_recovery, run_post_redeploy_recheck
|
||||||
from .uptime_kuma import HeartbeatSnapshot, UptimeKumaClient
|
from .uptime_kuma import HeartbeatSnapshot, UptimeKumaClient
|
||||||
from .state import (
|
from .state import (
|
||||||
ContainerState,
|
ContainerState,
|
||||||
@@ -63,6 +63,19 @@ def _request_stop(signum, _frame):
|
|||||||
_stop = True
|
_stop = True
|
||||||
|
|
||||||
|
|
||||||
|
def _is_normal(status: str | None, container_config: ContainerConfig) -> bool:
|
||||||
|
"""True when `status` means the container is working.
|
||||||
|
|
||||||
|
HEALTHY always counts. NO HEALTH CHECK (running, but the image defines no
|
||||||
|
health check) counts too, unless the user made it a restart trigger.
|
||||||
|
Many images have no health check, and without this their error count and
|
||||||
|
restart episode would never reset.
|
||||||
|
"""
|
||||||
|
if status == HEALTHY:
|
||||||
|
return True
|
||||||
|
return status == NO_HEALTH_CHECK and NO_HEALTH_CHECK not in container_config.restart_on
|
||||||
|
|
||||||
|
|
||||||
def _apply_reachability_override(
|
def _apply_reachability_override(
|
||||||
container_config: ContainerConfig,
|
container_config: ContainerConfig,
|
||||||
result: HealthResult,
|
result: HealthResult,
|
||||||
@@ -140,6 +153,11 @@ def _check_one(
|
|||||||
if notification_mute.is_muted(name):
|
if notification_mute.is_muted(name):
|
||||||
notifier = NullNotifier()
|
notifier = NullNotifier()
|
||||||
|
|
||||||
|
if state.redeploy_recheck_at is not None and now >= state.redeploy_recheck_at:
|
||||||
|
run_post_redeploy_recheck(
|
||||||
|
container_config=container_config, state=state, notifier=notifier, now=now,
|
||||||
|
)
|
||||||
|
|
||||||
if state.previous_status != state.current_status:
|
if state.previous_status != state.current_status:
|
||||||
log.info(
|
log.info(
|
||||||
"%-*s status: %s -> %s%s",
|
"%-*s status: %s -> %s%s",
|
||||||
@@ -158,19 +176,25 @@ def _check_one(
|
|||||||
message=result.message,
|
message=result.message,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
is_normal = _is_normal(state.current_status, container_config)
|
||||||
if (
|
if (
|
||||||
state.previous_status is not None
|
state.previous_status is not None
|
||||||
and state.previous_status != HEALTHY
|
and not _is_normal(state.previous_status, container_config)
|
||||||
and state.current_status == HEALTHY
|
and is_normal
|
||||||
|
# HEALTHY here means "a recovery was already announced", for either status.
|
||||||
and state.last_notification_state != HEALTHY
|
and state.last_notification_state != HEALTHY
|
||||||
):
|
):
|
||||||
log.info("%-*s recovered: %s -> HEALTHY", width, name, state.previous_status)
|
log.info("%-*s recovered: %s -> %s", width, name, state.previous_status, state.current_status)
|
||||||
|
back_to = (
|
||||||
|
"**HEALTHY**" if state.current_status == HEALTHY
|
||||||
|
else "**running** (no health check)"
|
||||||
|
)
|
||||||
notifier.notify(
|
notifier.notify(
|
||||||
title=f"✅ Inspector: {name} recovered",
|
title=f"✅ Inspector: {name} recovered",
|
||||||
message=(
|
message=(
|
||||||
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
|
f"**{name}** is back to {back_to} (was *{state.previous_status}*).\n\n"
|
||||||
f"- Errors before recovery: **{state.error_count}**\n"
|
f"- Errors before recovery: **{state.error_count}**\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=3,
|
priority=3,
|
||||||
)
|
)
|
||||||
@@ -178,7 +202,7 @@ def _check_one(
|
|||||||
container=name,
|
container=name,
|
||||||
event_type=EVENT_RECOVERY,
|
event_type=EVENT_RECOVERY,
|
||||||
old_status=state.previous_status,
|
old_status=state.previous_status,
|
||||||
new_status=HEALTHY,
|
new_status=state.current_status,
|
||||||
error_count=state.error_count,
|
error_count=state.error_count,
|
||||||
)
|
)
|
||||||
state.last_notification_state = HEALTHY
|
state.last_notification_state = HEALTHY
|
||||||
@@ -196,7 +220,7 @@ def _check_one(
|
|||||||
f"**{name}** is missing from Docker.\n\n"
|
f"**{name}** is missing from Docker.\n\n"
|
||||||
+ (f"- Detail: {result.message}\n" if result.message else "")
|
+ (f"- Detail: {result.message}\n" if result.message else "")
|
||||||
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
|
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=7,
|
priority=7,
|
||||||
)
|
)
|
||||||
@@ -240,7 +264,7 @@ def _check_one(
|
|||||||
+ ".\n\n"
|
+ ".\n\n"
|
||||||
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
|
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
|
||||||
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
|
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=5,
|
priority=5,
|
||||||
)
|
)
|
||||||
@@ -253,9 +277,9 @@ def _check_one(
|
|||||||
message=result.message,
|
message=result.message,
|
||||||
)
|
)
|
||||||
state.last_notification_state = result.status
|
state.last_notification_state = result.status
|
||||||
elif result.status == HEALTHY:
|
elif is_normal:
|
||||||
if state.error_count != 0:
|
if state.error_count != 0:
|
||||||
log.info("%-*s healthy again; resetting error count", width, name)
|
log.info("%-*s back to normal; resetting error count", width, name)
|
||||||
state.error_count = 0
|
state.error_count = 0
|
||||||
state.restarted_this_episode = False
|
state.restarted_this_episode = False
|
||||||
|
|
||||||
@@ -276,7 +300,7 @@ def _check_one(
|
|||||||
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
|
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
|
||||||
f"- Current status: **{result.status}**\n"
|
f"- Current status: **{result.status}**\n"
|
||||||
f"- Restarts paused until the hourly window clears\n"
|
f"- Restarts paused until the hourly window clears\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=9,
|
priority=9,
|
||||||
)
|
)
|
||||||
@@ -293,8 +317,10 @@ def _check_one(
|
|||||||
|
|
||||||
if ok:
|
if ok:
|
||||||
dep = container_config.depends_on
|
dep = container_config.depends_on
|
||||||
if dep:
|
# Only wait on a dependency Inspector is watching. A disabled one has
|
||||||
dep_status = states[dep].current_status if dep in states else None
|
# no state, and waiting on it would block recovery forever.
|
||||||
|
if dep and dep in states:
|
||||||
|
dep_status = states[dep].current_status
|
||||||
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
|
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
|
||||||
if container_config.vpn_routed:
|
if container_config.vpn_routed:
|
||||||
log.info(
|
log.info(
|
||||||
@@ -454,8 +480,10 @@ def run_loop(config_ref: ConfigRef) -> int:
|
|||||||
for gone in states.keys() - current_enabled.keys():
|
for gone in states.keys() - current_enabled.keys():
|
||||||
states.pop(gone, None)
|
states.pop(gone, None)
|
||||||
log.info("Stopped watching container (removed or disabled): %s", gone)
|
log.info("Stopped watching container (removed or disabled): %s", gone)
|
||||||
|
if current_enabled != enabled_configs:
|
||||||
|
# Only re-order (and re-log ordering warnings) when the config changed.
|
||||||
|
check_order = _topological_order(current_enabled)
|
||||||
enabled_configs = current_enabled
|
enabled_configs = current_enabled
|
||||||
check_order = _topological_order(enabled_configs)
|
|
||||||
|
|
||||||
# Integration clients hold their config; rebuild them when the
|
# Integration clients hold their config; rebuild them when the
|
||||||
# settings page changes Uptime Kuma / Portainer.
|
# settings page changes Uptime Kuma / Portainer.
|
||||||
|
|||||||
@@ -33,9 +33,10 @@ class GotifyNotifier:
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
try:
|
try:
|
||||||
|
# Header, not ?token=, so the token stays out of access logs.
|
||||||
resp = requests.post(
|
resp = requests.post(
|
||||||
url,
|
url,
|
||||||
params={"token": self._config.token},
|
headers={"X-Gotify-Key": self._config.token},
|
||||||
json=payload,
|
json=payload,
|
||||||
timeout=self._timeout,
|
timeout=self._timeout,
|
||||||
)
|
)
|
||||||
|
|||||||
+46
-24
@@ -1,7 +1,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
import time
|
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
from .config import AppConfig, ContainerConfig, PortainerConfig
|
from .config import AppConfig, ContainerConfig, PortainerConfig
|
||||||
@@ -175,7 +174,8 @@ def _perform_restart_then_redeploy(
|
|||||||
- the container is still failing on a later recovery cycle
|
- the container is still failing on a later recovery cycle
|
||||||
(`state.restarted_this_episode` is already set), meaning the restart didn't stick.
|
(`state.restarted_this_episode` is already set), meaning the restart didn't stick.
|
||||||
|
|
||||||
The episode flag resets when the container is next seen HEALTHY (see main._check_one).
|
The episode flag resets when the container is next seen working: HEALTHY, or
|
||||||
|
NO HEALTH CHECK when that isn't a restart trigger (see main._is_normal).
|
||||||
"""
|
"""
|
||||||
name = container_config.name
|
name = container_config.name
|
||||||
|
|
||||||
@@ -228,7 +228,7 @@ def _perform_restart_then_redeploy(
|
|||||||
f"**{result.status}** check(s).\n\n"
|
f"**{result.status}** check(s).\n\n"
|
||||||
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
|
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
|
||||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
|
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=6,
|
priority=6,
|
||||||
)
|
)
|
||||||
@@ -242,7 +242,7 @@ def _perform_restart_then_redeploy(
|
|||||||
state.restart_limit_notified = False
|
state.restart_limit_notified = False
|
||||||
log.info(
|
log.info(
|
||||||
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
|
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
|
||||||
width, name, state.cooldown_until.isoformat(),
|
width, name, format_timestamp(state.cooldown_until, with_zone=True),
|
||||||
)
|
)
|
||||||
eventstore.record(
|
eventstore.record(
|
||||||
container=name,
|
container=name,
|
||||||
@@ -250,15 +250,15 @@ def _perform_restart_then_redeploy(
|
|||||||
old_status=state.previous_status,
|
old_status=state.previous_status,
|
||||||
new_status=result.status,
|
new_status=result.status,
|
||||||
error_count=state.error_count,
|
error_count=state.error_count,
|
||||||
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
message=f"restart succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
|
||||||
)
|
)
|
||||||
notifier.notify(
|
notifier.notify(
|
||||||
title=f"✅ Inspector: {name} restart completed",
|
title=f"✅ Inspector: {name} restart completed",
|
||||||
message=(
|
message=(
|
||||||
f"**{name}** was successfully restarted by Docker.\n\n"
|
f"**{name}** was successfully restarted by Docker.\n\n"
|
||||||
f"- Previous status: *{result.status}*\n"
|
f"- Previous status: *{result.status}*\n"
|
||||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=3,
|
priority=3,
|
||||||
)
|
)
|
||||||
@@ -321,7 +321,7 @@ def _perform_docker_action(
|
|||||||
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
|
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
|
||||||
f"**{result.status}** check(s).\n\n"
|
f"**{result.status}** check(s).\n\n"
|
||||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
|
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=6,
|
priority=6,
|
||||||
)
|
)
|
||||||
@@ -335,22 +335,22 @@ def _perform_docker_action(
|
|||||||
state.record_restart(now, container_config.cooldown_seconds)
|
state.record_restart(now, container_config.cooldown_seconds)
|
||||||
state.last_notification_state = "RESTARTED"
|
state.last_notification_state = "RESTARTED"
|
||||||
state.restart_limit_notified = False
|
state.restart_limit_notified = False
|
||||||
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat())
|
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, format_timestamp(state.cooldown_until, with_zone=True))
|
||||||
eventstore.record(
|
eventstore.record(
|
||||||
container=name,
|
container=name,
|
||||||
event_type=EVENT_RESTART,
|
event_type=EVENT_RESTART,
|
||||||
old_status=state.previous_status,
|
old_status=state.previous_status,
|
||||||
new_status=result.status,
|
new_status=result.status,
|
||||||
error_count=state.error_count,
|
error_count=state.error_count,
|
||||||
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
message=f"{verb} succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
|
||||||
)
|
)
|
||||||
notifier.notify(
|
notifier.notify(
|
||||||
title=f"✅ Inspector: {name} {verb} completed",
|
title=f"✅ Inspector: {name} {verb} completed",
|
||||||
message=(
|
message=(
|
||||||
f"**{name}** was successfully {verb}ed by Docker.\n\n"
|
f"**{name}** was successfully {verb}ed by Docker.\n\n"
|
||||||
f"- Previous status: *{result.status}*\n"
|
f"- Previous status: *{result.status}*\n"
|
||||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=3,
|
priority=3,
|
||||||
)
|
)
|
||||||
@@ -362,7 +362,7 @@ def _perform_docker_action(
|
|||||||
f"{verb.capitalize()} of **{name}** failed.\n\n"
|
f"{verb.capitalize()} of **{name}** failed.\n\n"
|
||||||
f"- Status: **{result.status}**\n"
|
f"- Status: **{result.status}**\n"
|
||||||
f"- Error count: **{state.error_count}**\n"
|
f"- Error count: **{state.error_count}**\n"
|
||||||
f"- Time: {format_timestamp(now)}\n\n"
|
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
|
||||||
f"```\n{err}\n```"
|
f"```\n{err}\n```"
|
||||||
),
|
),
|
||||||
priority=9,
|
priority=9,
|
||||||
@@ -469,7 +469,7 @@ def _perform_portainer_redeploy(
|
|||||||
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
|
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
|
||||||
f"to recover **{name}**.\n\n"
|
f"to recover **{name}**.\n\n"
|
||||||
f"- Status: **{result.status}**\n"
|
f"- Status: **{result.status}**\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=6,
|
priority=6,
|
||||||
)
|
)
|
||||||
@@ -483,7 +483,7 @@ def _perform_portainer_redeploy(
|
|||||||
message=(
|
message=(
|
||||||
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
|
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
|
||||||
f"- Triggered by: **{name}** ({result.status})\n"
|
f"- Triggered by: **{name}** ({result.status})\n"
|
||||||
f"- Time: {format_timestamp(now)}\n\n"
|
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
|
||||||
f"```\n{redeploy_result.message}\n```"
|
f"```\n{redeploy_result.message}\n```"
|
||||||
),
|
),
|
||||||
priority=9,
|
priority=9,
|
||||||
@@ -516,29 +516,51 @@ def _perform_portainer_redeploy(
|
|||||||
f"successfully redeployed to recover **{name}**.\n\n"
|
f"successfully redeployed to recover **{name}**.\n\n"
|
||||||
f"- Triggered by: *{result.status}*\n"
|
f"- Triggered by: *{result.status}*\n"
|
||||||
f"- Post-redeploy wait: {wait_seconds}s\n"
|
f"- Post-redeploy wait: {wait_seconds}s\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=4,
|
priority=4,
|
||||||
)
|
)
|
||||||
|
|
||||||
log.info("waiting %d seconds after stack redeploy", wait_seconds)
|
# Don't block the loop while the stack settles: keep this container in
|
||||||
time.sleep(wait_seconds)
|
# cooldown for the wait plus its normal cooldown, and do the one-time HTTP
|
||||||
|
# re-check on its first regular check after the wait (main._check_one).
|
||||||
|
log.info(
|
||||||
|
"%-*s letting stack %s settle for %ds; other containers keep being checked",
|
||||||
|
width, name, key, wait_seconds,
|
||||||
|
)
|
||||||
|
state.record_restart(now, wait_seconds + container_config.cooldown_seconds)
|
||||||
if container_config.http_check.enabled:
|
if container_config.http_check.enabled:
|
||||||
|
state.redeploy_recheck_at = now + timedelta(seconds=wait_seconds)
|
||||||
|
|
||||||
|
|
||||||
|
def run_post_redeploy_recheck(
|
||||||
|
*,
|
||||||
|
container_config: ContainerConfig,
|
||||||
|
state: ContainerState,
|
||||||
|
notifier: GotifyNotifier,
|
||||||
|
now: datetime,
|
||||||
|
) -> None:
|
||||||
|
"""One HTTP re-check after a stack redeploy has had time to settle.
|
||||||
|
|
||||||
|
Called by main._check_one once `state.redeploy_recheck_at` has passed;
|
||||||
|
alerts if the app still doesn't answer.
|
||||||
|
"""
|
||||||
|
state.redeploy_recheck_at = None
|
||||||
|
name = container_config.name
|
||||||
|
if not container_config.http_check.enabled:
|
||||||
|
return
|
||||||
recheck = check_http(container_config.http_check)
|
recheck = check_http(container_config.http_check)
|
||||||
if recheck.ok:
|
if recheck.ok:
|
||||||
log.info("recheck succeeded for %s", name)
|
log.info("recheck succeeded for %s", name)
|
||||||
else:
|
return
|
||||||
log.warning("recheck failed for %s: %s", name, recheck.message)
|
log.warning("recheck failed for %s: %s", name, recheck.message)
|
||||||
notifier.notify(
|
notifier.notify(
|
||||||
title=f"⚠️ Inspector: {name} recheck failed after redeploy",
|
title=f"⚠️ Inspector: {name} recheck failed after redeploy",
|
||||||
message=(
|
message=(
|
||||||
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
|
f"Portainer stack **{container_config.portainer_stack_name}** was redeployed, but the HTTP recheck "
|
||||||
f"for **{name}** still reports a failure.\n\n"
|
f"for **{name}** still reports a failure.\n\n"
|
||||||
f"- Recheck result: *{recheck.message}*\n"
|
f"- Recheck result: *{recheck.message}*\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=6,
|
priority=6,
|
||||||
)
|
)
|
||||||
|
|
||||||
state.record_restart(now, container_config.cooldown_seconds)
|
|
||||||
@@ -29,6 +29,8 @@ class ContainerState:
|
|||||||
restart_limit_notified: bool = False
|
restart_limit_notified: bool = False
|
||||||
restarted_this_episode: bool = False
|
restarted_this_episode: bool = False
|
||||||
last_check_at: datetime | None = None
|
last_check_at: datetime | None = None
|
||||||
|
# After a stack redeploy: when to do the one-time HTTP re-check.
|
||||||
|
redeploy_recheck_at: datetime | None = None
|
||||||
|
|
||||||
def in_cooldown(self, now: datetime) -> bool:
|
def in_cooldown(self, now: datetime) -> bool:
|
||||||
return self.cooldown_until is not None and now < self.cooldown_until
|
return self.cooldown_until is not None and now < self.cooldown_until
|
||||||
|
|||||||
Reference in new issue
Block a user