Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
57100d0a7b | ||
|
|
b4da527732 | ||
|
|
284ba2ea15 | ||
|
|
bd78835f4a | ||
|
|
7013a13cba |
No files matched your search
+128
@@ -7,6 +7,134 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## [0.9.3] — 2026-10-28
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Every time follows the `TZ` of the Inspector stack.** Before, the dashboard,
|
||||||
|
Events page and alerts were always in US Pacific time while the maintenance window
|
||||||
|
and logs used `TZ`, so anyone outside Pacific saw times hours off (e.g. a 02:00
|
||||||
|
Berlin window shown as 5:00 PM). Now the dashboard, the Events page (including
|
||||||
|
where each day starts), alert times, cooldown times and logs all use `TZ`.
|
||||||
|
- Alert times include the zone abbreviation (`10-28 09:27 AM CET`), and "Cooldown
|
||||||
|
until" in alerts, event messages and logs is a local time instead of a UTC
|
||||||
|
timestamp.
|
||||||
|
- The active time zone is shown at the bottom of every Settings page, on the Events
|
||||||
|
page and on the Maintenance page.
|
||||||
|
- **If you never set `TZ`**, times are now shown in UTC instead of Pacific. Set `TZ`
|
||||||
|
in the stack (e.g. `America/New_York`). An unrecognized name logs a warning and
|
||||||
|
falls back to UTC.
|
||||||
|
|
||||||
|
### Docs
|
||||||
|
|
||||||
|
- INSTALL: `TZ` is now marked as needed, with a "Why the time zone matters" section
|
||||||
|
(maintenance window, alerts, dashboard, logs), how to find your zone name, and a
|
||||||
|
troubleshooting row. Guide: a Time zone section and a first-login check. Compose
|
||||||
|
and README updated.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [0.9.2] — 2026-10-28
|
||||||
|
|
||||||
|
Fixes from a code and security review. None of them change how containers are
|
||||||
|
monitored or recovered.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **A bad value saved from Settings can no longer stop Inspector from starting.**
|
||||||
|
The forms now reject invalid times (maintenance start/end), unknown log levels and
|
||||||
|
out-of-range numbers (for example `interval_seconds: 0`, which made the loop spin).
|
||||||
|
If `config.yml` already has such a value, Inspector starts with the default and
|
||||||
|
logs a warning instead of crashing in a restart loop.
|
||||||
|
- **The dashboard password can no longer be cleared.** Saving an empty password (or
|
||||||
|
username) is refused, and an empty password never signs anyone in. Before, clearing
|
||||||
|
it left the running dashboard open to anyone.
|
||||||
|
- **Passwords with non-ASCII characters (é, ü, £ …) now work.** Before, every login
|
||||||
|
attempt with one crashed.
|
||||||
|
- The Events page no longer errors for a moment after a container is removed.
|
||||||
|
- The HTTP check hint no longer suggests `302` as an expected status; redirects are
|
||||||
|
followed, so the final page's code is what counts.
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
- **Sign-in throttling:** after 5 failed sign-ins (form or Basic Auth) from one address
|
||||||
|
within 5 minutes, that address is refused for the rest of the window.
|
||||||
|
- **Changing the dashboard username or password signs out existing sessions.**
|
||||||
|
Expired sessions are cleaned up.
|
||||||
|
- Request bodies are limited to 1 MB; an invalid `Content-Length` gets a 400.
|
||||||
|
- The Gotify token is sent in the `X-Gotify-Key` header instead of the URL, so it no
|
||||||
|
longer appears in Gotify or reverse-proxy access logs.
|
||||||
|
- Portainer token environment variables must be named `PORTAINER_…` (a name already in
|
||||||
|
your `config.yml` keeps working), so Test connection can't be used to send other
|
||||||
|
container secrets to an arbitrary URL.
|
||||||
|
- Single quotes are escaped in HTML, and the delete confirmations build their text safely.
|
||||||
|
- Two settings saves at the same moment can no longer overwrite each other.
|
||||||
|
- `requests` updated to 2.32.5 (CVE-2024-47081).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [0.9.1] — 2026-10-28
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **A dependency on a disabled container no longer blocks recovery forever.** If
|
||||||
|
`depends_on` named a container that is disabled in Inspector, the dependent app was
|
||||||
|
never restarted. Inspector now only waits for dependencies it is watching, and the
|
||||||
|
container page warns when the selected dependency is disabled.
|
||||||
|
- **`PORTAINER_API_TOKEN` now reaches the container.** The public `compose.yml` didn't
|
||||||
|
pass it in, so setting it as a Portainer stack variable had no effect. Existing
|
||||||
|
stacks need the line `- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}` added under
|
||||||
|
`environment:` (Stacks → inspector → Editor).
|
||||||
|
- The "depends_on … is disabled or not found" warning was logged on every check
|
||||||
|
(since 0.8.0). It is now logged once, and again only when the container settings
|
||||||
|
change.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Guide: new sections **6. Apps behind a VPN container** (why depends_on matters, how to
|
||||||
|
spot a VPN-routed app, setup, what Inspector does) and **7. Portainer stack redeploy**
|
||||||
|
(why it's needed, what a redeploy does to the whole stack, requirements, where to find
|
||||||
|
every value, adding the token, the other settings, Test connection errors).
|
||||||
|
- Clearer form hints: `vpn_routed` only affects logging; `depends_on` works for any
|
||||||
|
dependency; where to find the endpoint ID.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [0.9.0] — 2026-10-28
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Every container option is now in the dashboard.** The container page has three
|
||||||
|
new groups:
|
||||||
|
- **Reachability**: HTTP check (URL, timeout, expected status) and Uptime Kuma
|
||||||
|
monitor.
|
||||||
|
- **Dependencies**: `depends_on` (pick from your containers) and `vpn_routed`.
|
||||||
|
- **Recovery**: `recovery_action` with plain-language descriptions, plus the
|
||||||
|
Portainer environment and stack name for stack redeploys.
|
||||||
|
- **Integrations settings.** New **Uptime Kuma** and **Portainer** pages under
|
||||||
|
Settings. Portainer environments can be added, edited, renamed (containers
|
||||||
|
that use them are updated) and deleted (refused while a container uses one).
|
||||||
|
The API token can be an environment variable name (recommended) or a masked
|
||||||
|
value saved in `config.yml`.
|
||||||
|
- **Test connection** buttons for Uptime Kuma (lists monitors) and each Portainer
|
||||||
|
environment (lists stacks on the endpoint), using the values in the form even
|
||||||
|
before saving.
|
||||||
|
- The All containers grid shows each container's extras (HTTP check, Uptime Kuma,
|
||||||
|
dependency, recovery action).
|
||||||
|
- Guide: new sections on reachability checks and apps behind a VPN container.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Uptime Kuma and Portainer changes take effect on the next check; Inspector
|
||||||
|
rebuilds those clients when their settings change instead of needing a restart.
|
||||||
|
- Saving a container form unchanged now leaves `config.yml` byte-identical. Blank
|
||||||
|
optional fields are removed instead of written as empty values (`color: ''`,
|
||||||
|
`maintenance_window: false`), and an unchanged `restart_on` list keeps its
|
||||||
|
order and the comments after it. An explicit `recovery_action: restart_container`
|
||||||
|
is kept.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## [0.8.0] — 2026-10-27
|
## [0.8.0] — 2026-10-27
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
+30
-1
@@ -52,10 +52,38 @@ TZ=America/Los_Angeles
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
|
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
|
||||||
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
|
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
|
||||||
| `TZ` | Your [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), so times on the dashboard are local. | Recommended |
|
| `TZ` | **Your time zone**, e.g. `America/New_York` or `Europe/Berlin`. Without it Inspector uses UTC, so the maintenance window, alerts and dashboard times are off by your UTC offset. See [Why the time zone matters](#why-the-time-zone-matters). | **Yes** (default `UTC`) |
|
||||||
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
|
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
|
||||||
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
|
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
|
||||||
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
|
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
|
||||||
|
| `PORTAINER_API_TOKEN` | A Portainer access token. Only needed if you use **Portainer stack redeploy** for VPN-routed apps; see the in-app guide (Help → Portainer). | No |
|
||||||
|
|
||||||
|
### Why the time zone matters
|
||||||
|
|
||||||
|
Inspector uses `TZ` for every time it shows or acts on. If it's wrong, things
|
||||||
|
still work, but at the wrong hour or with confusing times:
|
||||||
|
|
||||||
|
- **The maintenance window runs in this time zone.** A window of `02:00–05:00`
|
||||||
|
with `TZ` left at UTC runs at 02:00 UTC, which is the middle of the evening in
|
||||||
|
the Americas or mid-morning in Asia. Inspector would then restart containers
|
||||||
|
during your nightly backups, and ignore real problems during the day.
|
||||||
|
- **Alerts** (Gotify / Pushover) show the time of the problem in this zone, with
|
||||||
|
its abbreviation, e.g. `10-28 03:15 AM CET`.
|
||||||
|
- **The dashboard and Events page** show times in this zone, and the Events
|
||||||
|
page's days run midnight to midnight in this zone.
|
||||||
|
- **Logs** are stamped in this zone.
|
||||||
|
|
||||||
|
**How to set it:** use your zone's name from the
|
||||||
|
[list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
|
||||||
|
("TZ identifier" column), e.g. `America/New_York`, `America/Chicago`,
|
||||||
|
`Europe/London`, `Europe/Berlin`, `Australia/Sydney`. Abbreviations like `EST` or
|
||||||
|
`CET` don't work. On Unraid, **Settings → Date and Time** shows the zone your
|
||||||
|
server uses.
|
||||||
|
|
||||||
|
**How to check it:** the bottom of every Settings page shows
|
||||||
|
`time zone: <name>`. If it says `UTC` and you're not on UTC, set `TZ` in the
|
||||||
|
stack and click **Update the stack**. If the name you set isn't recognized,
|
||||||
|
Inspector logs a warning and uses UTC.
|
||||||
|
|
||||||
## 3. Deploy and sign in
|
## 3. Deploy and sign in
|
||||||
|
|
||||||
@@ -115,6 +143,7 @@ starts and restarts the ones you add to it.
|
|||||||
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
|
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
|
||||||
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
|
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
|
||||||
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
|
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
|
||||||
|
| Times are a few hours off, or the maintenance window runs at the wrong time | `TZ` is missing or wrong. Check `time zone:` at the bottom of Settings, set `TZ` to your zone name (e.g. `Europe/London`), and update the stack. |
|
||||||
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
|
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
|
||||||
|
|
||||||
## Security notes
|
## Security notes
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="docs/logo.png" alt="Inspector logo" width="140">
|
<img src="docs/logo_outline.png" alt="Inspector logo" width="140">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<h1 align="center">Inspector</h1>
|
<h1 align="center">Inspector</h1>
|
||||||
@@ -35,6 +35,8 @@ restarts them when they break, and tells your phone about it.</p>
|
|||||||
- **Stays out of the way when you're working.** Put a container in maintenance or
|
- **Stays out of the way when you're working.** Put a container in maintenance or
|
||||||
mute its alerts with one click, or set a nightly maintenance window for backups
|
mute its alerts with one click, or set a nightly maintenance window for backups
|
||||||
and updates.
|
and updates.
|
||||||
|
- **Works in your time zone.** Set `TZ` in the stack and the maintenance window,
|
||||||
|
alerts, dashboard and logs all follow it.
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
@@ -52,11 +54,11 @@ guide (the same guide is in this repo at
|
|||||||
download it and open it in a browser to read it here).
|
download it and open it in a browser to read it here).
|
||||||
|
|
||||||
| Add a container | Restart rules |
|
| Add a container | Restart rules |
|
||||||
|---|---|
|
| ------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
||||||
|  |  |
|
|  |  |
|
||||||
|
|
||||||
| Notifications | Events history |
|
| Notifications | Events history |
|
||||||
|---|---|
|
| ------------------------------------------------------------------------------- | --------------------------------------------------------- |
|
||||||
|  |  |
|
|  |  |
|
||||||
|
|
||||||
## What it isn't
|
## What it isn't
|
||||||
@@ -69,18 +71,20 @@ download it and open it in a browser to read it here).
|
|||||||
|
|
||||||
## Advanced options
|
## Advanced options
|
||||||
|
|
||||||
Most people never need these. They live in `config.yml` (in your
|
Most people never need these. They are all on each container's page in the
|
||||||
`INSPECTOR_DATA/config` folder) rather than the dashboard; see
|
dashboard (and in `config.yml`, see
|
||||||
[`config.example.yml`](./config.example.yml) for every option with comments.
|
[`config.example.yml`](./config.example.yml) for every option with comments).
|
||||||
|
|
||||||
- **HTTP checks**: mark a container `UNREACHABLE` when a URL stops answering,
|
- **HTTP checks**: mark a container `UNREACHABLE` when a URL stops answering,
|
||||||
even if Docker says it's healthy (useful for apps behind a VPN container).
|
even if Docker says it's healthy (useful for apps behind a VPN container).
|
||||||
- **Uptime Kuma**: use a Kuma status page as an extra reachability signal.
|
- **Uptime Kuma**: use a Kuma status page as an extra reachability signal.
|
||||||
|
**Test connection** lists the monitors Inspector can see.
|
||||||
- **Dependencies** (`depends_on`): don't restart an app while the container it
|
- **Dependencies** (`depends_on`): don't restart an app while the container it
|
||||||
relies on (e.g. a VPN) is down.
|
relies on (e.g. a VPN) is down.
|
||||||
- **Portainer stack redeploy**: for containers that share a VPN container's
|
- **Portainer stack redeploy**: for containers that share a VPN container's
|
||||||
network, recover by redeploying the whole stack instead of restarting one
|
network, recover by redeploying the whole stack instead of restarting one
|
||||||
container.
|
container. Set up the Portainer connection under **Settings → Portainer**;
|
||||||
|
**Test connection** lists your stacks.
|
||||||
|
|
||||||
## Security
|
## Security
|
||||||
|
|
||||||
@@ -89,7 +93,8 @@ and that socket is effectively root on your server. So:
|
|||||||
|
|
||||||
- Keep the dashboard on your home network, or put it behind a VPN or a reverse
|
- Keep the dashboard on your home network, or put it behind a VPN or a reverse
|
||||||
proxy with authentication. Don't expose port 8080 to the internet.
|
proxy with authentication. Don't expose port 8080 to the internet.
|
||||||
- Use a real dashboard password.
|
- Use a real dashboard password. Repeated failed sign-ins from one address are
|
||||||
|
blocked for a few minutes.
|
||||||
- Inspector itself runs as an unprivileged user; it starts as root only to detect
|
- Inspector itself runs as an unprivileged user; it starts as root only to detect
|
||||||
the socket's permissions and fix ownership of its own folders.
|
the socket's permissions and fix ownership of its own folders.
|
||||||
|
|
||||||
|
|||||||
+6
-1
@@ -7,13 +7,17 @@
|
|||||||
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
|
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
|
||||||
# INSPECTOR_PASSWORD Dashboard password (username is "admin").
|
# INSPECTOR_PASSWORD Dashboard password (username is "admin").
|
||||||
# Only used on first start; change it later in Settings.
|
# Only used on first start; change it later in Settings.
|
||||||
# TZ Your timezone, e.g. America/Los_Angeles
|
# TZ Your time zone name, e.g. America/New_York, Europe/Berlin.
|
||||||
|
# Important: the maintenance window, alerts, dashboard and
|
||||||
|
# logs all use it. Left unset, everything is in UTC.
|
||||||
#
|
#
|
||||||
# Optional:
|
# Optional:
|
||||||
# INSPECTOR_PORT Port for the dashboard on your server (default 8080).
|
# INSPECTOR_PORT Port for the dashboard on your server (default 8080).
|
||||||
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
|
# INSPECTOR_TAG Image version to run (default "latest"; e.g. 0.8.0 to pin).
|
||||||
# PUID / PGID User/group Inspector runs as (default 1000/1000).
|
# PUID / PGID User/group Inspector runs as (default 1000/1000).
|
||||||
# Unraid users often prefer 99 / 100.
|
# Unraid users often prefer 99 / 100.
|
||||||
|
# PORTAINER_API_TOKEN Portainer access token, only if you use stack redeploy
|
||||||
|
# (Settings -> Portainer in the dashboard).
|
||||||
#
|
#
|
||||||
# No config file to create, no docker group ID to look up: Inspector creates
|
# No config file to create, no docker group ID to look up: Inspector creates
|
||||||
# its config on first start and detects Docker socket permissions by itself.
|
# its config on first start and detects Docker socket permissions by itself.
|
||||||
@@ -36,3 +40,4 @@ services:
|
|||||||
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
|
- INSPECTOR_PASSWORD=${INSPECTOR_PASSWORD:-}
|
||||||
- PUID=${PUID:-1000}
|
- PUID=${PUID:-1000}
|
||||||
- PGID=${PGID:-1000}
|
- PGID=${PGID:-1000}
|
||||||
|
- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}
|
||||||
Executable
BIN
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
+1
-1
@@ -1,5 +1,5 @@
|
|||||||
docker==7.1.0
|
docker==7.1.0
|
||||||
PyYAML==6.0.2
|
PyYAML==6.0.2
|
||||||
requests==2.32.3
|
requests==2.32.5
|
||||||
colorlog==6.9.0
|
colorlog==6.9.0
|
||||||
ruamel.yaml==0.18.6
|
ruamel.yaml==0.18.6
|
||||||
@@ -1 +1 @@
|
|||||||
__version__ = "0.8.0"
|
__version__ = "0.9.3"
|
||||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
@@ -78,10 +78,12 @@
|
|||||||
<a href="#status">3. The Status page</a>
|
<a href="#status">3. The Status page</a>
|
||||||
<a href="#notifications">4. Notifications</a>
|
<a href="#notifications">4. Notifications</a>
|
||||||
<a href="#containers">5. Managing containers</a>
|
<a href="#containers">5. Managing containers</a>
|
||||||
<a href="#events">6. Events history</a>
|
<a href="#vpn">6. Apps behind a VPN</a>
|
||||||
<a href="#settings">7. Other settings</a>
|
<a href="#portainer">7. Portainer stack redeploy</a>
|
||||||
<a href="#restart">8. "Requires restart" fields</a>
|
<a href="#events">8. Events history</a>
|
||||||
<a href="#trouble">9. Troubleshooting</a>
|
<a href="#settings">9. Other settings</a>
|
||||||
|
<a href="#restart">10. "Requires restart" fields</a>
|
||||||
|
<a href="#trouble">11. Troubleshooting</a>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
<main>
|
<main>
|
||||||
@@ -101,6 +103,8 @@ dashboard; you never need to touch a config file.</p>
|
|||||||
<i>Inspector dashboard login</i> with the username and password.</li>
|
<i>Inspector dashboard login</i> with the username and password.</li>
|
||||||
<li>Open <code>http://<your-server>:8080</code> and sign in.</li>
|
<li>Open <code>http://<your-server>:8080</code> and sign in.</li>
|
||||||
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
|
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
|
||||||
|
<li>Check the <b>time zone</b>: at the bottom of any Settings page it says
|
||||||
|
<code>time zone: …</code>. If that's not your zone, see <a href="#settings">Time zone</a>.</li>
|
||||||
</ol>
|
</ol>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
@@ -175,7 +179,7 @@ Each card shows:</p>
|
|||||||
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
|
under Apps). <b>Pushover</b>: enter your application API token and your user key.</li>
|
||||||
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
|
<li>Click <b>Send test</b>. You should get a message on your phone within seconds. If it fails,
|
||||||
the reason appears next to the button.</li>
|
the reason appears next to the button.</li>
|
||||||
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 8</a>),
|
<li>Click <b>Save</b>, then restart Inspector once (see <a href="#restart">section 10</a>),
|
||||||
because turning a channel on or off is a "requires restart" field.</li>
|
because turning a channel on or off is a "requires restart" field.</li>
|
||||||
</ol>
|
</ol>
|
||||||
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
|
<p>You'll get one alert when a container first fails, when Inspector restarts it, and when it
|
||||||
@@ -197,15 +201,182 @@ does <i>not</i> touch the container itself.</p>
|
|||||||
<tr><th>Status</th><th>When it happens</th></tr>
|
<tr><th>Status</th><th>When it happens</th></tr>
|
||||||
<tr><td>UNHEALTHY</td><td>The container's own health check is failing.</td></tr>
|
<tr><td>UNHEALTHY</td><td>The container's own health check is failing.</td></tr>
|
||||||
<tr><td>NOT RUNNING</td><td>Stopped, exited, or crashed.</td></tr>
|
<tr><td>NOT RUNNING</td><td>Stopped, exited, or crashed.</td></tr>
|
||||||
<tr><td>UNREACHABLE</td><td>Running, but an HTTP check failed (advanced, set in config.yml).</td></tr>
|
<tr><td>UNREACHABLE</td><td>Running, but its HTTP check or Uptime Kuma monitor says it's down (see below).</td></tr>
|
||||||
<tr><td>ERROR</td><td>Inspector couldn't check it (Docker API error).</td></tr>
|
<tr><td>ERROR</td><td>Inspector couldn't check it (Docker API error).</td></tr>
|
||||||
<tr><td>STARTING</td><td>Still in its health-check start period. Usually leave this off.</td></tr>
|
<tr><td>STARTING</td><td>Still in its health-check start period. Usually leave this off.</td></tr>
|
||||||
<tr><td>HEALTHY / NO HEALTH CHECK</td><td>Normal states; almost never a reason to restart.</td></tr>
|
<tr><td>HEALTHY / NO HEALTH CHECK</td><td>Normal states; almost never a reason to restart.</td></tr>
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
|
<h3>Reachability: catching apps that look fine but aren't</h3>
|
||||||
|
<p>Docker only knows whether a container is running and whether its own health check passes. An
|
||||||
|
app can pass both and still be unusable, for example when it routes through a VPN container
|
||||||
|
whose tunnel dropped. Under <b>Reachability</b> on the container page you can add:</p>
|
||||||
|
<ul>
|
||||||
|
<li><b>HTTP check</b>: a URL Inspector requests every check (for example
|
||||||
|
<code>http://192.168.1.10:9696/ping</code>). Set <b>expected_status</b> to the code the app
|
||||||
|
returns when it's healthy; most return 200, some return 401 or 302.</li>
|
||||||
|
<li><b>Uptime Kuma monitor</b>: if you already run Uptime Kuma, name one of its monitors. Set up
|
||||||
|
the connection first in <b>Settings → Uptime Kuma</b>; <b>Test connection</b> lists the
|
||||||
|
monitor names it can see.</li>
|
||||||
|
</ul>
|
||||||
|
<p>A failed check reports <b>UNREACHABLE</b>. Tick UNREACHABLE in the restart triggers if you want
|
||||||
|
Inspector to act on it.</p>
|
||||||
|
<figure><img src="container-advanced.png" alt="Reachability, Dependencies and Recovery groups">
|
||||||
|
<figcaption>Bottom of a container's page: an app behind a VPN container with an HTTP check and stack-redeploy recovery.</figcaption></figure>
|
||||||
|
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id="vpn">
|
||||||
|
<h2>6. Apps behind a VPN container</h2>
|
||||||
|
<h3>Why these apps need special handling</h3>
|
||||||
|
<p>A common setup routes download and indexer apps through a VPN container. Those apps don't get a
|
||||||
|
network of their own: their compose file says <code>network_mode: container:<vpn></code> (or
|
||||||
|
<code>service:<vpn></code>) and all their traffic goes through the VPN container. That causes
|
||||||
|
two problems for a watchdog:</p>
|
||||||
|
<ul>
|
||||||
|
<li><b>When the VPN breaks, every app behind it breaks at the same time.</b> Restarting the apps
|
||||||
|
does nothing while the tunnel is down. Without help, Inspector would restart each app over and
|
||||||
|
over, use up their hourly restart limits, and flood you with alerts, all for a problem in a
|
||||||
|
different container.</li>
|
||||||
|
<li><b>When the VPN container is recreated</b> (an image update, or someone redeploys its stack),
|
||||||
|
it gets a new container ID. The apps still point at the old one, so a normal restart fails with
|
||||||
|
errors like <i>"No such container"</i> or <i>"joining network namespace"</i>. The only fix is
|
||||||
|
to recreate the apps too, which is what a Portainer stack redeploy does
|
||||||
|
(<a href="#portainer">section 7</a>).</li>
|
||||||
|
</ul>
|
||||||
|
<h3>How to tell if an app is behind a VPN</h3>
|
||||||
|
<ul>
|
||||||
|
<li>In Portainer open <b>Stacks → your stack → Editor</b> and look for
|
||||||
|
<code>network_mode: container:…</code> or <code>network_mode: service:…</code> on the app.</li>
|
||||||
|
<li>Or open <b>Containers → the app</b>: under <b>Network</b> it shows the VPN container instead
|
||||||
|
of a normal network, and it has no ports of its own (they're published on the VPN container).</li>
|
||||||
|
</ul>
|
||||||
|
<h3>Setting it up</h3>
|
||||||
|
<ol class="steps">
|
||||||
|
<li><b>Add the VPN container to Inspector and keep it enabled</b> (restart on UNHEALTHY and NOT
|
||||||
|
RUNNING is a good start). Inspector can only wait for containers it is watching: if the VPN
|
||||||
|
container is disabled, its apps are recovered normally, as if there were no dependency, and the
|
||||||
|
container page shows a warning.</li>
|
||||||
|
<li>On each app behind it, set <b>Dependencies → depends_on</b> to the VPN container.</li>
|
||||||
|
<li>Turn on <b>vpn_routed</b>. This only changes the log: Inspector writes down when it holds off
|
||||||
|
on the app because the VPN is down. The waiting itself comes from depends_on.</li>
|
||||||
|
<li>Add an <b>HTTP check</b> (<a href="#containers">section 5</a>) using your server's IP and the
|
||||||
|
port published on the VPN container, and tick <b>UNREACHABLE</b>. Docker often still reports
|
||||||
|
these apps as healthy when the tunnel is down; the HTTP check notices.</li>
|
||||||
|
<li>Under <b>Recovery</b>, choose <b>Restart, then redeploy stack if that fails</b> and enter the
|
||||||
|
stack name. This needs the Portainer connection from <a href="#portainer">section 7</a>.</li>
|
||||||
|
</ol>
|
||||||
|
<h3>What Inspector then does</h3>
|
||||||
|
<p>Every check, it looks at the VPN container first. While the VPN is anything other than healthy
|
||||||
|
(or starting, or running without a health check), the apps that depend on it are not restarted;
|
||||||
|
Inspector fixes the VPN first. Once the VPN is healthy again, the apps are handled normally: a
|
||||||
|
restart first, and a stack redeploy if the restart fails or doesn't help.</p>
|
||||||
|
<p class="note"><b>depends_on isn't only for VPNs.</b> Use it for any "B needs A" pair, such as an
|
||||||
|
app and its database: Inspector won't restart the app while its database is down.</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id="portainer">
|
||||||
|
<h2>7. Portainer stack redeploy</h2>
|
||||||
|
<h3>Why Inspector needs Portainer</h3>
|
||||||
|
<p>Inspector talks to Docker directly, so it can restart or start a container. It can't
|
||||||
|
<i>recreate</i> one: that needs the container's compose definition, which lives in Portainer. A
|
||||||
|
stack redeploy asks Portainer to do exactly what its <b>Update the stack</b> button does. That's
|
||||||
|
how VPN-routed apps get reattached to a recreated VPN container. You only need this if you use one
|
||||||
|
of the two "redeploy" recovery actions.</p>
|
||||||
|
<p class="note warn"><b>A redeploy affects the whole stack.</b> Portainer pulls the latest images
|
||||||
|
and recreates <i>every</i> container in that stack, not just the failing app, so all of them blink
|
||||||
|
off for a moment. That's why <b>Restart, then redeploy stack if that fails</b> is the recommended
|
||||||
|
choice: most problems are fixed by the plain restart, and the redeploy is only used when that
|
||||||
|
doesn't work.</p>
|
||||||
|
<h3>Requirements</h3>
|
||||||
|
<ul>
|
||||||
|
<li>The app must belong to a stack that was <b>created in Portainer</b>. In <b>Stacks</b>, stacks
|
||||||
|
created elsewhere (for example with <code>docker compose</code> on the command line) are marked
|
||||||
|
<b>Limited</b>, and Portainer can't redeploy them.</li>
|
||||||
|
<li>An access token for a Portainer user who can manage that stack. An administrator's token is
|
||||||
|
the simplest.</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h3>Step 1: find the values</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Setting</th><th>Where to find it</th></tr>
|
||||||
|
<tr><td>base_url</td><td>The address you open Portainer at, without anything after the port:
|
||||||
|
usually <code>http://<server-ip>:9000</code> or <code>https://<server-ip>:9443</code>.
|
||||||
|
Use the server's IP, not <code>localhost</code>: Inspector runs in its own container.</td></tr>
|
||||||
|
<tr><td>endpoint_id</td><td>On Portainer's <b>Home</b> page, click your environment (often
|
||||||
|
<b>local</b>). The browser address then ends like <code>#!/2/docker/dashboard</code>: the
|
||||||
|
number after <code>#!/</code> is the endpoint ID (here <code>2</code>).</td></tr>
|
||||||
|
<tr><td>stack name</td><td><b>Stacks</b> list, <b>Name</b> column, spelled exactly the same.
|
||||||
|
This goes on each app's container page, not on the Portainer page.</td></tr>
|
||||||
|
<tr><td>verify_ssl</td><td>Leave off for <code>http://</code> and for Portainer's built-in
|
||||||
|
self-signed certificate on 9443. Turn on only if Portainer has a real certificate.</td></tr>
|
||||||
|
<tr><td>API token</td><td>Click your username (top right) → <b>My account</b> →
|
||||||
|
<b>Access tokens</b> → <b>Add access token</b>. Name it <code>inspector</code> and copy the
|
||||||
|
token straight away: Portainer shows it only once.</td></tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>Step 2: give Inspector the token</h3>
|
||||||
|
<p><b>Recommended: an environment variable.</b> The token then never lands in
|
||||||
|
<code>config.yml</code> or its backups.</p>
|
||||||
|
<ol class="steps">
|
||||||
|
<li>In Portainer open <b>Stacks → inspector → Editor</b>.</li>
|
||||||
|
<li>Check that the compose file has this line under <code>environment:</code> (it's in the
|
||||||
|
compose file from version 0.9.1 on; add it if your stack is older):
|
||||||
|
<code>- PORTAINER_API_TOKEN=${PORTAINER_API_TOKEN:-}</code></li>
|
||||||
|
<li>Under <b>Environment variables</b> click <b>Add an environment variable</b>: name
|
||||||
|
<code>PORTAINER_API_TOKEN</code>, value the token.</li>
|
||||||
|
<li>Click <b>Update the stack</b>. Inspector restarts with the token available.</li>
|
||||||
|
</ol>
|
||||||
|
<p><b>Alternative: save it in Inspector.</b> In the next step choose <b>Saved in config.yml</b>
|
||||||
|
and paste the token. It is masked in the dashboard, but anyone who can read Inspector's config
|
||||||
|
folder (or a backup of it) can read the token.</p>
|
||||||
|
|
||||||
|
<h3>Step 3: connect Inspector</h3>
|
||||||
|
<ol class="steps">
|
||||||
|
<li>Open <b>Settings → Portainer</b>. Under <b>Add environment</b>, give it a name (your own
|
||||||
|
label, e.g. <code>home</code>) and fill in base_url, endpoint_id and verify_ssl.</li>
|
||||||
|
<li><b>API token stored as</b>: <i>Environment variable</i> with the name
|
||||||
|
<code>PORTAINER_API_TOKEN</code>, or <i>Saved in config.yml</i> with the token.</li>
|
||||||
|
<li>Click <b>Test connection</b>. It should list the stacks on that endpoint; check that your
|
||||||
|
app's stack is there. Then <b>Save new environment</b>.</li>
|
||||||
|
<li>At the top of the page turn <b>enabled</b> on, pick the <b>default_environment</b>, and
|
||||||
|
<b>Save Portainer</b>.</li>
|
||||||
|
<li>On each app's container page, set the recovery action and stack name
|
||||||
|
(<a href="#vpn">section 6</a>). Leave <b>portainer_environment</b> on <i>Default</i> unless
|
||||||
|
you connected more than one Portainer.</li>
|
||||||
|
</ol>
|
||||||
|
<figure><img src="settings-portainer.png" alt="Portainer settings">
|
||||||
|
<figcaption>Settings → Portainer with one environment; the token comes from an environment variable.</figcaption></figure>
|
||||||
|
|
||||||
|
<h3>The other settings on the page</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Setting</th><th>What it does</th></tr>
|
||||||
|
<tr><td>default_environment</td><td>Which connection apps use when they don't pick one. With a
|
||||||
|
single environment you can leave it on Automatic.</td></tr>
|
||||||
|
<tr><td>stack_redeploy_cooldown_seconds</td><td>Minimum time between two redeploys of the same
|
||||||
|
stack (default 300). Several apps often share a stack; this stops them triggering a redeploy
|
||||||
|
each.</td></tr>
|
||||||
|
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector waits after a redeploy for the
|
||||||
|
stack to come up before judging it again (default 90).</td></tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>If Test connection fails</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Message</th><th>Fix</th></tr>
|
||||||
|
<tr><td>environment variable … is not set</td><td>The compose line from step 2 is missing, or
|
||||||
|
the stack wasn't updated after adding the variable.</td></tr>
|
||||||
|
<tr><td>the API token was rejected</td><td>Token copied wrong, deleted, or from a user without
|
||||||
|
access. Create a new one.</td></tr>
|
||||||
|
<tr><td>no stacks found on endpoint N</td><td>Wrong endpoint ID; check the number in the
|
||||||
|
Portainer address again.</td></tr>
|
||||||
|
<tr><td>request failed …</td><td>Wrong address or port, <code>http</code> vs <code>https</code>
|
||||||
|
mixed up, or <code>localhost</code> used instead of the server IP. With https and a
|
||||||
|
self-signed certificate, turn verify_ssl off.</td></tr>
|
||||||
|
</table>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="events">
|
<section id="events">
|
||||||
<h2>6. Events history</h2>
|
<h2>8. Events history</h2>
|
||||||
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
|
<p>The <b>Events</b> tab is a day-by-day timeline of failures, restarts, recoveries, mutes, and
|
||||||
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
||||||
<figure><img src="events.png" alt="Events timeline">
|
<figure><img src="events.png" alt="Events timeline">
|
||||||
@@ -213,7 +384,7 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
|||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="settings">
|
<section id="settings">
|
||||||
<h2>7. Other settings</h2>
|
<h2>9. Other settings</h2>
|
||||||
<figure><img src="settings-inspector.png" alt="Inspector loop settings">
|
<figure><img src="settings-inspector.png" alt="Inspector loop settings">
|
||||||
<figcaption>Settings → Inspector loop.</figcaption></figure>
|
<figcaption>Settings → Inspector loop.</figcaption></figure>
|
||||||
<table>
|
<table>
|
||||||
@@ -223,15 +394,33 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
|
|||||||
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
|
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
|
||||||
<tr><td>Event history</td><td>How many days of events to keep.</td></tr>
|
<tr><td>Event history</td><td>How many days of events to keep.</td></tr>
|
||||||
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
|
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
|
||||||
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
|
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00, in your <a href="#settings">time zone</a>) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
|
||||||
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
|
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
|
||||||
|
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
|
||||||
|
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#portainer">section 7</a>.</td></tr>
|
||||||
</table>
|
</table>
|
||||||
<p>At the bottom of every settings page is a strip showing where the config file lives and
|
<p>At the bottom of every settings page is a strip showing where the config file lives,
|
||||||
whether Inspector can write to it.</p>
|
whether Inspector can write to it, and which time zone it is using.</p>
|
||||||
|
|
||||||
|
<h3>Time zone</h3>
|
||||||
|
<p>Inspector uses the <code>TZ</code> setting of its Portainer stack for every time it shows or acts on:</p>
|
||||||
|
<ul>
|
||||||
|
<li><b>The maintenance window runs in this zone.</b> A 02:00–05:00 window with the wrong zone
|
||||||
|
runs hours off, so Inspector could restart containers during your backups and ignore
|
||||||
|
problems at other times.</li>
|
||||||
|
<li>Alert times (with the zone, e.g. <code>03:15 AM CET</code>), dashboard times, the Events
|
||||||
|
page's days (midnight to midnight), and logs.</li>
|
||||||
|
</ul>
|
||||||
|
<p>To change it: in Portainer open <b>Stacks → inspector → Editor</b>, set the <code>TZ</code>
|
||||||
|
environment variable to your zone's name, e.g. <code>America/New_York</code>,
|
||||||
|
<code>Europe/London</code> or <code>Australia/Sydney</code> (names from the
|
||||||
|
<a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">list of time zones</a>;
|
||||||
|
abbreviations like <code>EST</code> don't work), then click <b>Update the stack</b>. Without
|
||||||
|
<code>TZ</code>, Inspector uses UTC.</p>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="restart">
|
<section id="restart">
|
||||||
<h2>8. "Requires restart" fields</h2>
|
<h2>10. "Requires restart" fields</h2>
|
||||||
<p>Most changes apply on the next check. Fields marked
|
<p>Most changes apply on the next check. Fields marked
|
||||||
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
|
<span class="tag t-violet">REQUIRES RESTART</span> (like turning a notification channel or the
|
||||||
dashboard on or off) only take effect after Inspector restarts:</p>
|
dashboard on or off) only take effect after Inspector restarts:</p>
|
||||||
@@ -243,7 +432,7 @@ dashboard on or off) only take effect after Inspector restarts:</p>
|
|||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="trouble">
|
<section id="trouble">
|
||||||
<h2>9. Troubleshooting</h2>
|
<h2>11. Troubleshooting</h2>
|
||||||
<table>
|
<table>
|
||||||
<tr><th>Problem</th><th>Fix</th></tr>
|
<tr><th>Problem</th><th>Fix</th></tr>
|
||||||
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
|
<tr><td>Red <b>READ-ONLY</b> badge at the bottom of Settings</td>
|
||||||
|
|||||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 42 KiB |
+34
-4
@@ -1,10 +1,12 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import io
|
import io
|
||||||
|
import logging
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import threading
|
import threading
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import yaml
|
import yaml
|
||||||
@@ -23,6 +25,30 @@ VALID_STATUSES = {
|
|||||||
"UNREACHABLE",
|
"UNREACHABLE",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
log = logging.getLogger("inspector.config")
|
||||||
|
|
||||||
|
# Names logging.setLevel() accepts; anything else would crash at startup.
|
||||||
|
VALID_LOG_LEVELS = {"CRITICAL", "FATAL", "ERROR", "WARN", "WARNING", "INFO", "DEBUG", "NOTSET"}
|
||||||
|
|
||||||
|
|
||||||
|
def valid_hhmm(value: str) -> bool:
|
||||||
|
"""True if `value` parses the way the maintenance tracker parses it ("HH:MM")."""
|
||||||
|
try:
|
||||||
|
hour, minute = str(value).split(":")
|
||||||
|
time(int(hour), int(minute))
|
||||||
|
return True
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _startup_safe(value, default, ok: bool, key: str):
|
||||||
|
"""Return `value`, or `default` with a warning when it would crash Inspector."""
|
||||||
|
if ok:
|
||||||
|
return value
|
||||||
|
log.warning("config: invalid %s %r; using %r instead", key, value, default)
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
VALID_RECOVERY_ACTIONS = {
|
VALID_RECOVERY_ACTIONS = {
|
||||||
"restart_container",
|
"restart_container",
|
||||||
"start_container",
|
"start_container",
|
||||||
@@ -190,9 +216,11 @@ def build_config_from_raw(raw: dict) -> AppConfig:
|
|||||||
to validate an edited raw dict before writing it back to disk.
|
to validate an edited raw dict before writing it back to disk.
|
||||||
"""
|
"""
|
||||||
insp_raw = raw.get("inspector", {}) or {}
|
insp_raw = raw.get("inspector", {}) or {}
|
||||||
|
interval = int(insp_raw.get("interval_seconds", 60))
|
||||||
|
log_level = str(insp_raw.get("log_level", "INFO")).upper()
|
||||||
inspector = InspectorConfig(
|
inspector = InspectorConfig(
|
||||||
interval_seconds=int(insp_raw.get("interval_seconds", 60)),
|
interval_seconds=_startup_safe(interval, 1, interval >= 1, "inspector.interval_seconds"),
|
||||||
log_level=str(insp_raw.get("log_level", "INFO")).upper(),
|
log_level=_startup_safe(log_level, "INFO", log_level in VALID_LOG_LEVELS, "inspector.log_level"),
|
||||||
dry_run=bool(insp_raw.get("dry_run", False)),
|
dry_run=bool(insp_raw.get("dry_run", False)),
|
||||||
log_dir=str(insp_raw.get("log_dir", "/app/logs")),
|
log_dir=str(insp_raw.get("log_dir", "/app/logs")),
|
||||||
log_filename=str(insp_raw.get("log_filename", "inspector.log")),
|
log_filename=str(insp_raw.get("log_filename", "inspector.log")),
|
||||||
@@ -237,10 +265,12 @@ def build_config_from_raw(raw: dict) -> AppConfig:
|
|||||||
)
|
)
|
||||||
|
|
||||||
maint_raw = raw.get("maintenance") or {}
|
maint_raw = raw.get("maintenance") or {}
|
||||||
|
maint_start = str(maint_raw.get("start", "00:00"))
|
||||||
|
maint_end = str(maint_raw.get("end", "06:00"))
|
||||||
maintenance = MaintenanceConfig(
|
maintenance = MaintenanceConfig(
|
||||||
enabled=bool(maint_raw.get("enabled", False)),
|
enabled=bool(maint_raw.get("enabled", False)),
|
||||||
start=str(maint_raw.get("start", "00:00")),
|
start=_startup_safe(maint_start, "00:00", valid_hhmm(maint_start), "maintenance.start"),
|
||||||
end=str(maint_raw.get("end", "06:00")),
|
end=_startup_safe(maint_end, "06:00", valid_hhmm(maint_end), "maintenance.end"),
|
||||||
dir=str(maint_raw.get("dir", "/app/data/maintenance")),
|
dir=str(maint_raw.get("dir", "/app/data/maintenance")),
|
||||||
max_hours=int(maint_raw.get("max_hours", 8)),
|
max_hours=int(maint_raw.get("max_hours", 8)),
|
||||||
)
|
)
|
||||||
|
|||||||
+788
-77
File diff suppressed because it is too large.
Load diff
@@ -1,14 +1,49 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
import logging
|
||||||
from zoneinfo import ZoneInfo
|
import os
|
||||||
|
from datetime import datetime, timezone, tzinfo
|
||||||
|
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||||
|
|
||||||
|
|
||||||
PACIFIC = ZoneInfo("America/Los_Angeles")
|
log = logging.getLogger("inspector.formatting")
|
||||||
|
|
||||||
|
|
||||||
def format_timestamp(value: datetime | str | None) -> str:
|
def _resolve_local_tz(env: dict | None = None) -> tuple[tzinfo, str]:
|
||||||
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'.
|
"""The zone every displayed time uses, plus its name.
|
||||||
|
|
||||||
|
Follows the same rules as the C library (which drives log timestamps and
|
||||||
|
the maintenance window), so all times Inspector shows agree:
|
||||||
|
TZ set to a zone name -> that zone; TZ unset -> the container's
|
||||||
|
/etc/localtime; anything else -> UTC.
|
||||||
|
"""
|
||||||
|
env = os.environ if env is None else env
|
||||||
|
name = (env.get("TZ") or "").strip().lstrip(":")
|
||||||
|
if name:
|
||||||
|
try:
|
||||||
|
return ZoneInfo(name), name
|
||||||
|
except (ZoneInfoNotFoundError, ValueError):
|
||||||
|
log.warning(
|
||||||
|
"TZ=%r is not a known time zone name (e.g. Europe/Berlin); showing times in UTC",
|
||||||
|
name,
|
||||||
|
)
|
||||||
|
return timezone.utc, "UTC"
|
||||||
|
try:
|
||||||
|
with open("/etc/localtime", "rb") as fh:
|
||||||
|
zone = ZoneInfo.from_file(fh, key="localtime")
|
||||||
|
label = datetime.now(zone).tzname() or "localtime"
|
||||||
|
return zone, label
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return timezone.utc, "UTC"
|
||||||
|
|
||||||
|
|
||||||
|
# Resolved once at import; the container's TZ doesn't change while it runs.
|
||||||
|
LOCAL_TZ, LOCAL_TZ_NAME = _resolve_local_tz()
|
||||||
|
|
||||||
|
|
||||||
|
def format_timestamp(value: datetime | str | None, *, with_zone: bool = False) -> str:
|
||||||
|
"""Render a timestamp in the configured local zone, e.g. '06-14 03:45 PM'
|
||||||
|
(with_zone adds the abbreviation: '06-14 03:45 PM CET').
|
||||||
|
|
||||||
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
|
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
|
||||||
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
|
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
|
||||||
@@ -24,4 +59,8 @@ def format_timestamp(value: datetime | str | None) -> str:
|
|||||||
return value
|
return value
|
||||||
if value.tzinfo is None:
|
if value.tzinfo is None:
|
||||||
value = value.replace(tzinfo=timezone.utc)
|
value = value.replace(tzinfo=timezone.utc)
|
||||||
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p")
|
local = value.astimezone(LOCAL_TZ)
|
||||||
|
text = local.strftime("%m-%d %I:%M %p")
|
||||||
|
if with_zone:
|
||||||
|
text += f" {local.tzname()}"
|
||||||
|
return text
|
||||||
+20
-7
@@ -170,7 +170,7 @@ def _check_one(
|
|||||||
message=(
|
message=(
|
||||||
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
|
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
|
||||||
f"- Errors before recovery: **{state.error_count}**\n"
|
f"- Errors before recovery: **{state.error_count}**\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=3,
|
priority=3,
|
||||||
)
|
)
|
||||||
@@ -196,7 +196,7 @@ def _check_one(
|
|||||||
f"**{name}** is missing from Docker.\n\n"
|
f"**{name}** is missing from Docker.\n\n"
|
||||||
+ (f"- Detail: {result.message}\n" if result.message else "")
|
+ (f"- Detail: {result.message}\n" if result.message else "")
|
||||||
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
|
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=7,
|
priority=7,
|
||||||
)
|
)
|
||||||
@@ -240,7 +240,7 @@ def _check_one(
|
|||||||
+ ".\n\n"
|
+ ".\n\n"
|
||||||
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
|
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
|
||||||
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
|
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=5,
|
priority=5,
|
||||||
)
|
)
|
||||||
@@ -276,7 +276,7 @@ def _check_one(
|
|||||||
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
|
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
|
||||||
f"- Current status: **{result.status}**\n"
|
f"- Current status: **{result.status}**\n"
|
||||||
f"- Restarts paused until the hourly window clears\n"
|
f"- Restarts paused until the hourly window clears\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=9,
|
priority=9,
|
||||||
)
|
)
|
||||||
@@ -293,8 +293,10 @@ def _check_one(
|
|||||||
|
|
||||||
if ok:
|
if ok:
|
||||||
dep = container_config.depends_on
|
dep = container_config.depends_on
|
||||||
if dep:
|
# Only wait on a dependency Inspector is watching. A disabled one has
|
||||||
dep_status = states[dep].current_status if dep in states else None
|
# no state, and waiting on it would block recovery forever.
|
||||||
|
if dep and dep in states:
|
||||||
|
dep_status = states[dep].current_status
|
||||||
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
|
if dep_status not in (HEALTHY, STARTING, NO_HEALTH_CHECK):
|
||||||
if container_config.vpn_routed:
|
if container_config.vpn_routed:
|
||||||
log.info(
|
log.info(
|
||||||
@@ -417,6 +419,7 @@ def run_loop(config_ref: ConfigRef) -> int:
|
|||||||
|
|
||||||
check_order = _topological_order(enabled_configs)
|
check_order = _topological_order(enabled_configs)
|
||||||
portainer_client = PortainerClient(app.integrations.portainer)
|
portainer_client = PortainerClient(app.integrations.portainer)
|
||||||
|
integrations_cfg = app.integrations
|
||||||
stack_cooldowns: dict[str, datetime] = {}
|
stack_cooldowns: dict[str, datetime] = {}
|
||||||
|
|
||||||
dashboard = DashboardServer(
|
dashboard = DashboardServer(
|
||||||
@@ -453,8 +456,18 @@ def run_loop(config_ref: ConfigRef) -> int:
|
|||||||
for gone in states.keys() - current_enabled.keys():
|
for gone in states.keys() - current_enabled.keys():
|
||||||
states.pop(gone, None)
|
states.pop(gone, None)
|
||||||
log.info("Stopped watching container (removed or disabled): %s", gone)
|
log.info("Stopped watching container (removed or disabled): %s", gone)
|
||||||
|
if current_enabled != enabled_configs:
|
||||||
|
# Only re-order (and re-log ordering warnings) when the config changed.
|
||||||
|
check_order = _topological_order(current_enabled)
|
||||||
enabled_configs = current_enabled
|
enabled_configs = current_enabled
|
||||||
check_order = _topological_order(enabled_configs)
|
|
||||||
|
# Integration clients hold their config; rebuild them when the
|
||||||
|
# settings page changes Uptime Kuma / Portainer.
|
||||||
|
if app.integrations != integrations_cfg:
|
||||||
|
integrations_cfg = app.integrations
|
||||||
|
uptime_kuma_client = UptimeKumaClient(integrations_cfg.uptime_kuma)
|
||||||
|
portainer_client = PortainerClient(integrations_cfg.portainer)
|
||||||
|
log.info("Integration settings changed; reloaded Uptime Kuma / Portainer clients")
|
||||||
|
|
||||||
uptime_kuma_snapshot = uptime_kuma_client.fetch()
|
uptime_kuma_snapshot = uptime_kuma_client.fetch()
|
||||||
for name in check_order:
|
for name in check_order:
|
||||||
|
|||||||
@@ -33,9 +33,10 @@ class GotifyNotifier:
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
try:
|
try:
|
||||||
|
# Header, not ?token=, so the token stays out of access logs.
|
||||||
resp = requests.post(
|
resp = requests.post(
|
||||||
url,
|
url,
|
||||||
params={"token": self._config.token},
|
headers={"X-Gotify-Key": self._config.token},
|
||||||
json=payload,
|
json=payload,
|
||||||
timeout=self._timeout,
|
timeout=self._timeout,
|
||||||
)
|
)
|
||||||
|
|||||||
+15
-15
@@ -228,7 +228,7 @@ def _perform_restart_then_redeploy(
|
|||||||
f"**{result.status}** check(s).\n\n"
|
f"**{result.status}** check(s).\n\n"
|
||||||
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
|
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
|
||||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
|
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=6,
|
priority=6,
|
||||||
)
|
)
|
||||||
@@ -242,7 +242,7 @@ def _perform_restart_then_redeploy(
|
|||||||
state.restart_limit_notified = False
|
state.restart_limit_notified = False
|
||||||
log.info(
|
log.info(
|
||||||
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
|
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
|
||||||
width, name, state.cooldown_until.isoformat(),
|
width, name, format_timestamp(state.cooldown_until, with_zone=True),
|
||||||
)
|
)
|
||||||
eventstore.record(
|
eventstore.record(
|
||||||
container=name,
|
container=name,
|
||||||
@@ -250,15 +250,15 @@ def _perform_restart_then_redeploy(
|
|||||||
old_status=state.previous_status,
|
old_status=state.previous_status,
|
||||||
new_status=result.status,
|
new_status=result.status,
|
||||||
error_count=state.error_count,
|
error_count=state.error_count,
|
||||||
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
message=f"restart succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
|
||||||
)
|
)
|
||||||
notifier.notify(
|
notifier.notify(
|
||||||
title=f"✅ Inspector: {name} restart completed",
|
title=f"✅ Inspector: {name} restart completed",
|
||||||
message=(
|
message=(
|
||||||
f"**{name}** was successfully restarted by Docker.\n\n"
|
f"**{name}** was successfully restarted by Docker.\n\n"
|
||||||
f"- Previous status: *{result.status}*\n"
|
f"- Previous status: *{result.status}*\n"
|
||||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=3,
|
priority=3,
|
||||||
)
|
)
|
||||||
@@ -321,7 +321,7 @@ def _perform_docker_action(
|
|||||||
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
|
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
|
||||||
f"**{result.status}** check(s).\n\n"
|
f"**{result.status}** check(s).\n\n"
|
||||||
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
|
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=6,
|
priority=6,
|
||||||
)
|
)
|
||||||
@@ -335,22 +335,22 @@ def _perform_docker_action(
|
|||||||
state.record_restart(now, container_config.cooldown_seconds)
|
state.record_restart(now, container_config.cooldown_seconds)
|
||||||
state.last_notification_state = "RESTARTED"
|
state.last_notification_state = "RESTARTED"
|
||||||
state.restart_limit_notified = False
|
state.restart_limit_notified = False
|
||||||
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat())
|
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, format_timestamp(state.cooldown_until, with_zone=True))
|
||||||
eventstore.record(
|
eventstore.record(
|
||||||
container=name,
|
container=name,
|
||||||
event_type=EVENT_RESTART,
|
event_type=EVENT_RESTART,
|
||||||
old_status=state.previous_status,
|
old_status=state.previous_status,
|
||||||
new_status=result.status,
|
new_status=result.status,
|
||||||
error_count=state.error_count,
|
error_count=state.error_count,
|
||||||
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}",
|
message=f"{verb} succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
|
||||||
)
|
)
|
||||||
notifier.notify(
|
notifier.notify(
|
||||||
title=f"✅ Inspector: {name} {verb} completed",
|
title=f"✅ Inspector: {name} {verb} completed",
|
||||||
message=(
|
message=(
|
||||||
f"**{name}** was successfully {verb}ed by Docker.\n\n"
|
f"**{name}** was successfully {verb}ed by Docker.\n\n"
|
||||||
f"- Previous status: *{result.status}*\n"
|
f"- Previous status: *{result.status}*\n"
|
||||||
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
|
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=3,
|
priority=3,
|
||||||
)
|
)
|
||||||
@@ -362,7 +362,7 @@ def _perform_docker_action(
|
|||||||
f"{verb.capitalize()} of **{name}** failed.\n\n"
|
f"{verb.capitalize()} of **{name}** failed.\n\n"
|
||||||
f"- Status: **{result.status}**\n"
|
f"- Status: **{result.status}**\n"
|
||||||
f"- Error count: **{state.error_count}**\n"
|
f"- Error count: **{state.error_count}**\n"
|
||||||
f"- Time: {format_timestamp(now)}\n\n"
|
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
|
||||||
f"```\n{err}\n```"
|
f"```\n{err}\n```"
|
||||||
),
|
),
|
||||||
priority=9,
|
priority=9,
|
||||||
@@ -469,7 +469,7 @@ def _perform_portainer_redeploy(
|
|||||||
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
|
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
|
||||||
f"to recover **{name}**.\n\n"
|
f"to recover **{name}**.\n\n"
|
||||||
f"- Status: **{result.status}**\n"
|
f"- Status: **{result.status}**\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=6,
|
priority=6,
|
||||||
)
|
)
|
||||||
@@ -483,7 +483,7 @@ def _perform_portainer_redeploy(
|
|||||||
message=(
|
message=(
|
||||||
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
|
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
|
||||||
f"- Triggered by: **{name}** ({result.status})\n"
|
f"- Triggered by: **{name}** ({result.status})\n"
|
||||||
f"- Time: {format_timestamp(now)}\n\n"
|
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
|
||||||
f"```\n{redeploy_result.message}\n```"
|
f"```\n{redeploy_result.message}\n```"
|
||||||
),
|
),
|
||||||
priority=9,
|
priority=9,
|
||||||
@@ -516,7 +516,7 @@ def _perform_portainer_redeploy(
|
|||||||
f"successfully redeployed to recover **{name}**.\n\n"
|
f"successfully redeployed to recover **{name}**.\n\n"
|
||||||
f"- Triggered by: *{result.status}*\n"
|
f"- Triggered by: *{result.status}*\n"
|
||||||
f"- Post-redeploy wait: {wait_seconds}s\n"
|
f"- Post-redeploy wait: {wait_seconds}s\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=4,
|
priority=4,
|
||||||
)
|
)
|
||||||
@@ -536,7 +536,7 @@ def _perform_portainer_redeploy(
|
|||||||
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
|
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
|
||||||
f"for **{name}** still reports a failure.\n\n"
|
f"for **{name}** still reports a failure.\n\n"
|
||||||
f"- Recheck result: *{recheck.message}*\n"
|
f"- Recheck result: *{recheck.message}*\n"
|
||||||
f"- Time: {format_timestamp(now)}"
|
f"- Time: {format_timestamp(now, with_zone=True)}"
|
||||||
),
|
),
|
||||||
priority=6,
|
priority=6,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -41,20 +41,26 @@ class UptimeKumaClient:
|
|||||||
and bool(self._config.status_page_slug)
|
and bool(self._config.status_page_slug)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def _heartbeat_url(self) -> str:
|
||||||
|
return f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
|
||||||
|
|
||||||
def fetch(self) -> HeartbeatSnapshot:
|
def fetch(self) -> HeartbeatSnapshot:
|
||||||
if not self.enabled:
|
if not self.enabled:
|
||||||
return _EMPTY
|
return _EMPTY
|
||||||
|
|
||||||
url = f"{self._config.url}/api/status-page/heartbeat/{self._config.status_page_slug}"
|
|
||||||
try:
|
try:
|
||||||
resp = requests.get(url, timeout=self._timeout)
|
return self.fetch_or_raise()
|
||||||
resp.raise_for_status()
|
|
||||||
data = resp.json()
|
|
||||||
return self._parse(data)
|
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", url, exc)
|
log.warning("Uptime Kuma heartbeat fetch failed (%s): %s", self._heartbeat_url(), exc)
|
||||||
return _EMPTY
|
return _EMPTY
|
||||||
|
|
||||||
|
def fetch_or_raise(self) -> HeartbeatSnapshot:
|
||||||
|
"""Like fetch(), but lets request/parse errors propagate (used by the
|
||||||
|
settings page's "Test connection" button)."""
|
||||||
|
resp = requests.get(self._heartbeat_url(), timeout=self._timeout)
|
||||||
|
resp.raise_for_status()
|
||||||
|
return self._parse(resp.json())
|
||||||
|
|
||||||
def _parse(self, data: dict) -> HeartbeatSnapshot:
|
def _parse(self, data: dict) -> HeartbeatSnapshot:
|
||||||
# Expected shape:
|
# Expected shape:
|
||||||
# {"publicGroupList": [{"monitorList": [{"id": 1, "name": "..."}]}],
|
# {"publicGroupList": [{"monitorList": [{"id": 1, "name": "..."}]}],
|
||||||
|
|||||||
Reference in new issue
Block a user