3 Commits
11 changed files with 268 additions and 61 deletions

No files matched your search

+62
View File
@@ -7,6 +7,68 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
---
## [0.9.5] — 2026-10-28
### Fixed
- **A stack redeploy no longer freezes Inspector.** After a successful Portainer
redeploy, Inspector used to stop everything for `post_redeploy_wait_seconds`
(default 90 s): no other container was checked, dashboard times stood still, and
stopping Inspector during the wait wasn't clean. Now the redeployed app is left to
settle while every other container keeps being checked, and its one-time HTTP
re-check (with the "recheck failed" alert) happens on its first check after the wait.
- The redeployed app's cooldown now starts after the settle time. Before, the wait
ate into it (with the defaults, 300 s cooldown became 210 s).
---
## [0.9.4] — 2026-10-28
### Fixed
- **Containers without a Docker health check now "recover" properly.** Many images
don't define a health check, so Docker reports them as `NO HEALTH CHECK` even when
they work. Inspector only treated `HEALTHY` as back to normal, so for these
containers:
- the error count never reset, and one-off failures days apart added up to a restart;
- no "recovered" alert was sent;
- with *Restart, then redeploy*, the next unrelated problem skipped the restart and
redeployed the whole stack.
Now `NO HEALTH CHECK` counts as working (unless you made it a restart trigger): the
count resets, the "recovered" alert is sent ("back to running (no health check)"),
and the next problem starts with a plain restart. Containers with a health check
behave exactly as before.
---
## [0.9.3] — 2026-10-28
### Changed
- **Every time follows the `TZ` of the Inspector stack.** Before, the dashboard,
Events page and alerts were always in US Pacific time while the maintenance window
and logs used `TZ`, so anyone outside Pacific saw times hours off (e.g. a 02:00
Berlin window shown as 5:00 PM). Now the dashboard, the Events page (including
where each day starts), alert times, cooldown times and logs all use `TZ`.
- Alert times include the zone abbreviation (`10-28 09:27 AM CET`), and "Cooldown
until" in alerts, event messages and logs is a local time instead of a UTC
timestamp.
- The active time zone is shown at the bottom of every Settings page, on the Events
page and on the Maintenance page.
- **If you never set `TZ`**, times are now shown in UTC instead of Pacific. Set `TZ`
in the stack (e.g. `America/New_York`). An unrecognized name logs a warning and
falls back to UTC.
### Docs
- INSTALL: `TZ` is now marked as needed, with a "Why the time zone matters" section
(maintenance window, alerts, dashboard, logs), how to find your zone name, and a
troubleshooting row. Guide: a Time zone section and a first-login check. Compose
and README updated.
---
## [0.9.2] — 2026-10-28
Fixes from a code and security review. None of them change how containers are
+29 -1
View File
@@ -52,12 +52,39 @@ TZ=America/Los_Angeles
|---|---|---|
| `INSPECTOR_DATA` | Folder on your server where Inspector keeps its settings, logs, and history. **Unraid:** `/mnt/user/appdata/inspector`. **Other Linux:** `/opt/inspector`. Created automatically. | Yes |
| `INSPECTOR_PASSWORD` | Password for the dashboard (username `admin`). Used only on first start; change it later in the dashboard. Leave it out and Inspector generates one and prints it in the container logs. | Recommended |
| `TZ` | Your [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), so times on the dashboard are local. | Recommended |
| `TZ` | **Your time zone**, e.g. `America/New_York` or `Europe/Berlin`. Without it Inspector uses UTC, so the maintenance window, alerts and dashboard times are off by your UTC offset. See [Why the time zone matters](#why-the-time-zone-matters). | **Yes** (default `UTC`) |
| `INSPECTOR_PORT` | Port on your server for the dashboard. Change it if `8080` is taken. | No (default `8080`) |
| `INSPECTOR_TAG` | Version to run, e.g. `0.8.0`. See [Updating](#4-updating). | No (default `latest`) |
| `PUID` / `PGID` | User and group that own Inspector's files. Unraid users often use `99` / `100` so the files match other appdata. | No (default `1000` / `1000`) |
| `PORTAINER_API_TOKEN` | A Portainer access token. Only needed if you use **Portainer stack redeploy** for VPN-routed apps; see the in-app guide (Help → Portainer). | No |
### Why the time zone matters
Inspector uses `TZ` for every time it shows or acts on. If it's wrong, things
still work, but at the wrong hour or with confusing times:
- **The maintenance window runs in this time zone.** A window of `02:00–05:00`
with `TZ` left at UTC runs at 02:00 UTC, which is the middle of the evening in
the Americas or mid-morning in Asia. Inspector would then restart containers
during your nightly backups, and ignore real problems during the day.
- **Alerts** (Gotify / Pushover) show the time of the problem in this zone, with
its abbreviation, e.g. `10-28 03:15 AM CET`.
- **The dashboard and Events page** show times in this zone, and the Events
page's days run midnight to midnight in this zone.
- **Logs** are stamped in this zone.
**How to set it:** use your zone's name from the
[list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
("TZ identifier" column), e.g. `America/New_York`, `America/Chicago`,
`Europe/London`, `Europe/Berlin`, `Australia/Sydney`. Abbreviations like `EST` or
`CET` don't work. On Unraid, **Settings → Date and Time** shows the zone your
server uses.
**How to check it:** the bottom of every Settings page shows
`time zone: <name>`. If it says `UTC` and you're not on UTC, set `TZ` in the
stack and click **Update the stack**. If the name you set isn't recognized,
Inspector logs a warning and uses UTC.
## 3. Deploy and sign in
1. Click **Deploy the stack**.
@@ -116,6 +143,7 @@ starts and restarts the ones you add to it.
| Logs say *Permission denied on /var/run/docker.sock* | Make sure the stack has no `user:` line, then update the stack. Inspector needs to start normally to set up socket access. |
| Red **READ-ONLY** badge at the bottom of Settings | The config folder mount ends in `:ro` or the folder isn't writable. Remove `:ro` and update the stack. |
| Can't sign in | Look for the generated password in the container logs. If you've lost it, open `INSPECTOR_DATA/config/config.yml` (e.g. via your Unraid share), set a new `password:` under `dashboard:`, and restart Inspector. |
| Times are a few hours off, or the maintenance window runs at the wrong time | `TZ` is missing or wrong. Check `time zone:` at the bottom of Settings, set `TZ` to your zone name (e.g. `Europe/London`), and update the stack. |
| A container shows **CONTAINER NOT FOUND** | The name in Inspector doesn't match Docker. Copy the exact name from Portainer → Containers. |
## Security notes
+2
View File
@@ -35,6 +35,8 @@ restarts them when they break, and tells your phone about it.</p>
- **Stays out of the way when you're working.** Put a container in maintenance or
mute its alerts with one click, or set a nightly maintenance window for backups
and updates.
- **Works in your time zone.** Set `TZ` in the stack and the maintenance window,
alerts, dashboard and logs all follow it.
## Install
+3 -1
View File
@@ -7,7 +7,9 @@
# Unraid: /mnt/user/appdata/inspector Other: /opt/inspector
# INSPECTOR_PASSWORD Dashboard password (username is "admin").
# Only used on first start; change it later in Settings.
# TZ Your timezone, e.g. America/Los_Angeles
# TZ Your time zone name, e.g. America/New_York, Europe/Berlin.
# Important: the maintenance window, alerts, dashboard and
# logs all use it. Left unset, everything is in UTC.
#
# Optional:
# INSPECTOR_PORT Port for the dashboard on your server (default 8080).
+1 -1
View File
@@ -1 +1 @@
__version__ = "0.9.2"
__version__ = "0.9.5"
+24 -5
View File
@@ -103,6 +103,8 @@ dashboard; you never need to touch a config file.</p>
<i>Inspector dashboard login</i> with the username and password.</li>
<li>Open <code>http://&lt;your-server&gt;:8080</code> and sign in.</li>
<li>Change the password under <b>Settings → Dashboard</b> if you like.</li>
<li>Check the <b>time zone</b>: at the bottom of any Settings page it says
<code>time zone: …</code>. If that's not your zone, see <a href="#settings">Time zone</a>.</li>
</ol>
</section>
@@ -354,8 +356,9 @@ folder (or a backup of it) can read the token.</p>
<tr><td>stack_redeploy_cooldown_seconds</td><td>Minimum time between two redeploys of the same
stack (default 300). Several apps often share a stack; this stops them triggering a redeploy
each.</td></tr>
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector waits after a redeploy for the
stack to come up before judging it again (default 90).</td></tr>
<tr><td>post_redeploy_wait_seconds</td><td>How long Inspector gives the stack to come up after
a redeploy before judging that app again (default 90). Everything else keeps being checked
meanwhile.</td></tr>
</table>
<h3>If Test connection fails</h3>
@@ -392,13 +395,29 @@ maintenance. Use <b>Prev day</b> / <b>Next day</b> to look back.</p>
<tr><td>Dashboard</td><td>Login username and password for this web UI.</td></tr>
<tr><td>Event history</td><td>How many days of events to keep.</td></tr>
<tr><td>Health summary</td><td>A periodic digest (e.g. daily) of what happened, sent through your notification channels.</td></tr>
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
<tr><td>Maintenance</td><td>A daily quiet window (e.g. 02:00–05:00, in your <a href="#settings">time zone</a>) so nightly backups or updates don't trigger restarts. Turn on <b>maintenance_window</b> on each container that should follow it.</td></tr>
<tr><td>Notification mute</td><td>Turns the per-container <b>Mute notifications</b> buttons on or off, and sets how long a mute lasts.</td></tr>
<tr><td>Uptime Kuma</td><td>Connection to your Uptime Kuma status page, used by container reachability checks.</td></tr>
<tr><td>Portainer</td><td>Connection to Portainer, used to redeploy stacks. See <a href="#portainer">section 7</a>.</td></tr>
</table>
<p>At the bottom of every settings page is a strip showing where the config file lives and
whether Inspector can write to it.</p>
<p>At the bottom of every settings page is a strip showing where the config file lives,
whether Inspector can write to it, and which time zone it is using.</p>
<h3>Time zone</h3>
<p>Inspector uses the <code>TZ</code> setting of its Portainer stack for every time it shows or acts on:</p>
<ul>
<li><b>The maintenance window runs in this zone.</b> A 02:00–05:00 window with the wrong zone
runs hours off, so Inspector could restart containers during your backups and ignore
problems at other times.</li>
<li>Alert times (with the zone, e.g. <code>03:15 AM CET</code>), dashboard times, the Events
page's days (midnight to midnight), and logs.</li>
</ul>
<p>To change it: in Portainer open <b>Stacks → inspector → Editor</b>, set the <code>TZ</code>
environment variable to your zone's name, e.g. <code>America/New_York</code>,
<code>Europe/London</code> or <code>Australia/Sydney</code> (names from the
<a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">list of time zones</a>;
abbreviations like <code>EST</code> don't work), then click <b>Update the stack</b>. Without
<code>TZ</code>, Inspector uses UTC.</p>
</section>
<section id="restart">
+18 -11
View File
@@ -30,7 +30,8 @@ from .config import (
valid_hhmm,
write_config_raw,
)
from .formatting import PACIFIC, format_timestamp
from . import formatting
from .formatting import format_timestamp
from .health import CONTAINER_NOT_FOUND, ERROR, HEALTHY, NOT_RUNNING, UNHEALTHY, UNREACHABLE
from .maintenance import MaintenanceTracker
from .notification_mute import NotificationMuteTracker
@@ -180,20 +181,20 @@ def _status_snapshot(
def _parse_date_param(value: str | None) -> date:
"""Parse a ?date=YYYY-MM-DD query param; falls back to "today" in
America/Los_Angeles on missing/invalid input."""
"""Parse a ?date=YYYY-MM-DD query param; falls back to "today" in the
configured time zone on missing/invalid input."""
if value:
try:
return date.fromisoformat(value)
except ValueError:
pass
return datetime.now(PACIFIC).date()
return datetime.now(formatting.LOCAL_TZ).date()
def _day_bounds_utc(day: date) -> tuple[datetime, datetime]:
"""UTC [start, end) bounds for a Pacific calendar day, extended one hour
"""UTC [start, end) bounds for a local calendar day, extended one hour
into the previous day for the lookback-overlap."""
midnight = datetime.combine(day, datetime.min.time(), tzinfo=PACIFIC)
midnight = datetime.combine(day, datetime.min.time(), tzinfo=formatting.LOCAL_TZ)
start = (midnight - timedelta(hours=1)).astimezone(timezone.utc)
end = (midnight + timedelta(days=1)).astimezone(timezone.utc)
return start, end
@@ -218,8 +219,8 @@ def _read_events_range(db_path: str, start: datetime, end: datetime) -> list[dic
def _hour_markers(day: date) -> list[datetime]:
"""24 Pacific-local hour boundaries for `day`, 00:00..23:00 ascending."""
midnight = datetime.combine(day, datetime.min.time(), tzinfo=PACIFIC)
"""24 local hour boundaries for `day`, 00:00..23:00 ascending."""
midnight = datetime.combine(day, datetime.min.time(), tzinfo=formatting.LOCAL_TZ)
return [midnight + timedelta(hours=h) for h in range(24)]
@@ -239,7 +240,7 @@ def _group_events_by_hour(
for ev in events:
try:
ts = datetime.fromisoformat(ev["ts"]).astimezone(PACIFIC)
ts = datetime.fromisoformat(ev["ts"]).astimezone(formatting.LOCAL_TZ)
except (ValueError, KeyError):
continue
if prev_hour_start <= ts < markers[0]:
@@ -1111,12 +1112,15 @@ def _render_events_page(
prev_day = (selected_day - timedelta(days=1)).isoformat()
next_day = (selected_day + timedelta(days=1)).isoformat()
today = datetime.now(PACIFIC).date()
today = datetime.now(formatting.LOCAL_TZ).date()
day_label = selected_day.strftime("%A, %B %-d, %Y")
parts.append('<div class="events-nav">\n')
parts.append(f'<a class="day-nav" href="/events?date={prev_day}">&larr; Prev day</a>\n')
parts.append(f'<span class="day-label">{_escape(day_label)}</span>\n')
parts.append(
f'<span class="day-label">{_escape(day_label)}'
f' <span style="color:var(--muted); font-weight:400; font-size:0.8em;">· times in {_escape(formatting.LOCAL_TZ_NAME)}</span></span>\n'
)
if selected_day >= today:
parts.append('<span class="day-nav disabled">Next day &rarr;</span>\n')
else:
@@ -2062,6 +2066,7 @@ def _render_diag_strip(config_ref: ConfigRef) -> str:
return (
'<div class="diag-strip">'
f'<span>config: <code>{_escape(str(path))}</code></span>'
f'<span>time zone: <code>{_escape(formatting.LOCAL_TZ_NAME)}</code></span>'
f"<span>{badge}{hint}</span>"
f'<span style="margin-left:auto;">Inspector v{__version__}</span>'
"</div>\n"
@@ -2201,6 +2206,8 @@ def _settings_body_for(section: str, app: AppConfig, csrf_token: str) -> str:
return (
'<h3 class="pane-title">Scheduled maintenance window</h3>'
'<p class="pane-sub">Daily quiet hours. Containers that opt in (<code>maintenance_window: true</code> in their config) are not acted on during the window.</p>'
f'<p class="pane-sub" style="margin-top:-1rem;">Times are in <b>{_escape(formatting.LOCAL_TZ_NAME)}</b>, '
"from the <code>TZ</code> setting of the Inspector stack. If that's not your time zone, the window runs at the wrong hours.</p>"
+ _settings_form("maintenance", "Maintenance", body, csrf_token)
)
+45 -6
View File
@@ -1,14 +1,49 @@
from __future__ import annotations
from datetime import datetime, timezone
from zoneinfo import ZoneInfo
import logging
import os
from datetime import datetime, timezone, tzinfo
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
PACIFIC = ZoneInfo("America/Los_Angeles")
log = logging.getLogger("inspector.formatting")
def format_timestamp(value: datetime | str | None) -> str:
"""Render a timestamp in US Pacific time, e.g. '06-14 03:45 PM'.
def _resolve_local_tz(env: dict | None = None) -> tuple[tzinfo, str]:
"""The zone every displayed time uses, plus its name.
Follows the same rules as the C library (which drives log timestamps and
the maintenance window), so all times Inspector shows agree:
TZ set to a zone name -> that zone; TZ unset -> the container's
/etc/localtime; anything else -> UTC.
"""
env = os.environ if env is None else env
name = (env.get("TZ") or "").strip().lstrip(":")
if name:
try:
return ZoneInfo(name), name
except (ZoneInfoNotFoundError, ValueError):
log.warning(
"TZ=%r is not a known time zone name (e.g. Europe/Berlin); showing times in UTC",
name,
)
return timezone.utc, "UTC"
try:
with open("/etc/localtime", "rb") as fh:
zone = ZoneInfo.from_file(fh, key="localtime")
label = datetime.now(zone).tzname() or "localtime"
return zone, label
except (OSError, ValueError):
return timezone.utc, "UTC"
# Resolved once at import; the container's TZ doesn't change while it runs.
LOCAL_TZ, LOCAL_TZ_NAME = _resolve_local_tz()
def format_timestamp(value: datetime | str | None, *, with_zone: bool = False) -> str:
"""Render a timestamp in the configured local zone, e.g. '06-14 03:45 PM'
(with_zone adds the abbreviation: '06-14 03:45 PM CET').
Accepts a datetime, an ISO 8601 string (as stored by EventStore /
ContainerState), or None/"" (returns ""). Naive datetimes/strings are
@@ -24,4 +59,8 @@ def format_timestamp(value: datetime | str | None) -> str:
return value
if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc)
return value.astimezone(PACIFIC).strftime("%m-%d %I:%M %p")
local = value.astimezone(LOCAL_TZ)
text = local.strftime("%m-%d %I:%M %p")
if with_zone:
text += f" {local.tzname()}"
return text
+36 -12
View File
@@ -41,7 +41,7 @@ from .maintenance import MaintenanceTracker
from .notification_mute import NotificationMuteTracker
from .notifier import GotifyNotifier, MultiNotifier, NullNotifier, PushoverNotifier
from .portainer_client import PortainerClient
from .recovery import perform_recovery
from .recovery import perform_recovery, run_post_redeploy_recheck
from .uptime_kuma import HeartbeatSnapshot, UptimeKumaClient
from .state import (
ContainerState,
@@ -63,6 +63,19 @@ def _request_stop(signum, _frame):
_stop = True
def _is_normal(status: str | None, container_config: ContainerConfig) -> bool:
"""True when `status` means the container is working.
HEALTHY always counts. NO HEALTH CHECK (running, but the image defines no
health check) counts too, unless the user made it a restart trigger.
Many images have no health check, and without this their error count and
restart episode would never reset.
"""
if status == HEALTHY:
return True
return status == NO_HEALTH_CHECK and NO_HEALTH_CHECK not in container_config.restart_on
def _apply_reachability_override(
container_config: ContainerConfig,
result: HealthResult,
@@ -140,6 +153,11 @@ def _check_one(
if notification_mute.is_muted(name):
notifier = NullNotifier()
if state.redeploy_recheck_at is not None and now >= state.redeploy_recheck_at:
run_post_redeploy_recheck(
container_config=container_config, state=state, notifier=notifier, now=now,
)
if state.previous_status != state.current_status:
log.info(
"%-*s status: %s -> %s%s",
@@ -158,19 +176,25 @@ def _check_one(
message=result.message,
)
is_normal = _is_normal(state.current_status, container_config)
if (
state.previous_status is not None
and state.previous_status != HEALTHY
and state.current_status == HEALTHY
and not _is_normal(state.previous_status, container_config)
and is_normal
# HEALTHY here means "a recovery was already announced", for either status.
and state.last_notification_state != HEALTHY
):
log.info("%-*s recovered: %s -> HEALTHY", width, name, state.previous_status)
log.info("%-*s recovered: %s -> %s", width, name, state.previous_status, state.current_status)
back_to = (
"**HEALTHY**" if state.current_status == HEALTHY
else "**running** (no health check)"
)
notifier.notify(
title=f"✅ Inspector: {name} recovered",
message=(
f"**{name}** is back to **HEALTHY** (was *{state.previous_status}*).\n\n"
f"**{name}** is back to {back_to} (was *{state.previous_status}*).\n\n"
f"- Errors before recovery: **{state.error_count}**\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=3,
)
@@ -178,7 +202,7 @@ def _check_one(
container=name,
event_type=EVENT_RECOVERY,
old_status=state.previous_status,
new_status=HEALTHY,
new_status=state.current_status,
error_count=state.error_count,
)
state.last_notification_state = HEALTHY
@@ -196,7 +220,7 @@ def _check_one(
f"**{name}** is missing from Docker.\n\n"
+ (f"- Detail: {result.message}\n" if result.message else "")
+ f"- Last known status: *{state.previous_status or 'unknown'}*\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=7,
)
@@ -240,7 +264,7 @@ def _check_one(
+ ".\n\n"
f"- Error count: **{state.error_count}/{container_config.max_error_count}**\n"
f"- Previous status: *{state.previous_status or 'unknown'}*\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=5,
)
@@ -253,9 +277,9 @@ def _check_one(
message=result.message,
)
state.last_notification_state = result.status
elif result.status == HEALTHY:
elif is_normal:
if state.error_count != 0:
log.info("%-*s healthy again; resetting error count", width, name)
log.info("%-*s back to normal; resetting error count", width, name)
state.error_count = 0
state.restarted_this_episode = False
@@ -276,7 +300,7 @@ def _check_one(
f"**{container_config.max_restarts_per_hour}** restart(s)/hour.\n\n"
f"- Current status: **{result.status}**\n"
f"- Restarts paused until the hourly window clears\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=9,
)
+46 -24
View File
@@ -1,7 +1,6 @@
from __future__ import annotations
import logging
import time
from datetime import datetime, timedelta
from .config import AppConfig, ContainerConfig, PortainerConfig
@@ -175,7 +174,8 @@ def _perform_restart_then_redeploy(
- the container is still failing on a later recovery cycle
(`state.restarted_this_episode` is already set), meaning the restart didn't stick.
The episode flag resets when the container is next seen HEALTHY (see main._check_one).
The episode flag resets when the container is next seen working: HEALTHY, or
NO HEALTH CHECK when that isn't a restart trigger (see main._is_normal).
"""
name = container_config.name
@@ -228,7 +228,7 @@ def _perform_restart_then_redeploy(
f"**{result.status}** check(s).\n\n"
f"- If it doesn't recover, Inspector escalates to a Portainer stack redeploy\n"
f"- Cooldown: **{container_config.cooldown_seconds}s** after restart\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
@@ -242,7 +242,7 @@ def _perform_restart_then_redeploy(
state.restart_limit_notified = False
log.info(
"%-*s restart succeeded; cooldown until %s (will redeploy if still failing)",
width, name, state.cooldown_until.isoformat(),
width, name, format_timestamp(state.cooldown_until, with_zone=True),
)
eventstore.record(
container=name,
@@ -250,15 +250,15 @@ def _perform_restart_then_redeploy(
old_status=state.previous_status,
new_status=result.status,
error_count=state.error_count,
message=f"restart succeeded; cooldown until {state.cooldown_until.isoformat()}",
message=f"restart succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
)
notifier.notify(
title=f"✅ Inspector: {name} restart completed",
message=(
f"**{name}** was successfully restarted by Docker.\n\n"
f"- Previous status: *{result.status}*\n"
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
f"- Time: {format_timestamp(now)}"
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=3,
)
@@ -321,7 +321,7 @@ def _perform_docker_action(
f"{verb.capitalize()}ing **{name}** after **{state.error_count}** consecutive "
f"**{result.status}** check(s).\n\n"
f"- Cooldown: **{container_config.cooldown_seconds}s** after {verb}\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
@@ -335,22 +335,22 @@ def _perform_docker_action(
state.record_restart(now, container_config.cooldown_seconds)
state.last_notification_state = "RESTARTED"
state.restart_limit_notified = False
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, state.cooldown_until.isoformat())
log.info("%-*s %s succeeded; cooldown until %s", width, name, verb, format_timestamp(state.cooldown_until, with_zone=True))
eventstore.record(
container=name,
event_type=EVENT_RESTART,
old_status=state.previous_status,
new_status=result.status,
error_count=state.error_count,
message=f"{verb} succeeded; cooldown until {state.cooldown_until.isoformat()}",
message=f"{verb} succeeded; cooldown until {format_timestamp(state.cooldown_until, with_zone=True)}",
)
notifier.notify(
title=f"✅ Inspector: {name} {verb} completed",
message=(
f"**{name}** was successfully {verb}ed by Docker.\n\n"
f"- Previous status: *{result.status}*\n"
f"- Cooldown until: {state.cooldown_until.isoformat()}\n"
f"- Time: {format_timestamp(now)}"
f"- Cooldown until: {format_timestamp(state.cooldown_until, with_zone=True)}\n"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=3,
)
@@ -362,7 +362,7 @@ def _perform_docker_action(
f"{verb.capitalize()} of **{name}** failed.\n\n"
f"- Status: **{result.status}**\n"
f"- Error count: **{state.error_count}**\n"
f"- Time: {format_timestamp(now)}\n\n"
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
f"```\n{err}\n```"
),
priority=9,
@@ -469,7 +469,7 @@ def _perform_portainer_redeploy(
f"Redeploying Portainer stack **{stack_name}** (environment **{environment}**) "
f"to recover **{name}**.\n\n"
f"- Status: **{result.status}**\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
@@ -483,7 +483,7 @@ def _perform_portainer_redeploy(
message=(
f"Portainer redeploy of **{stack_name}** (environment **{environment}**) failed.\n\n"
f"- Triggered by: **{name}** ({result.status})\n"
f"- Time: {format_timestamp(now)}\n\n"
f"- Time: {format_timestamp(now, with_zone=True)}\n\n"
f"```\n{redeploy_result.message}\n```"
),
priority=9,
@@ -516,29 +516,51 @@ def _perform_portainer_redeploy(
f"successfully redeployed to recover **{name}**.\n\n"
f"- Triggered by: *{result.status}*\n"
f"- Post-redeploy wait: {wait_seconds}s\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=4,
)
log.info("waiting %d seconds after stack redeploy", wait_seconds)
time.sleep(wait_seconds)
# Don't block the loop while the stack settles: keep this container in
# cooldown for the wait plus its normal cooldown, and do the one-time HTTP
# re-check on its first regular check after the wait (main._check_one).
log.info(
"%-*s letting stack %s settle for %ds; other containers keep being checked",
width, name, key, wait_seconds,
)
state.record_restart(now, wait_seconds + container_config.cooldown_seconds)
if container_config.http_check.enabled:
state.redeploy_recheck_at = now + timedelta(seconds=wait_seconds)
def run_post_redeploy_recheck(
*,
container_config: ContainerConfig,
state: ContainerState,
notifier: GotifyNotifier,
now: datetime,
) -> None:
"""One HTTP re-check after a stack redeploy has had time to settle.
Called by main._check_one once `state.redeploy_recheck_at` has passed;
alerts if the app still doesn't answer.
"""
state.redeploy_recheck_at = None
name = container_config.name
if not container_config.http_check.enabled:
return
recheck = check_http(container_config.http_check)
if recheck.ok:
log.info("recheck succeeded for %s", name)
else:
return
log.warning("recheck failed for %s: %s", name, recheck.message)
notifier.notify(
title=f"⚠️ Inspector: {name} recheck failed after redeploy",
message=(
f"Portainer stack **{stack_name}** was redeployed, but the HTTP recheck "
f"Portainer stack **{container_config.portainer_stack_name}** was redeployed, but the HTTP recheck "
f"for **{name}** still reports a failure.\n\n"
f"- Recheck result: *{recheck.message}*\n"
f"- Time: {format_timestamp(now)}"
f"- Time: {format_timestamp(now, with_zone=True)}"
),
priority=6,
)
state.record_restart(now, container_config.cooldown_seconds)
+2
View File
@@ -29,6 +29,8 @@ class ContainerState:
restart_limit_notified: bool = False
restarted_this_episode: bool = False
last_check_at: datetime | None = None
# After a stack redeploy: when to do the one-time HTTP re-check.
redeploy_recheck_at: datetime | None = None
def in_cooldown(self, now: datetime) -> bool:
return self.cooldown_until is not None and now < self.cooldown_until