Incus Server¶

Monitoring¶

updown.io sits at the top of the monitoring stack: it watches both Uptime Kuma and the Incus server via Pulse, a cron-based dead man's switch. Everything else in the infrastructure is monitored by Uptime Kuma itself. Alerts from updown.io are sent via SMS, since all self-hosted notification channels (email, etc.) could be down at the same time.

The Incus server sends a Pulse ping every hour; if the ping stops arriving, updown.io triggers an alert.

0 * * * * curl -sSo /dev/null -m 10 --retry 5 https://pulse.updown.io/<token>/<token>

systemd-networkd Configuration¶

When systemd-networkd restarts (whether triggered by needrestart, systemctl daemon-reexec, or anything else), it flushes routing state it considers "foreign", including Tailscale's ip policy rules and the throw <subnet> entries in routing table 52. This breaks container routing whenever an exit node is active.

Two [Network] settings are required to stop networkd from touching routing state it did not create:

• ManageForeignRoutingPolicyRules=no: prevents networkd from flushing Tailscale's ip policy rules at priorities 5210–5270 (see Post Mortem: Tailscale Subnet Route Lost)
• ManageForeignRoutes=no: prevents networkd from flushing the throw routes inside Tailscale's routing table 52 (see Post Mortem: Tailscale Throw Route Lost After needrestart Restarted networkd)

[Network]
ManageForeignRoutingPolicyRules=no
ManageForeignRoutes=no

With both settings in place, networkd will not disturb Tailscale's routing state on restart, regardless of what triggered the restart.

needrestart Configuration¶

The networkd settings above are the primary fix, but a needrestart blacklist remains as defense-in-depth so that routine upgrades do not bounce networking services unnecessarily. Two services must not be restarted automatically:

• systemd-networkd: avoid restart churn even though ManageForeign*=no now protects Tailscale routing state
• tailscaled: must only restart after systemd-networkd if at all, never automatically

$nrconf{blacklist_rc} = [
    qr(^systemd-networkd$),
    qr(^tailscaled$),
];

blacklist_rc tells needrestart to skip these services entirely. Restarts remain available via systemctl restart when genuinely needed.