My home lab fixed itself at 3 AM last week. I slept through the whole thing.
Here's what happened: Mr. Peepers, my AI agent, had a session file get corrupted during an overnight task. A Discord watchdog bot detected that he'd been disconnected for too long, SSHed into the server, identified the corrupted session files, cleared them, restarted his agent service, verified it came back healthy, and posted a summary to Discord.
By the time I woke up, there was a message waiting: "Session corruption detected and resolved at 3:14 AM. Service restored. No data loss."
This is what self-healing infrastructure looks like in a home lab. Not expensive enterprise tools — just watchdog scripts, health monitors, and an AI agent that knows how to fix its own problems.
The stack: systemd watchdogs, cron-based health checks, Uptime Kuma monitoring 50+ services, Discord webhooks for alerting, and an AI agent with SSH access to every host. When something breaks at 3 AM, the system handles it. When it can't, it escalates to Discord and I deal with it in the morning.
The goal isn't zero downtime. It's zero-wake-up downtime.
#DevOps #Automation #HomeLab #SelfHealing #Infrastructure
— Continue reading at https://dizydiz.com/blog/my-lab-fixes-itself-at-3-am.html