Self-Healing Linux Server: systemd, Scripts and Alerts

A self-healing Linux server is not magic, and it is not “AI”. It is three plain things working together: automatic detection, safe automatic remediation, and an alert that a human will actually read. Most of it is already built into systemd. The rest is a small bash script, a timer and a webhook. In this guide we build the full stack layer by layer, with every unit file and script tested on Ubuntu 24.04 (systemd 255).

What a self-healing Linux server should and should not do

Self-healing means the server fixes the boring, well-understood failures on its own. For example, it restarts a crashed service, clears a known cache directory when the disk fills up, or rotates one oversized log. Then it tells you what it did and why.

Just as important is the list of things it must never do:

No blind deletion. Never rm -rf a path from a variable. Delete only specific, old files inside one known directory.
No automatic reboots of stateful services. Restarting PostgreSQL, MySQL or a message broker because a check failed can turn a slow query into data loss. Alert instead.
No silent actions. Every remediation writes to the journal and sends an alert, even if it succeeded.
No infinite retries. A cooldown and a circuit breaker stop the loop of “restart, crash, restart”.

With those rules in place, automation becomes something you can trust at 3 a.m. IST, and the audit trail also helps when CERT-In asks for logs from the last 180 days.

Layer 1: native systemd restarts and watchdogs

Before writing any script, use what systemd already offers. The example unit below shows the four settings that matter.

# /etc/systemd/system/myapp.service
[Unit]
Description=Example application with watchdog
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5
OnFailure=alert@%n.service

[Service]
Type=notify
User=myapp
Group=myapp
ExecStart=/opt/myapp/bin/myapp --serve
Restart=on-failure
RestartSec=5s
WatchdogSec=30s
LoadCredential=db-password:/etc/credstore/myapp-db-password
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
StateDirectory=myapp

[Install]
WantedBy=multi-user.target

Restart=on-failure restarts the process after a non-zero exit, a signal or a timeout, but not after a clean stop. RestartSec=5s adds a pause so a crashing app does not spin the CPU. StartLimitIntervalSec=300 and StartLimitBurst=5 live in the [Unit] section and mean “at most five starts in five minutes”. After that, systemd marks the unit failed and stops trying, which is exactly when a human should be paged.

WatchdogSec=30s handles the worst case: a process that is alive but hung. The application must call sd_notify("WATCHDOG=1") at least every 15 seconds (systemd exports the interval as WATCHDOG_USEC). If the ping stops, systemd kills and restarts the service. Most languages have a library for this: sdnotify for Python, github.com/coreos/go-systemd for Go, libsystemd for C.

Finally, OnFailure=alert@%n.service starts a templated alert unit whenever this unit fails. %n expands to the full unit name, so the alert knows who died.

The OnFailure alert unit

# /etc/systemd/system/alert@.service
[Unit]
Description=Send failure alert for %i

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/send-alert.sh "unit-failed" "%i" "Unit %i entered failed state. Runbook: https://wiki.example.in/runbooks/%i"
LoadCredential=webhook-url:/etc/credstore/alert-webhook-url

Because the webhook URL arrives through LoadCredential=, it never appears in the unit file or in ps output.

Layer 2: health checks, remediation and a circuit breaker

Restart policies only see process exits, so a self-healing Linux server needs a second layer. They cannot tell that nginx returns 502, that / is 95 percent full, or that a certificate expires on Sunday. For that we need a check script run by a systemd timer.

The remediation script

The script below checks HTTP health, disk space, memory and OOM events, and TLS expiry. It remediates only the first two, because those have safe, reversible fixes. A state directory records the last action per check and enforces a 10-minute cooldown plus a limit of three actions per hour.

#!/usr/bin/env bash
# /usr/local/sbin/healthcheck.sh - detect, remediate safely, alert.
set -euo pipefail

STATE_DIR="${STATE_DIRECTORY:-/var/lib/healthcheck}"
COOLDOWN=600; MAX_ACTIONS=3; WINDOW=3600
HTTP_URL="${HTTP_URL:-http://127.0.0.1:8080/healthz}"
APP_UNIT="${APP_UNIT:-myapp.service}"
DISK_MOUNT="${DISK_MOUNT:-/}"; DISK_LIMIT=90
CACHE_DIR="${CACHE_DIR:-/var/cache/myapp/tmp}"
BIG_LOG="${BIG_LOG:-/var/log/myapp/app.log}"; LOG_LIMIT_MB=512
MEM_FREE_MIN_MB=200
CERT_HOST="${CERT_HOST:-}"; CERT_WARN_DAYS=14
RUNBOOK="https://wiki.example.in/runbooks"
ALERT="${ALERT_CMD:-/usr/local/sbin/send-alert.sh}"
DRY_RUN="${DRY_RUN:-0}"

mkdir -p "$STATE_DIR"
log()   { logger -t self-heal -p "daemon.$1" -- "${*:2}"; echo "[$1] ${*:2}"; }
alert() { "$ALERT" "$1" "$2" "$3 Runbook: $RUNBOOK/$1" || true; }
run()   { if [[ "$DRY_RUN" == 1 ]]; then echo "DRY: $*"; else "$@"; fi; }

# Circuit breaker: allow a remediation for check $1 only outside cooldown
# and below MAX_ACTIONS per WINDOW. Otherwise escalate to humans.
allowed() {
  local name="$1" now last first count
  now=$(date +%s)
  last=$(cat "$STATE_DIR/$name.last" 2>/dev/null || echo 0)
  if [[ -f "$STATE_DIR/$name.window" ]]; then
    read -r first count < "$STATE_DIR/$name.window"
  else
    first=$now; count=0
  fi
  if (( now - first > WINDOW )); then first=$now; count=0; fi
  if (( now - last < COOLDOWN )); then
    log notice "$name: in cooldown, skipping"; return 1
  fi
  if (( count >= MAX_ACTIONS )); then
    log err "$name: circuit open ($count actions in ${WINDOW}s)"
    alert "circuit-open" "$name" "Remediation stopped after $count attempts on $(hostname)."
    return 1
  fi
  echo "$now" > "$STATE_DIR/$name.last"
  echo "$first $((count + 1))" > "$STATE_DIR/$name.window"
}

# 1. HTTP health -> restart the app unit
if ! curl -fsS --max-time 5 -o /dev/null "$HTTP_URL"; then
  log warning "http: $HTTP_URL failed"
  if allowed http; then
    run systemctl restart "$APP_UNIT"
    alert "http-restart" "$APP_UNIT" "Health URL failed; restarted $APP_UNIT on $(hostname)."
  fi
else
  log info "http: ok"
fi

# 2. Disk space -> purge ONE known cache dir, rotate ONE known log. Never rm -rf.
used=$(df --output=pcent "$DISK_MOUNT" | tail -1 | tr -dc '0-9')
if (( used >= DISK_LIMIT )); then
  log warning "disk: $DISK_MOUNT at ${used}%"
  if allowed disk; then
    [[ -d "$CACHE_DIR" ]] && run find "$CACHE_DIR" -xdev -type f -mtime +1 -delete
    if [[ -f "$BIG_LOG" ]] && (( $(stat -c %s "$BIG_LOG") > LOG_LIMIT_MB * 1024 * 1024 )); then
      run cp "$BIG_LOG" "$BIG_LOG.$(date +%Y%m%d%H%M%S)" && run truncate -s 0 "$BIG_LOG"
    fi
    alert "disk-cleanup" "$DISK_MOUNT" "Disk at ${used}%; cache purged, log rotated. Verify manually."
  fi
else
  log info "disk: ${used}% ok"
fi

# 3. Memory and OOM -> detect and alert only
free_mb=$(awk '/MemAvailable/ {print int($2/1024)}' /proc/meminfo)
if (( free_mb < MEM_FREE_MIN_MB )); then
  alert "low-memory" "$(hostname)" "MemAvailable ${free_mb}MB. Top: $(ps -eo rss,comm --sort=-rss | sed -n 2p)"
fi
if journalctl -k --since "-5min" -q 2>/dev/null | grep -q "Out of memory: Killed process"; then
  alert "oom-kill" "$(hostname)" "Kernel OOM killer fired in the last 5 minutes."
fi

# 4. TLS expiry -> alert; renewal stays with certbot
if [[ -n "$CERT_HOST" ]]; then
  end=$(echo | openssl s_client -servername "$CERT_HOST" -connect "$CERT_HOST:443" 2>/dev/null \
        | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2 || true)
  if [[ -n "$end" ]]; then
    days=$(( ($(date -d "$end" +%s) - $(date +%s)) / 86400 ))
    (( days < CERT_WARN_DAYS )) && alert "cert-expiry" "$CERT_HOST" "Certificate expires in ${days} days."
  else
    log err "cert: could not read certificate for $CERT_HOST"
  fi
fi

Set DRY_RUN=1 on the first run and read the output before you let it act for real. Note that the memory check never kills anything; the kernel’s OOM killer and systemd-oomd already do that, and a script guessing which process to kill is how outages get worse.

The service and timer pair

# /etc/systemd/system/healthcheck.service
[Unit]
Description=Self-healing health check and remediation
After=network-online.target
Wants=network-online.target
OnFailure=alert@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/healthcheck.sh
LoadCredential=webhook-url:/etc/credstore/alert-webhook-url
NoNewPrivileges=yes
ProtectHome=yes
StateDirectory=healthcheck
LogsDirectory=healthcheck

# /etc/systemd/system/healthcheck.timer
[Unit]
Description=Run self-healing health check every 2 minutes

[Timer]
OnBootSec=3min
OnUnitActiveSec=2min
AccuracySec=15s
RandomizedDelaySec=20s
Persistent=true

[Install]
WantedBy=timers.target

StateDirectory=healthcheck makes systemd create /var/lib/healthcheck and pass it to the script as $STATE_DIRECTORY, so the state files need no extra setup. Enable it with systemctl enable --now healthcheck.timer and confirm the schedule with systemctl list-timers.

Layer 3: alerting with a runbook link

An alert without context creates a second incident: someone searching for what to do. Every alert from our script therefore carries a kind, the affected object and a runbook URL. The sender posts generic JSON, which Slack, Microsoft Teams, Google Chat, Mattermost and most incident tools can accept through an incoming webhook or a small relay.

#!/usr/bin/env bash
# /usr/local/sbin/send-alert.sh <kind> <subject> <message>
set -euo pipefail
kind="${1:-unknown}"; subject="${2:-no-subject}"; message="${3:-}"
host="$(hostname -f 2>/dev/null || hostname)"
ts="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
mail_to="${ALERT_MAIL_TO:-}"

logger -t self-heal -p daemon.warning "ALERT kind=${kind} subject=${subject} msg=${message}"
payload=$(printf '{"host":"%s","time":"%s","kind":"%s","subject":"%s","message":"%s"}' \
  "$host" "$ts" "$kind" "$subject" "${message//\"/\\\"}")

if [[ -n "${CREDENTIALS_DIRECTORY:-}" && -r "${CREDENTIALS_DIRECTORY}/webhook-url" ]]; then
  url="$(<"${CREDENTIALS_DIRECTORY}/webhook-url")"
  curl -fsS --max-time 10 -H 'Content-Type: application/json' -d "$payload" "$url" \
    || logger -t self-heal -p daemon.err "webhook delivery failed for ${subject}"
fi

if [[ -n "$mail_to" ]] && command -v sendmail >/dev/null; then
  printf 'Subject: [%s] %s: %s\nTo: %s\n\n%s\n' "$host" "$kind" "$subject" "$mail_to" "$message" \
    | sendmail -t || logger -t self-heal -p daemon.err "mail delivery failed"
fi

Store the webhook URL with install -m 600 -o root -g root webhook.txt /etc/credstore/alert-webhook-url. For email, any local MTA that provides sendmail works, for example Postfix in satellite mode relaying through your provider. Check your sending domain first with our free email security checker, because an alert that lands in spam is no alert at all.

Layer 4: monitoring stacks, and where ML or an LLM honestly helps

Once you run more than a handful of servers, move the alert routing to a proper tool. Netdata gives per-second metrics and built-in health alarms with almost no configuration. Prometheus with Alertmanager gives you deduplication, grouping, silences and on-call routing. Both can call the same webhook format as above, so the runbooks you wrote still apply.

Where does machine learning fit? Honestly, in two narrow places. First, anomaly detection: Netdata ships unsupervised models that flag metrics that deviate from their own baseline, which catches “the disk is filling faster than usual” before a static 90 percent threshold would. Second, summarising: an LLM can turn forty near-identical alerts into one readable paragraph for the on-call engineer.

Where is it risky? Letting a model choose and run shell commands on production. A model that misreads a log line and issues systemctl stop postgresql has no cooldown, no state file and no idea what your DPDP Act obligations are. Keep the action list fixed and human-written; let the model only read and summarise.

Testing your self-healing Linux server

Validate the units before loading them, then simulate real failures:

sudo systemd-analyze verify /etc/systemd/system/myapp.service \
  /etc/systemd/system/alert@.service /etc/systemd/system/healthcheck.{service,timer}
sudo systemctl daemon-reload
sudo systemctl kill -s SIGKILL myapp.service      # simulate a crash
sleep 6 && systemctl is-active myapp.service       # expect: active
journalctl -u myapp.service -u alert@myapp.service --since -10min
sudo DRY_RUN=1 STATE_DIRECTORY=/tmp/hc /usr/local/sbin/healthcheck.sh

Also test the circuit breaker on purpose: kill the service six times in five minutes and confirm that systemd gives up, that alert@myapp.service fires, and that the message reaches your channel.

Security checklist for the automation itself

Scripts that restart services and delete files run as root, so the self-healing Linux server automation is itself a target. Keep them root:root with mode 750 under /usr/local/sbin, and keep secrets out of unit files by using LoadCredential=. Run the application itself under its own User=, and give the health check only what it needs: ProtectHome=yes and NoNewPrivileges=yes cost nothing. Keep the journal persistent (Storage=persistent in journald.conf) so your audit trail of every automated action survives a reboot. For the wider baseline, our Linux server hardening checklist is free, and recent incidents such as the nginx October 2026 vulnerabilities are a reminder that self-healing does not replace patching.

Frequently asked questions

Is a self-healing Linux server the same as “AI ops”?

No. The core is deterministic: systemd restart rules, a timed health check, fixed remediation actions and alerts. ML can add anomaly baselines and an LLM can summarise alerts, but neither should execute commands.

Why not just set Restart=always on everything?

Restart=always also restarts services you stopped deliberately, and without StartLimitBurst a broken config restarts forever. Use on-failure with a start limit and an OnFailure= alert, so a persistent fault reaches a human.

Should the script restart my database when a query check fails?

No. Databases recover best when left alone or handled by a DBA. Detect and alert, then follow a runbook. Automatic restarts are for stateless workers and web front-ends.

How do I stop alert flapping?

Use the cooldown and the per-hour action limit in the state directory, and set StartLimitIntervalSec on the unit. At scale, add Alertmanager grouping and inhibition rules.

Does this help with CERT-In or DPDP compliance?

Indirectly. Every automated action is logged to the journal with a timestamp, which supports CERT-In’s 180-day log retention and gives you evidence for incident timelines. Run our free DPDP readiness check for the wider picture.

Related guides

Sources

Running Linux servers in production? Our Linux server hardening & security audit checks configuration, patching, SSH and logging. You can also use our free Linux server hardening checklist, or contact us.

What a self-healing Linux server should and should not do

Layer 1: native systemd restarts and watchdogs

The OnFailure alert unit

Layer 2: health checks, remediation and a circuit breaker

The remediation script

The service and timer pair

Layer 3: alerting with a runbook link

Layer 4: monitoring stacks, and where ML or an LLM honestly helps

Testing your self-healing Linux server

Security checklist for the automation itself

Frequently asked questions

Is a self-healing Linux server the same as “AI ops”?

Why not just set Restart=always on everything?

Should the script restart my database when a query check fails?

How do I stop alert flapping?

Does this help with CERT-In or DPDP compliance?

Related guides

Sources

Bhaskar Soni

Leave a Reply Cancel reply

What a self-healing Linux server should and should not do

Layer 1: native systemd restarts and watchdogs

The OnFailure alert unit

Layer 2: health checks, remediation and a circuit breaker

The remediation script

The service and timer pair

Layer 3: alerting with a runbook link

Layer 4: monitoring stacks, and where ML or an LLM honestly helps

Testing your self-healing Linux server

Security checklist for the automation itself

Frequently asked questions

Is a self-healing Linux server the same as “AI ops”?

Why not just set Restart=always on everything?

Should the script restart my database when a query check fails?

How do I stop alert flapping?

Does this help with CERT-In or DPDP compliance?

Related guides

Sources

Please Share This Share this content

Bhaskar Soni

You Might Also Like

Tailscale SSH Vulnerability: Root Risk, Fixes and What to Do

How to Set Up an SSL Certificate for Nginx: Let’s Encrypt and OpenSSL

10 Reasons to Use Linux in 2026

Secure Nginx Configuration for Production: Headers, TLS, Limits

Automate Server Monitoring: Metrics, Alerts and Where AI Helps

Node.js October 2026 Security Release: Node2Shell and 9 More Fixes

Leave a Reply Cancel reply

Share this content