Most outages announce themselves hours in advance. A disk fills a little faster than usual, a certificate creeps toward expiry, or load climbs after a deploy. Automated server monitoring turns those quiet signals into a message you actually read, before a customer does. In this guide we build a practical, free, open-source stack for a small team, show working Prometheus alert rules, and then give an honest view of where AI helps and where it does not.
What to monitor first
Start with the four “golden signals” from Google’s SRE book: latency, traffic, errors and saturation. For a Linux web server, that means request time, requests per second, HTTP 5xx rate, and CPU, memory and I/O pressure.
However, the golden signals miss the slow failures that cause most SMB downtime. Add these to every host:
- Disk space and inode usage, with a prediction of when the disk fills, not just a static threshold.
- TLS certificate expiry, because a 30-day warning is cheap and an expired cert is not.
- Backup freshness: the age of the last successful backup, exported as a metric.
- Security events: failed SSH logins, sudo use, new listening ports and file changes in
/etc. - Reboot and patch state: pending security updates and uptime after a kernel update.
In short, measure what breaks, not what is easy to graph.
A realistic open-source stack for automated server monitoring
You do not need a SaaS bill to do this well. The stack below runs on one small VM and covers single hosts, fleets, external checks and logs.
| Layer | Tool | Why |
|---|---|---|
| Per-host, instant visibility | Netdata | One command, per-second metrics, built-in anomaly detection |
| Fleet metrics and alerting | Prometheus + node_exporter + Alertmanager | Standard, scriptable, huge ecosystem |
| Dashboards | Grafana | Reads Prometheus and Loki directly |
| External uptime checks | Uptime Kuma | HTTP, TCP, DNS and certificate checks from outside |
| Logs and security events | journald + Loki with Grafana Alloy, or Wazuh | Retention and search; Wazuh adds rules and FIM |
Netdata for quick per-host visibility
Netdata is the fastest way to see what a single box is doing. The official one-line installer, from the docs, is:
wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh && sh /tmp/netdata-kickstart.sh --release-channel stableThe dashboard listens on port 19999. Then bind it to a private address in /etc/netdata/netdata.conf under [web] with bind to = 127.0.0.1, and reach it over an SSH tunnel or a reverse proxy. Netdata ships hundreds of alerts by default, so it covers the first week with no configuration.
Prometheus, node_exporter and Alertmanager for fleets
Prometheus pulls metrics from exporters on each host and evaluates alert rules. Alertmanager groups and routes the alerts. The outline below follows the official “getting started” guide; we validated every config file with promtool 3.5 and amtool 0.28.
First, install node_exporter on every server. Ubuntu 24.04 has a packaged version:
sudo apt install prometheus-node-exporterBy default it listens on all interfaces on port 9100. Because metrics leak hostnames, mounts and kernel versions, restrict it to the private network. Create a systemd override with sudo systemctl edit prometheus-node-exporter and set the arguments:
[Service]
ExecStart=
ExecStart=/usr/bin/prometheus-node-exporter --web.listen-address=10.0.0.11:9100Next, install Prometheus and Alertmanager on the monitoring VM. The distribution packages lag behind, so use the official release tarballs. Replace the versions with the current ones from the releases pages:
PROM_VER=3.5.0
curl -LO https://github.com/prometheus/prometheus/releases/download/v${PROM_VER}/prometheus-${PROM_VER}.linux-amd64.tar.gz
tar xzf prometheus-${PROM_VER}.linux-amd64.tar.gz
sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo install -o root -g root prometheus-${PROM_VER}.linux-amd64/{prometheus,promtool} /usr/local/bin/
sudo mkdir -p /etc/prometheus/rules /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheusRepeat the same pattern for Alertmanager with its own alertmanager user and /var/lib/alertmanager data directory. Both projects publish example systemd units in their docs.
A minimal /etc/prometheus/prometheus.yml looks like this:
global:
scrape_interval: 30s
evaluation_interval: 30s
rule_files:
- /etc/prometheus/rules/*.yml
alerting:
alertmanagers:
- static_configs:
- targets: ["127.0.0.1:9093"]
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["127.0.0.1:9090"]
- job_name: node
static_configs:
- targets:
- "10.0.0.11:9100"
- "10.0.0.12:9100"
labels:
env: productionThree alert rules worth copying
Save this as /etc/prometheus/rules/node-alerts.yml. We checked it with promtool check rules and unit-tested each rule with promtool test rules against synthetic data, so the expressions fire as described.
groups:
- name: node
rules:
- alert: InstanceDown
expr: up{job="node"} == 0
for: 3m
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} is not responding"
description: "node_exporter on {{ $labels.instance }} has been unreachable for 3 minutes."
- alert: DiskWillFillIn24Hours
expr: |
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24 * 3600) < 0
and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.20
for: 30m
labels:
severity: warning
annotations:
summary: "Disk {{ $labels.mountpoint }} on {{ $labels.instance }} is filling up"
description: "At the current rate, {{ $labels.mountpoint }} runs out of space within 24 hours."
- alert: HighLoad
expr: node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1.5
for: 15m
labels:
severity: warning
annotations:
summary: "High load on {{ $labels.instance }}"
description: "5-minute load average is {{ $value | printf \"%.2f\" }} per CPU core."Two design notes. The disk rule uses predict_linear over six hours of history and only fires once free space is under 20 percent, which avoids noise on a half-empty disk. The load rule divides by core count, because a load of 8 is fine on 16 cores and a problem on 2.
Reload Prometheus after each change with sudo systemctl reload prometheus, or run promtool check config /etc/prometheus/prometheus.yml first.
Routing alerts to email and webhooks
Alertmanager decides who hears about what. This /etc/alertmanager/alertmanager.yml sends everything by email and additionally posts critical alerts to a chat webhook:
global:
smtp_smarthost: "smtp.example.com:587"
smtp_from: "alerts@example.com"
smtp_auth_username: "alerts@example.com"
smtp_auth_password_file: /etc/alertmanager/smtp_password
route:
receiver: ops-email
group_by: ["alertname", "instance"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers:
- severity = critical
receiver: ops-webhook
continue: true
receivers:
- name: ops-email
email_configs:
- to: "ops@example.com"
- name: ops-webhook
webhook_configs:
- url: "https://chat.example.com/hooks/abc123"Note the SMTP password lives in a separate file with chmod 600, owned by the alertmanager user. Never commit it with the config. Validate with amtool check-config alertmanager.yml.
Uptime Kuma for the outside view
Everything above runs inside your network, so a broken uplink or firewall rule goes unnoticed. Uptime Kuma, run from a cheap VM elsewhere, checks your public endpoints the way users see them. The official README install is a single Docker command:
docker run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:2Add HTTP keyword checks for each site, enable certificate expiry notifications, and point it at the same email or webhook as Alertmanager.
Log monitoring and CERT-In retention
Metrics tell you that something broke; logs tell you why. For a small fleet, keep journald persistent (Storage=persistent in /etc/systemd/journald.conf) and ship logs centrally. Grafana Loki stores them cheaply, and Grafana Alloy is now the official shipper, because Promtail reached end of life in March 2026 according to the Loki docs.
For security events specifically, Wazuh is a stronger fit. Its agent collects auth logs, runs file integrity monitoring and maps events to rules such as brute-force detection. It also produces the evidence you need for incident reporting.
That matters in India. CERT-In’s 2022 directions require covered entities to report specified incidents within six hours and to retain logs for a rolling 180 days within Indian jurisdiction. Set Loki or Wazuh retention to at least 180 days and test that a search for a date five months back still works. Our DPDP readiness check covers the related data-protection obligations.
Where AI actually helps, and where it does not
Marketing often sells “AI monitoring” as a replacement for thresholds. In practice, it is a useful layer on top of rule-based automated server monitoring, not a substitute.
Anomaly detection. Netdata trains small unsupervised models per metric on the agent itself. ML is on by default when the database mode is dbengine, and each data point carries an anomaly bit that rolls up into an anomaly rate per chart and per host. This helps with “something changed at 02:10” questions. With Prometheus, you get the same effect more crudely: compare a metric to its own value one week ago with offset 1w, or alert when a z-score against a rolling average exceeds 3.
LLM alert summarisation. Feeding an alert burst plus recent logs to a language model for a short summary can cut triage time at 3 a.m. Keep the model on read-only data, and keep humans on the decision.
The limits. Anomaly models need baselines, so a new server or a seasonal business looks “anomalous” for weeks. False positives are real; treat anomaly rate as a dashboard signal, not a pager. Finally, never let an AI component execute remediation on its own. Restarting services or scaling nodes belongs to tested runbooks with explicit approval, not to a probabilistic suggestion.
Securing the monitoring stack itself
A monitoring server knows every hostname, mount and version in your estate. Harden it like any other critical host:
- Bind exporters, Prometheus (
--web.listen-address=127.0.0.1:9090) and Alertmanager to private interfaces, and firewall ports 9090, 9093, 9100 and 19999 from the internet. - Put Grafana behind Nginx with TLS. Prometheus and Alertmanager support
--web.config.filewithbasic_auth_usersand bcrypt hashes; the official basic-auth guide shows how to generate them withpython3-bcrypt. - Change Grafana’s default
adminpassword on first login and disable anonymous access. - Run each service as its own unprivileged user, as in the install steps above.
- Patch the stack like everything else; our Nginx October 2026 patch guide shows the routine.
The free Linux server hardening checklist covers the remaining host basics.
Cost and sizing
Every tool in this automated server monitoring stack is free and open source. For up to about 50 hosts at a 30-second scrape interval, a 2 vCPU, 4 GB RAM VM with 100 GB of SSD is comfortable for Prometheus, Alertmanager and Grafana. Loki or Wazuh with 180-day retention needs its own disk, sized by log volume rather than host count. On Indian cloud providers that is a few thousand rupees a month, far less than one hour of downtime.
Frequently asked questions
Is automated server monitoring overkill for five servers?
No. Five servers still fill disks and expire certificates. Start with Netdata and Uptime Kuma, which take under an hour, and add Prometheus when you want alert rules that are versioned in Git.
Should I use Netdata or Prometheus?
Both, for different jobs. Netdata gives instant per-host detail and anomaly detection with no setup. Prometheus gives fleet-wide rules, long-term queries and integration with Grafana and Alertmanager.
How do I avoid alert fatigue?
Use for: durations so brief spikes do not page, group alerts by instance in Alertmanager, and route only critical to phones. Review every alert that fired in the last month and delete or tune the ones nobody acted on.
Can an AI tool fix problems automatically?
It can suggest, but it should not act unsupervised. Keep remediation in tested scripts triggered by a human or by narrow, deterministic rules, and log every action for later review.
What retention do I need for Indian compliance?
CERT-In’s 2022 directions set a rolling 180-day log retention requirement and a six-hour incident reporting window for covered entities. Confirm your obligations with the directions and your legal advisor.
Sources
- Prometheus: Getting started
- Prometheus: Alerting rules
- Prometheus: Securing endpoints with basic auth
- Alertmanager configuration reference
- Netdata: Install on Linux
- Netdata: ML configuration
- Uptime Kuma README
- Grafana Loki: Promtail end of life
- CERT-In directions under section 70B (April 2022)
Running Linux servers in production? Our Linux server hardening & security audit checks configuration, patching, SSH and logging. You can also use our free Linux server hardening checklist, or contact us.
