Enhancements — System Health Monitor¶

Once the core script works, these extensions turn it into a production-grade tool.

Enhancement 1 — Per-Filesystem Disk Checks¶

The basic version only checks /. Real servers have multiple mount points:

check_all_disks() {
    local alert_count=0
    while IFS= read -r line; do
        local mount pct
        mount=$(echo "$line" | awk '{print $6}')
        pct=$(echo "$line" | awk '{gsub(/%/,"",$5); print $5}')
        check_metric "Disk $mount" "$pct" "$THRESHOLD_DISK" "%" || (( alert_count++ )) || true
    done < <(df -h | tail -n +2 | grep -v tmpfs)
    return $alert_count
}

Enhancement 2 — Process-Level Monitoring¶

Alert when a specific process exceeds a CPU or memory threshold:

# Add to config:
# WATCH_PROCS="nginx mysql postgres"
# THRESHOLD_PROC_MEM=20

check_processes() {
    for proc in ${WATCH_PROCS:-}; do
        local mem_pct
        mem_pct=$(ps aux | awk -v p="$proc" '$0 ~ p && !/awk/ {sum += $4} END {print sum+0}')
        check_metric "Process $proc mem" "$mem_pct" "${THRESHOLD_PROC_MEM:-20}" "%" || true
    done
}

Enhancement 3 — HTTP Endpoint Check¶

Verify a web service is responding:

# Add to config: WATCH_URLS="http://localhost:8080/health http://localhost:3000"

check_urls() {
    for url in ${WATCH_URLS:-}; do
        local status
        status=$(curl -sf -o /dev/null -w "%{http_code}" --max-time 5 "$url" 2>/dev/null || echo "000")
        if [[ "$status" == "200" ]]; then
            log_info "URL OK: $url (HTTP $status)"
        else
            log_alert "URL FAIL: $url (HTTP $status)"
            send_alert "Endpoint $url returned HTTP $status"
        fi
    done
}

Write metrics to a CSV file for graphing or analysis:

METRICS_CSV="${SCRIPT_DIR}/logs/metrics.csv"

record_metrics() {
    local disk_pct="$1" mem_pct="$2" cpu_load="$3"
    # Write CSV header on first run
    [[ -f "$METRICS_CSV" ]] || echo "timestamp,disk_pct,mem_pct,cpu_load" > "$METRICS_CSV"
    echo "$(date +%s),${disk_pct},${mem_pct},${cpu_load}" >> "$METRICS_CSV"
}

Analyze trends:

# Average disk usage over the last 288 entries (24h at 5-min intervals)
tail -288 metrics.csv | awk -F, 'NR>1 {sum+=$2; n++} END {printf "Avg disk: %.1f%%\n", sum/n}'

Enhancement 5 — HTML Dashboard¶

Generate a simple HTML status page:

generate_html() {
    local disk_pct="$1" mem_pct="$2" cpu_load="$3"
    local status_color
    (( disk_pct > THRESHOLD_DISK || mem_pct > THRESHOLD_MEM )) && status_color="red" || status_color="green"

    cat > "${SCRIPT_DIR}/dashboard.html" << EOF
<!DOCTYPE html>
<html>
<head><title>Health Monitor — $(hostname)</title>
<meta http-equiv="refresh" content="60">
<style>body{font-family:monospace;background:#0f172a;color:#f1f5f9;padding:2em}
.ok{color:#2dd4bf}.alert{color:#f59e0b}</style></head>
<body>
<h1>$(hostname) — $(date '+%Y-%m-%d %H:%M:%S')</h1>
<p class="${disk_pct:-0} > ${THRESHOLD_DISK:-85} ? 'alert' : 'ok'">Disk: ${disk_pct}%</p>
<p>Memory: ${mem_pct}%</p>
<p>CPU load: ${cpu_load}</p>
</body></html>
EOF
    log_info "Dashboard updated: ${SCRIPT_DIR}/dashboard.html"
}

Serve it:

cd "${SCRIPT_DIR}"
python3 -m http.server 8080 &
# Now open http://localhost:8080/dashboard.html

Enhancement 6 — Multi-Server Monitoring via SSH¶

Run the check on remote servers from a central machine:

#!/usr/bin/env bash
# remote_check.sh — run health_check.sh on multiple servers

SERVERS="web01 web02 db01"
SCRIPT="/home/deploy/health_check.sh"

for server in $SERVERS; do
    echo "=== $server ==="
    ssh -o ConnectTimeout=5 "deploy@$server" "bash $SCRIPT" || \
        echo "ERROR: Could not reach $server"
done

testing | solution