Implementation — System Health Monitor

Build the script incrementally. Each step is runnable and testable on its own.

Step 1 — Scaffold and Config Loading

Start with the header, set -euo pipefail, and config loading:

#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CONFIG="${SCRIPT_DIR}/health_check.conf"

# Load config (sets THRESHOLD_DISK, THRESHOLD_MEM, etc.)
[[ -f "$CONFIG" ]] && source "$CONFIG"

# Defaults if config not present
THRESHOLD_DISK="${THRESHOLD_DISK:-85}"
THRESHOLD_MEM="${THRESHOLD_MEM:-80}"
THRESHOLD_CPU="${THRESHOLD_CPU:-4.0}"
LOGFILE="${LOGFILE:-/tmp/health_check.log}"

Test: bash -n health_check.sh — no syntax errors. ✓

Step 2 — Logging Functions

Add structured logging before any metric code:

log() {
    local level="$1"; shift
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $*" | tee -a "$LOGFILE"
}

log_info()  { log "INFO " "$@"; }
log_warn()  { log "WARN " "$@"; }
log_alert() { log "ALERT" "$@"; }

Test:

source health_check.sh   # won't work yet — add main() call guard first
# Instead, add this temporarily at the bottom:
log_info "Logging works"
bash health_check.sh

Step 3 — Metric Collection Functions

Each function collects one metric and prints a plain number:

get_disk_pct() {
    df / | awk 'NR==2 { gsub(/%/,"",$5); print $5 }'
}

get_mem_pct() {
    free | awk '/^Mem:/ { printf "%.0f", $3/$2*100 }'
}

get_cpu_load() {
    uptime | awk -F'load average:' '{ print $2 }' | awk '{ print $1 }' | tr -d ','
}

Test each function in isolation:

bash -c 'source health_check.sh; get_disk_pct'
bash -c 'source health_check.sh; get_mem_pct'
bash -c 'source health_check.sh; get_cpu_load'

Test functions independently

Wrapping everything in functions lets you test each piece without running the entire script. This is the practical value of modular design.

Step 4 — Threshold Checks

Add a check_metric function that compares a value against a threshold:

check_metric() {
    local name="$1"
    local value="$2"
    local threshold="$3"
    local unit="${4:-%}"

    # Use awk for float comparison (bash can't compare decimals)
    local breached
    breached=$(awk -v v="$value" -v t="$threshold" 'BEGIN { print (v+0 > t+0) ? "yes" : "no" }')

    if [[ "$breached" == "yes" ]]; then
        log_alert "$name at ${value}${unit} — threshold ${threshold}${unit} exceeded"
        return 1
    else
        log_info "$name OK: ${value}${unit} (threshold: ${threshold}${unit})"
        return 0
    fi
}

Why awk for float comparison

Bash arithmetic only handles integers. (( 4.5 > 4.0 )) is a syntax error. Use awk or bc whenever you need to compare decimal numbers.

Step 5 — Alert Function

Add an optional Slack alert (fires only if SLACK_WEBHOOK_URL is set):

send_alert() {
    local message="$1"
    [[ -n "${SLACK_WEBHOOK_URL:-}" ]] || return 0

    curl -sf -X POST "$SLACK_WEBHOOK_URL" \
         -H "Content-Type: application/json" \
         -d "{\"text\": \"[$(hostname)] ALERT: $message\"}" \
         >/dev/null
}

Step 6 — Log Rotation

Prevent the log file from growing forever:

rotate_log() {
    local max_lines="${LOG_MAX_LINES:-1000}"
    [[ -f "$LOGFILE" ]] || return 0
    local current_lines
    current_lines=$(wc -l < "$LOGFILE")
    if (( current_lines > max_lines )); then
        tail -n "$max_lines" "$LOGFILE" > "${LOGFILE}.tmp"
        mv "${LOGFILE}.tmp" "$LOGFILE"
    fi
}

Step 7 — The main() Function

Wire everything together:

main() {
    local hostname
    hostname=$(hostname)
    local alerts=0

    log_info "=== Health check started on $hostname ==="

    # Disk
    local disk_pct
    disk_pct=$(get_disk_pct)
    check_metric "Disk /" "$disk_pct" "$THRESHOLD_DISK" "%" || {
        (( alerts++ ))
        send_alert "Disk / at ${disk_pct}%"
    }

    # Memory
    local mem_pct
    mem_pct=$(get_mem_pct)
    check_metric "Memory" "$mem_pct" "$THRESHOLD_MEM" "%" || {
        (( alerts++ ))
        send_alert "Memory at ${mem_pct}%"
    }

    # CPU load
    local cpu_load
    cpu_load=$(get_cpu_load)
    check_metric "CPU load" "$cpu_load" "$THRESHOLD_CPU" "" || {
        (( alerts++ ))
        send_alert "CPU load at $cpu_load"
    }

    if (( alerts == 0 )); then
        log_info "=== All systems normal ==="
    else
        log_warn "=== $alerts alert(s) triggered ==="
    fi

    rotate_log
}

main "$@"

Step 8 — Schedule with cron

# Edit your crontab
crontab -e

Add this line (adjust the path to match your setup):

*/5 * * * * /home/user/projects/health-monitor/health_check.sh

Verify the cron job was added:

crontab -l

Use absolute paths in cron

cron does not load your ~/.bashrc or $PATH. Use full absolute paths: /home/user/projects/..., not ~/projects/....

---

setup | testing