Setup — Log Analyzer

Generate a sample log file and verify your parsing commands before writing the script.

Project Structure

mkdir -p ~/projects/log-analyzer
cd ~/projects/log-analyzer
touch log_analyzer.sh
chmod +x log_analyzer.sh

Generate a Sample Log File

Real log files can be hundreds of megabytes. Use this generator to create a realistic 1000-line sample:

cat > ~/projects/log-analyzer/gen_sample_log.sh << 'EOF'
#!/usr/bin/env bash
# Generates a realistic Apache Combined Log Format sample
set -euo pipefail

LINES="${1:-1000}"
OUTPUT="${2:-sample_access.log}"

IPS=("192.168.1.42" "10.0.0.5" "172.16.0.10" "192.168.1.99" "8.8.8.8"
     "1.2.3.4" "10.10.10.10" "192.0.2.1" "203.0.113.5" "198.51.100.2")
PATHS=("/index.html" "/api/v2/users" "/api/v2/products" "/static/app.js"
       "/static/style.css" "/login" "/logout" "/api/upload" "/favicon.ico" "/robots.txt")
METHODS=("GET" "GET" "GET" "GET" "GET" "POST" "POST" "PUT" "DELETE" "HEAD")
CODES=(200 200 200 200 200 200 304 301 404 404 500 403)
AGENTS=("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
        "curl/7.88.1" "python-requests/2.28.0" "Mozilla/5.0 (Windows NT 10.0)")

for (( i=1; i<=LINES; i++ )); do
    ip="${IPS[$((RANDOM % ${#IPS[@]}))]}"
    path="${PATHS[$((RANDOM % ${#PATHS[@]}))]}"
    method="${METHODS[$((RANDOM % ${#METHODS[@]}))]}"
    code="${CODES[$((RANDOM % ${#CODES[@]}))]}"
    bytes=$(( RANDOM % 50000 + 100 ))
    agent="${AGENTS[$((RANDOM % ${#AGENTS[@]}))]}"
    day=$(printf "%02d" $(( (RANDOM % 28) + 1 )))
    hour=$(printf "%02d" $(( RANDOM % 24 )))
    min=$(printf "%02d" $(( RANDOM % 60 )))
    sec=$(printf "%02d" $(( RANDOM % 60 )))
    echo "${ip} - - [${day}/Jan/2024:${hour}:${min}:${sec} +0000] \"${method} ${path} HTTP/1.1\" ${code} ${bytes} \"-\" \"${agent}\""
done > "$OUTPUT"

echo "Generated $LINES lines → $OUTPUT"
EOF
chmod +x gen_sample_log.sh
bash gen_sample_log.sh 2000 sample_access.log

Verify the Log Format

head -3 sample_access.log

192.168.1.42 - - [15/Jan/2024:09:30:00 +0000] "GET /index.html HTTP/1.1" 200 1234 "-" "Mozilla/5.0 ..." 10.0.0.5 - - [15/Jan/2024:09:30:01 +0000] "POST /api/v2/users HTTP/1.1" 201 89 "-" "curl/7.88.1" ...

Verify Each Parsing Command

Test each building block before combining them:

# Count total lines
wc -l sample_access.log

2000 sample_access.log

# Extract status codes and count each
awk '{print $9}' sample_access.log | sort | uniq -c | sort -rn

834 200 312 404 ...

# Extract unique IPs
awk '{print $1}' sample_access.log | sort -u | wc -l

10

# Extract top 5 requested paths
awk '{print $7}' sample_access.log | sort | uniq -c | sort -rn | head -5

218 /index.html 204 /api/v2/users ...

# Extract total bytes transferred
awk '{sum += $10} END {print sum}' sample_access.log

47291842

# Filter 5xx errors
awk '$9 ~ /^5/' sample_access.log | head -3

192.168.1.99 - - [15/Jan/2024:14:23:01 +0000] "POST /api/upload HTTP/1.1" 500 0 "-" "curl/7.88.1" ...

---

requirements | implementation