Requirements — Log Analyzer¶
What you will build and what you need before starting.
Learning Objectives¶
- Parse structured log files with
awk,grep,sed, andsort | uniq -c - Extract fields by position and pattern from Apache/Nginx access logs
- Generate a human-readable summary report from raw data
- Handle large files efficiently (streaming, no loading into memory)
- Write reusable analysis functions that can be composed
What You Will Build¶
log_analyzer.sh — a script that reads an Apache/Nginx access log and produces a summary report:
=== Log Analysis Report ===
File: /var/log/nginx/access.log
Period: 2024-01-15 00:00 – 2024-01-15 23:59
Lines: 48,293
── Traffic ─────────────────────────────
Total requests: 48,293
Unique IPs: 891
Total bytes: 2.3 GB
── Status Codes ─────────────────────────
200 OK 41,204 (85.3%)
304 Not Modified 3,102 (6.4%)
404 Not Found 2,891 (5.9%)
500 Server Error 196 (0.4%)
Other 900 (1.9%)
── Top 10 IPs ───────────────────────────
192.168.1.42 4,210 requests
10.0.0.5 2,891 requests
...
── Top 10 Requested Paths ───────────────
/api/v2/users 8,201 requests
/static/app.js 5,102 requests
...
── Top 10 User Agents ───────────────────
Mozilla/5.0 ... 22,103 requests
curl/7.88.1 1,204 requests
...
── Errors (5xx) ─────────────────────────
[2024-01-15 14:23:01] 192.168.1.99 "POST /api/upload" 500
...
Prerequisites¶
| Skill | Where to review |
|---|---|
awk field extraction |
Day 02 Part 2 — Stream Editing |
grep with regex |
Day 02 Part 1 — Text Processing |
sort, uniq -c |
Day 02 Part 1 — Text Processing |
| Pipes and redirection | Day 05 Part 1 — I/O & Pipes |
| Functions | Week 02 Day 01 Part 1 |
printf formatting |
Day 03 Part 2 — User Input & Expansion |
Apache Combined Log Format¶
Each line follows this pattern:
IP - USER [DATE TIME ZONE] "METHOD PATH PROTOCOL" STATUS BYTES "REFERER" "USER_AGENT"
Example:
192.168.1.42 - - [15/Jan/2024:09:30:00 +0000] "GET /index.html HTTP/1.1" 200 1234 "-" "Mozilla/5.0 ..."
Field positions for awk:
- $1 — client IP
- $4 — [date:time
- $7 — request path
- $9 — HTTP status code
- $10 — response bytes
Command-Line Interface¶
log_analyzer.sh [OPTIONS] LOGFILE
Options:
-n, --top N Show top N results (default: 10)
-o, --output FILE Write report to FILE instead of stdout
-s, --status CODE Filter analysis to a specific status code
-h, --help Show this help
Time Estimate¶
| Part | Time |
|---|---|
| Setup + sample log | 15 min |
| Traffic totals | 20 min |
| Status code breakdown | 20 min |
| Top IPs and paths | 25 min |
| Error listing | 15 min |
| Report formatting | 20 min |
| Total | ~115 min |