Networking & APIs from the Shell¶
curl is the command-line HTTP client — with it you can call any REST API, download files, and automate web interactions without opening a browser.
Learning Objectives¶
- Download files and make HTTP requests with
curlandwget - Make REST API calls and parse JSON responses with
jq - Use SSH for remote access and file transfer with
scp - Handle authentication (API keys, basic auth, tokens)
curl — Transfer Data with URLs¶
# Basic GET request
curl https://api.github.com/users/KirkYagami
# Download a file
curl -O https://example.com/file.tar.gz # save with original filename
curl -o myfile.tar.gz https://example.com/file # save with custom filename
curl -L -O https://example.com/redirect # follow redirects
# Show headers only
curl -I https://example.com
# POST request
curl -X POST \
-H "Content-Type: application/json" \
-d '{"name": "test", "value": 42}' \
https://api.example.com/endpoint
Common curl flags:
| Flag | Meaning |
|---|---|
-s |
Silent (no progress bar) |
-S |
Show errors even with -s |
-L |
Follow redirects |
-o FILE |
Write output to file |
-O |
Save with server filename |
-H "Header: Value" |
Add request header |
-d DATA |
POST body data |
-u user:pass |
Basic auth |
-f |
Fail silently on HTTP errors (returns non-zero exit code) |
Authentication¶
# API key in header
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/data
# API key as query parameter
curl "https://api.example.com/data?api_key=$API_KEY"
# Basic auth
curl -u "username:password" https://api.example.com/
# Store credentials in ~/.netrc (safer than command line)
cat ~/.netrc
# machine api.example.com login myuser password mypass
Never hardcode credentials
Do not put API keys or passwords directly in scripts — they end up in git history, ps aux output, and shell history. Use environment variables or a credentials file (~/.netrc, a vault tool).
Parsing JSON with jq¶
jq is a command-line JSON processor:
# Pretty-print JSON
curl -s https://api.github.com/users/KirkYagami | jq '.'
# Extract a field
curl -s https://api.github.com/users/KirkYagami | jq '.login'
# "KirkYagami"
# Extract nested field
curl -s https://api.github.com/repos/KirkYagami/shell-scripting-for-beginners \
| jq '.stargazers_count'
# Extract from an array
curl -s https://api.github.com/users/KirkYagami/repos \
| jq '.[].name'
# Multiple fields
curl -s https://api.github.com/users/KirkYagami \
| jq '{login: .login, repos: .public_repos, followers: .followers}'
jq in a Script¶
#!/usr/bin/env bash
set -euo pipefail
API_URL="https://wttr.in/London?format=j1"
response=$(curl -sf "$API_URL")
temp_c=$(echo "$response" | jq -r '.current_condition[0].temp_C')
desc=$(echo "$response" | jq -r '.current_condition[0].weatherDesc[0].value')
echo "London: ${temp_c}°C — $desc"
wget — Downloading Files¶
wget is simpler than curl for bulk downloading:
wget https://example.com/file.tar.gz # download file
wget -q https://example.com/file.tar.gz # quiet mode
wget -r -l2 https://example.com/docs/ # recursive download, depth 2
wget -c https://example.com/largefile.tar.gz # resume interrupted download
SSH Basics¶
ssh user@hostname # connect to remote host
ssh -p 2222 user@hostname # custom port
ssh -i ~/.ssh/id_ed25519 user@host # specify key file
# Run a command without an interactive shell
ssh user@host "ls -la ~/scripts/"
# SSH key generation (do this once)
ssh-keygen -t ed25519 -C "your@email.com"
ssh-copy-id user@hostname # install your public key on remote
scp — Secure Copy¶
scp file.txt user@host:~/ # copy to remote home
scp user@host:~/file.txt . # copy from remote
scp -r local_dir/ user@host:~/ # copy directory recursively
Building a Simple Alerting Script¶
#!/usr/bin/env bash
set -euo pipefail
WEBHOOK_URL="${SLACK_WEBHOOK_URL:?SLACK_WEBHOOK_URL must be set}"
HOSTNAME=$(hostname)
MESSAGE="$*"
curl -sf -X POST "$WEBHOOK_URL" \
-H "Content-Type: application/json" \
-d "{\"text\": \"[$HOSTNAME] $MESSAGE\"}"
Practice Exercises¶
Main (write a short script)¶
Create ~/scripts/api_check.sh that monitors an HTTP endpoint:
#!/usr/bin/env bash
set -euo pipefail
URL="${1:?Usage: $0 <url>}"
LOGFILE="/tmp/api_check.log"
log() { echo "[$(date '+%H:%M:%S')] $*" | tee -a "$LOGFILE"; }
response=$(curl -sf -o /dev/null -w "%{http_code} %{time_total}" "$URL" 2>/dev/null || echo "000 0")
status_code="${response%% *}"
response_time="${response##* }"
if [[ "$status_code" == "200" ]]; then
log "OK $URL — HTTP $status_code in ${response_time}s"
else
log "FAIL $URL — HTTP $status_code"
exit 1
fi
Stretch¶
- Write a script that queries the GitHub API to list all public repositories for a given username, formatted as a markdown table.
- Research
curl --retryandcurl --retry-delay. Add retry logic to your API check script.
Interview Questions¶
- What does
curl -sfdo?
Show answer
-s is silent mode (suppresses progress bars and error messages to stderr). -f makes curl exit with a non-zero status code when the HTTP response indicates an error (4xx or 5xx), instead of printing the error page body. Together they are useful in scripts: failure is detected by exit code, not by parsing output.
- How do you pass an API token to a REST API with curl?
Show answer
In the Authorization header: curl -H "Authorization: Bearer $TOKEN" URL. Some APIs use X-API-Key: $TOKEN. Never put tokens on the command line as query parameters (they appear in server logs and ps aux output). Store tokens in environment variables and inject them via -H.
- What is the difference between
curlandwget?
Show answer
curl is more versatile — it supports dozens of protocols, has richer output options, and is better for REST API interactions. wget is simpler and better for recursive downloads and resuming interrupted transfers. curl is generally preferred in scripts; wget is convenient for one-off downloads.