Skip to content

Portal | Level: L1: Foundations | Topics: curl & wget, Linux Fundamentals | Domain: CLI Tools

curl & wget — HTTP Tools Primer

Why This Matters

curl and wget are how you interact with HTTP from the command line. Health checks, API calls, downloading artifacts, testing load balancers, debugging SSL, sending webhooks — all of it flows through these tools. curl is available on every modern Linux, macOS, and Windows system. If you cannot drive curl fluently, you are opening a browser or writing a script for something that should take five seconds.

Name origin: "curl" stands for "Client URL." Daniel Stenberg started it in 1998 as "httpget," renamed it to "urlget," and finally "curl." The project has been in continuous development for over 25 years and supports over 25 protocols. Stenberg estimates curl is installed on over 20 billion devices worldwide.

curl — The HTTP Swiss Army Knife

Basic Requests

# GET request (default)
curl https://api.example.com/status

# Verbose output (see headers, TLS handshake, everything)
curl -v https://api.example.com/status

# Silent mode (suppress progress bar)
curl -s https://api.example.com/status

# Silent with errors shown
curl -sS https://api.example.com/status

# Follow redirects
curl -L https://example.com/download

# Show only response headers
curl -I https://example.com

# Show response headers AND body
curl -i https://api.example.com/data

HTTP Methods

# POST
curl -X POST https://api.example.com/users

# PUT
curl -X PUT https://api.example.com/users/123

# PATCH
curl -X PATCH https://api.example.com/users/123

# DELETE
curl -X DELETE https://api.example.com/users/123

# HEAD (same as -I)
curl -X HEAD https://api.example.com/status

Note: -X POST is not needed when you use -d (data). curl infers POST from the presence of a body.

Gotcha: Using -X POST with -d is harmless but redundant. However, using -X GET with -d will send a GET request with a body, which confuses many servers and proxies. If you see a GET with -X and -d in a script, it is almost always a bug.

Sending JSON

# POST JSON payload
curl -s -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name": "alice", "role": "admin"}'

# JSON from a file
curl -s -X POST https://api.example.com/config \
  -H "Content-Type: application/json" \
  -d @payload.json

# PUT JSON update
curl -s -X PUT https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"role": "viewer"}'

# Pipe JSON from stdin
echo '{"status": "active"}' | curl -s -X POST https://api.example.com/status \
  -H "Content-Type: application/json" \
  -d @-

Authentication

# Basic auth
curl -u username:password https://api.example.com/secure

# Bearer token
curl -H "Authorization: Bearer eyJhbGci..." https://api.example.com/data

# OAuth2 bearer (shorthand, requires curl 7.73+)
curl --oauth2-bearer eyJhbGci... https://api.example.com/data

# API key in header
curl -H "X-API-Key: abc123def456" https://api.example.com/data

# Netrc file (avoid credentials in command history)
curl --netrc-file ~/.netrc https://api.example.com/data
# ~/.netrc format:
# machine api.example.com login user password secret

Downloading Files

# Save with remote filename
curl -O https://releases.example.com/app-v2.3.tar.gz

# Save with custom filename
curl -o app.tar.gz https://releases.example.com/app-v2.3.tar.gz

# Resume interrupted download
curl -C - -O https://releases.example.com/large-file.iso

# Download multiple files
curl -O https://example.com/file1.txt -O https://example.com/file2.txt

# Limit download speed
curl --limit-rate 1M -O https://example.com/large.tar.gz

# Download only if newer than local file
curl -z local-file.txt -O https://example.com/file.txt

# Save cookies to a file
curl -c cookies.txt https://example.com/login \
  -d "user=alice&pass=secret"

# Send cookies from a file
curl -b cookies.txt https://example.com/dashboard

# Send a specific cookie
curl -b "session=abc123" https://example.com/api

# Save and send cookies in one flow
curl -c cookies.txt -b cookies.txt https://example.com/auth

Timing and Performance

# Measure response time breakdown
curl -o /dev/null -s -w "\
    DNS:        %{time_namelookup}s\n\
    Connect:    %{time_connect}s\n\
    TLS:        %{time_appconnect}s\n\
    TTFB:       %{time_starttransfer}s\n\
    Total:      %{time_total}s\n\
    Size:       %{size_download} bytes\n\
    HTTP Code:  %{http_code}\n" \
  https://api.example.com/health

# Output:
#     DNS:        0.012s
#     Connect:    0.045s
#     TLS:        0.120s
#     TTFB:       0.250s
#     Total:      0.255s
#     Size:       42 bytes
#     HTTP Code:  200

# Set connection timeout
curl --connect-timeout 5 https://api.example.com/health

# Set maximum time for entire operation
curl --max-time 30 https://api.example.com/slow-endpoint

# Both together (standard for health checks)
curl --connect-timeout 5 --max-time 10 -sf https://api.example.com/health

SSL/TLS Options

# Skip certificate verification (DANGEROUS — dev/debug only)
curl -k https://self-signed.example.com

# Remember: -k is for "insecure" (think "k for kaution")

# Specify CA certificate
curl --cacert /etc/ssl/certs/internal-ca.pem https://internal.example.com

# Client certificate authentication (mTLS)
curl --cert /etc/ssl/client.pem --key /etc/ssl/client-key.pem \
  https://mtls.example.com

# Show certificate details
curl -vvI https://example.com 2>&1 | grep -A 20 "Server certificate"

# Force specific TLS version
curl --tlsv1.2 https://example.com
curl --tlsv1.3 https://example.com

Multipart Form Data

# Upload a file
curl -F "file=@report.pdf" https://api.example.com/upload

# Upload with additional fields
curl -F "file=@report.pdf" -F "description=Monthly report" \
  https://api.example.com/upload

# Multiple files
curl -F "file1=@doc1.pdf" -F "file2=@doc2.pdf" \
  https://api.example.com/upload

# Specify content type for upload
curl -F "file=@data.csv;type=text/csv" https://api.example.com/import

Working with APIs

# REST API — create, read, update, delete
# Create
curl -s -X POST https://api.example.com/items \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"name": "widget", "count": 5}'

# Read
curl -s -H "Authorization: Bearer $TOKEN" \
  https://api.example.com/items/42 | jq .

# Update
curl -s -X PUT https://api.example.com/items/42 \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"count": 10}'

# Delete
curl -s -X DELETE -H "Authorization: Bearer $TOKEN" \
  https://api.example.com/items/42

# GraphQL
curl -s -X POST https://api.example.com/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ user(id: 1) { name email } }"}'

# Pagination (follow Link headers)
curl -sI https://api.example.com/items?page=1 | grep -i "^link:"

Retry and Resilience

# Retry on transient failures
curl --retry 3 --retry-delay 2 https://api.example.com/health

# Retry on connection refused and HTTP 5xx
curl --retry 5 --retry-all-errors https://api.example.com/health

# Retry with exponential backoff (max wait)
curl --retry 3 --retry-max-time 60 https://api.example.com/health

wget — Bulk Downloader and Site Mirrorer

wget is optimized for downloading files and mirroring sites. It handles recursive downloads, retries, and bandwidth limiting out of the box.

Basic Downloads

# Download a file
wget https://releases.example.com/app-v2.3.tar.gz

# Save with custom filename
wget -O app.tar.gz https://releases.example.com/app-v2.3.tar.gz

# Download to a specific directory
wget -P /tmp/downloads/ https://example.com/file.tar.gz

# Resume interrupted download
wget -c https://releases.example.com/large-file.iso

# Quiet mode (no output except errors)
wget -q https://example.com/file.tar.gz

# Background download
wget -b https://releases.example.com/huge.tar.gz
# Check progress: tail -f wget-log

Recursive Downloads and Mirroring

# Mirror a website for offline viewing
wget --mirror --convert-links --adjust-extension \
  --page-requisites --no-parent \
  https://docs.example.com/

# Same as above (shorthand)
wget -mkEpnp https://docs.example.com/

# Download specific file types only
wget -r -A "*.pdf" https://example.com/reports/

# Reject specific file types
wget -r -R "*.gif,*.jpg,*.png" https://example.com/docs/

# Limit recursion depth
wget -r -l 2 https://example.com/

# Download a list of URLs from a file
wget -i urls.txt

# Restrict to the same domain (do not follow external links)
wget -r --domains=example.com https://example.com/

wget vs curl

Feature curl wget
HTTP methods All (GET, POST, PUT, DELETE, PATCH) GET, POST only
JSON/API work Excellent Poor
Recursive download No Yes
Site mirroring No Yes
Resume interrupted Yes (-C -) Yes (-c)
Protocol support HTTP, FTP, SMTP, IMAP, SCP, many more HTTP, FTP
Output control -o, -O, stdout -O, -P
Cookie handling -b, -c --load-cookies, --save-cookies
Auth methods Basic, Bearer, Digest, NTLM, Kerberos Basic, Digest
Progress bar Optional (--progress-bar) Default
Available by default macOS, most Linux Most Linux

Rule of thumb: curl for APIs, headers, and HTTP method control. wget for downloading files and mirroring sites.

Remember: Mnemonic for when to reach for which tool: "curl Calls, wget Gets." curl is for calling APIs (interactive, bidirectional). wget is for getting files (download-and-save).

Interview tip: When asked "how would you test an API endpoint from a server with no browser," the answer is always curl. Bonus points for showing -w format strings to extract timing and status codes in a health-check script.

httpie — The Modern Alternative

httpie (http command) offers a more human-friendly syntax for API work:

# GET request
http https://api.example.com/status

# POST JSON (auto-sets Content-Type)
http POST https://api.example.com/users name=alice role=admin

# Bearer auth
http https://api.example.com/data "Authorization:Bearer token123"

# Form data
http --form POST https://example.com/login user=alice pass=secret

# Download
http --download https://example.com/file.tar.gz

httpie is not installed by default, so you cannot rely on it in scripts or on production servers. Learn curl first — it is universal. Use httpie for interactive API exploration on your workstation.

Fun fact: wget was written by Hrvoje Niksic in 1996 as part of the GNU project. The name is a portmanteau of "World Wide Web" and "get." Unlike curl, wget was designed from the start for non-interactive use -- downloading files unattended, which is why it writes to disk by default while curl writes to stdout.

Wiki Navigation

Prerequisites

Related Content