Skip to content

S3-Compatible Object Storage Footguns

1. Treating object storage like a filesystem

You build an application that reads and writes thousands of small files (< 1 KB each) to MinIO, expecting filesystem-like performance. Each write is an HTTP PUT with ~50ms round-trip overhead. What would be a microsecond file write becomes a 50ms network call. The application is 500x slower than expected.

Fix: Object storage is designed for large objects (MBs to GBs), not small random files. For small files, batch them: aggregate logs, concatenate records, or use a local buffer that flushes to object storage in large chunks. Use a real filesystem or database for hot small-file workloads.

Under the hood: Every S3/MinIO PUT operation is a full HTTP request: TCP handshake, TLS negotiation (if enabled), HTTP headers, request signing, metadata write to the index, erasure-coded data write, and response. For a 100-byte object, the protocol overhead is 50-100x the payload size. AWS S3 pricing also works against small files — you pay per request regardless of object size, so 1 million 1KB objects costs the same in PUT requests as 1 million 1GB objects.

2. Deleting a bucket that has versioned objects

You run mc rb local/my-bucket and get an error saying the bucket isn't empty. You run mc rb --force local/my-bucket. With versioning enabled, --force only removes the current versions and delete markers — previous versions remain, the bucket can't actually be deleted, and you now have a partially destroyed bucket in an inconsistent state.

Fix: To delete a versioned bucket, first remove all versions and delete markers explicitly: 

# Remove all versions (current and noncurrent)
aws --endpoint-url=http://minio:9000 s3api list-object-versions \
  --bucket my-bucket --query 'Versions[*].[Key,VersionId]' --output text | \
  while read key vid; do
    aws --endpoint-url=http://minio:9000 s3api delete-object \
      --bucket my-bucket --key "$key" --version-id "$vid"
  done
# Same for DeleteMarkers
# Then: mc rb local/my-bucket

3. Using the root credentials in application code

You configure Loki, Thanos, Velero, and your backup scripts all using MINIO_ROOT_USER / MINIO_ROOT_PASSWORD. The root credential has full control: it can delete any bucket, change policies, create users. When you need to rotate it, you have to update every application simultaneously.

Fix: Create per-application service accounts with minimum required permissions: 

mc admin user add local lokirw lokirwpassword
mc admin policy attach local readwrite --user lokirw
mc admin user svcacct add local lokirw
# Use the returned access key pair in Loki config, not root creds

4. Not enabling versioning on backup buckets — then accidentally deleting backups

Your backup job writes to backups/daily/ without versioning. A misconfigured cleanup script runs aws s3 rm s3://backups/ --recursive. All backups are permanently gone. There is no recycle bin.

Fix: Enable versioning on all backup and compliance buckets immediately: 

mc version enable local/backups
Versioning means deleted objects get a delete marker, not permanent deletion. You can recover by removing the delete marker. Also apply Object Lock for true WORM protection on compliance data.

5. mc mirror --remove without a dry-run first

You run mc mirror --remove /local/data/ local/backup/ to sync a directory and delete objects not in the source. You forgot that /local/data/ is mounted read-only and shows only 3 files. MinIO deletes everything except those 3 files from the bucket. Months of data gone.

Fix: Always dry-run destructive sync operations first: 

mc mirror --dry-run --remove /local/data/ local/backup/
# Review output carefully
# Only proceed if the delete list looks right
mc mirror --remove /local/data/ local/backup/

6. Not setting s3forcepathstyle for MinIO (path-style vs virtual-hosted-style URLs)

You configure Loki to use MinIO but it can't connect. The default AWS SDK behavior uses virtual-hosted-style URLs: http://bucket.minio:9000/key. MinIO's default setup doesn't have wildcard DNS for *.minio:9000, so those URLs don't resolve.

Fix: Set s3forcepathstyle: true (or equivalent) in every tool that uses MinIO: 

# Loki
storage_config:
  aws:
    s3forcepathstyle: true

# Thanos (objstore.yml)
config:
  http_config:
    insecure_skip_verify: false
  # no forcepathstyle needed — Thanos uses path style by default

# boto3
s3 = boto3.client('s3',
    endpoint_url='http://minio:9000',
    config=Config(s3={'addressing_style': 'path'})
)

7. Storing secrets or tokens in object metadata or object keys

You store API tokens in object key names like tokens/sk-abc123def456.key for "easy lookup". The token value is now visible in any ListObjects response, in access logs, and to anyone with bucket read access.

Fix: Never put secret values in key names or user-metadata. Use opaque identifiers as keys. Store sensitive config in a secrets manager (Vault, Kubernetes Secrets) and reference object storage keys from there. Apply bucket policies to restrict access to sensitive prefix paths.

8. Not configuring TLS on internet-facing MinIO

You deploy MinIO for a small team and expose it on port 9000 without TLS, planning to "add it later". Traffic (including access keys in Authorization headers) goes over the wire in plaintext.

Fix: Configure TLS from the start: 

# MinIO with TLS — provide cert and key
# Place certs at $HOME/.minio/certs/public.crt and private.key
# Or configure in environment:
MINIO_SERVER_URL=https://minio.example.com
# Use Let's Encrypt via Caddy or nginx as TLS terminator in front of MinIO
For Kubernetes: use cert-manager and an Ingress with TLS, or configure the MinIO Operator's TLS support.

9. Miscounting the EC stripe causing a MinIO cluster that can't tolerate any failures

You deploy MinIO with 3 nodes × 1 drive each = 3 drives total. MinIO uses EC:1 (1 parity drive). The minimum set size is 2 (1 data + 1 parity). If 2 drives fail simultaneously, data is lost. You assumed EC would give you N-2 resilience.

Fix: Understand MinIO's erasure coding. For a 3-drive cluster, you only get EC:1 (single drive failure tolerance). For meaningful redundancy: - 4 drives minimum: EC:2 (2 drive failures tolerated) - 8 drives (4+4): EC:4 - MinIO documentation states: "minimum 4 drives for EC:2"

Deploy at least 4 drives from the start. For production: 4+ nodes × 4+ drives = 16+ drives is the recommended minimum.

10. Running mc admin heal on a loaded cluster without throttling

You discover some objects are inconsistent and run mc admin heal local --recursive on a petabyte-scale bucket. Heal reads and rewrites every object to verify and repair erasure codes. On a loaded cluster, this saturates disk I/O and network for hours, degrading performance for all users.

Fix: Run heals during off-peak hours. Monitor heal progress and abort if needed: 

# Check heal status
mc admin heal local --status

# For a specific prefix only
mc admin heal local/my-bucket/logs/ --recursive

# Abort a running heal
mc admin heal local --stop

11. Not setting lifecycle rules on log/temp buckets — disk fills silently

You create a bucket for application logs and Loki chunk uploads. No lifecycle rule. Over 6 months, the bucket grows to 10 TB. MinIO runs out of space. All writes fail (both to object storage and to the primary application). Loki crashes. Metrics stop flowing. You discover the issue during the outage.

Fix: Every bucket with continuous write workload needs a lifecycle rule. Set expiration at creation time: 

mc ilm add --expiry-days 90 local/logs
mc ilm add --expiry-days 365 local/thanos
mc ilm add --expiry-days 30 local/loki-chunks
Alert on bucket size via Prometheus: 
# MinIO Prometheus metric
minio_bucket_usage_total_bytes{bucket="loki-chunks"} > 500e9

12. Using CopyObject to "rename" objects — not understanding it's a full copy

You implement a "move" operation in your application by calling CopyObject (source → destination) then DeleteObject (source). You assume this is fast because "it's all on the same server." For a 10 GB object, CopyObject reads all 10 GB, re-encodes it with erasure coding, and writes all 10 GB. For a bucket with thousands of large objects, your "rename directory" operation takes hours and saturates disk throughput.

Fix: Architect to avoid renames. Use key naming schemes that don't require post-write renaming (e.g., write to final key directly, not a temp key). If you must rename large files, do it off-peak and track progress. Consider whether your use case actually needs rename semantics — if so, object storage may be the wrong tool.