Nginx Troubleshooting

Common production errors and what they actually mean. Most issues fall into a small set of patterns once you know where to look.

Where to look first

sudo tail -f /var/log/nginx/error.log         # the most useful log
sudo tail -f /var/log/nginx/access.log        # request log; what's hitting you
sudo journalctl -u nginx -f                   # systemd-managed nginx
sudo nginx -t                                 # validate config
sudo nginx -T                                 # dump full effective config

Always check error.log before guessing. Nginx is verbose about why it returned a particular status.

502 Bad Gateway

Meaning: nginx couldn’t get a valid response from the upstream.

Top causes:

1. Upstream is down. curl http://upstream:port/ from the nginx box. If that fails, fix the upstream.
2. Connection refused. Upstream binding to 127.0.0.1 (only loopback), but nginx is on a different host. Bind to 0.0.0.0 or use the right IP.
3. Wrong port. Gunicorn on 8000, nginx points to 8080. ss -tlnp | grep <port> to see what’s actually listening.
4. SELinux blocking nginx → upstream. RHEL-flavored systems: setsebool -P httpd_can_network_connect 1.
5. Unix socket permissions wrong. ls -l /run/gunicorn.sock — nginx runs as www-data/nginx; the socket needs read+write for that user.
6. Upstream is up but rejecting. Misconfigured ALLOWED_HOSTS in Django returns 400 → nginx logs it as upstream error if buffering.

# Quick triage
curl -v http://upstream-host:port/healthz   # works?
ss -tlnp                                    # what's listening
ls -l /path/to/socket                       # socket perms
sudo tail -f /var/log/nginx/error.log       # nginx's view

504 Gateway Timeout

Meaning: upstream took longer to respond than proxy_read_timeout allows.

Causes:

• Upstream legitimately slow (DB query, external API call).
• Upstream hung.
• Network issues between nginx and upstream.

Fixes:

• Speed up the upstream (the right answer most of the time).
• Increase proxy_read_timeout if the slowness is legitimate (e.g. report generation).
• Move long operations to async (Celery / RQ / background task) and return early.

location /heavy-endpoint {
    proxy_read_timeout 300s;
    proxy_pass http://backend;
}

For SSE / long-polling endpoints, proxy_read_timeout 86400s; (or use HTTP/2 server push, WebSockets).

413 Request Entity Too Large

Meaning: client sent a body bigger than client_max_body_size.

http {
    client_max_body_size 50M;        # default is 1M
}

Set in http (global), server (per-vhost), or location (per-endpoint). For an upload endpoint specifically:

location /api/uploads {
    client_max_body_size 100M;
    proxy_pass http://backend;
}

499 Client Closed Request

Meaning: client gave up before nginx finished. Nginx-specific status, not standard HTTP.

Common when:

• Browser tab closed during long request.
• Mobile network hiccup.
• Reverse-proxy / CDN in front timed out and dropped the client connection.
• Frontend timeout shorter than backend.

Not necessarily a problem — clients leave all the time. Worry only when 499 spikes correlate with slow upstream.

500 Internal Server Error

Meaning: nginx itself errored, OR upstream returned 500 and nginx forwarded.

Check error.log:

• upstream sent invalid header — upstream returned malformed HTTP.
• unknown directive ... — config syntax error (only on reload).
• Stack trace from your Python app’s logs — most likely it’s the upstream.

400 Bad Request

Often means the request itself was malformed before it even reached the upstream:

• Invalid characters in headers.
• Content-Length mismatch.
• Bad TLS handshake (would actually be a TLS error in browser).

Less commonly: nginx-side. Mostly client-side or proxy-chain misconfigurations.

403 Forbidden

Meaning: nginx denied the request.

• deny 1.2.3.4; matched.
• auth_basic requires auth and credentials missing/wrong.
• internal; location accessed directly.
• Permission denied reading a static file.

404 Not Found

In nginx context:

• try_files couldn’t find the file.
• alias / root points at a missing path.
• Request URI doesn’t match any location in any server block.

For static files, the error log shows the actual path nginx tried:

2024/01/15 12:34:56 [error] open() "/var/www/static/missing.css" failed (2: No such file or directory)

Usually points right at the bug (wrong path, missing trailing slash on alias).

“address already in use”

Nginx couldn’t bind to a port because something else has it:

nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

Find the offender:

ss -tlnp | grep ':80 '
lsof -i :80

Likely Apache, another nginx, or a stuck old process.

“could not build server_names_hash”

Too many or too long server names:

http {
    server_names_hash_bucket_size 128;
    server_names_hash_max_size 2048;
}

Doubles when needed. Mostly hit on shared-hosting setups.

Worker connections exhaustion

[alert] 1234#0: 768 worker_connections are not enough

Bump it:

events {
    worker_connections 4096;        # or higher
}

Plus check OS file descriptor limit (ulimit -n) and worker_rlimit_nofile:

worker_rlimit_nofile 65535;

TLS errors

openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts

Shows full handshake, cipher chosen, cert chain, any errors.

High CPU on nginx

Usually:

• TLS handshakes (CPU-intensive). Solution: session caching, AES-NI CPUs (already universal), HTTP/2 to reuse connections.
• gzip on the fly with high gzip_comp_level. Drop to 5.
• Lots of regex location matches. Move regex blocks below prefix matches; add ^~ to short-circuit.
• Large file serving without sendfile. Always sendfile on;.

top -H -p $(pidof nginx) shows per-thread (worker) CPU.

High memory

Mostly from proxy_buffering of large responses. Tune:

proxy_buffer_size 4k;
proxy_buffers 8 4k;            # 8 buffers of 4k each
proxy_busy_buffers_size 8k;
proxy_max_temp_file_size 100m;     # spill to disk past this

Or disable buffering for large streams (proxy_buffering off).

Connection reset / hangups

• TCP keepalive misconfigured. keepalive_timeout 65; (longer than typical proxy idle).
• Upstream killed mid-response. Check upstream logs.
• Network device in between (firewall, NAT) timing out idle connections. Bump timeouts on app side, or use HTTP keep-alive headers.

“Why is my static file not loading?”

Most common: trailing slash mismatch in alias:

location /static/ {
    alias /var/www/static;          # no trailing slash, eats first char of path
}

Should be:

location /static/ {
    alias /var/www/static/;
}

Or use root instead — trailing slash doesn’t matter for root.

Logs filling disk

Two halves:

1. Rotation: /etc/logrotate.d/nginx — typically rotates daily, keeps 14 days. Check it’s running.

2. Reopen after rotation: nginx must be told to reopen log file handles after rotation:

nginx -s reopen     # SIGUSR1

logrotate handles this via postrotate directive in the rotation config. If logs vanish but disk stays full, check that logrotate’s postrotate is working.

Diagnostic checklist for “site is down”

1. sudo nginx -t — config valid?
2. sudo systemctl status nginx — process running?
3. ss -tlnp | grep -E ':80|:443' — listening on the right ports?
4. curl -v https://yourdomain.com/ from the server itself — does nginx respond?
5. curl -v http://upstream:port/ — is the backend up?
6. sudo tail -50 /var/log/nginx/error.log — what does nginx say?
7. Check DNS — dig yourdomain.com from outside.
8. Check firewall / SG — does port 443 ingress allow your client?
9. Check cert validity — echo | openssl s_client -connect host:443 -servername host 2>/dev/null | openssl x509 -noout -dates.

Top to bottom usually localizes the issue in <5 minutes.

Common interview confusions

• “502 means upstream is down.” — could be: down, refusing connections, on wrong port, blocked by firewall, returning malformed HTTP. Read the error log.
• “504 means nginx is slow.” — means upstream is slow (or hung). Nginx is fine; it gave up waiting.
• “413 is a security issue.” — it’s client_max_body_size too small. Default is 1MB.

Interview angle

• “You’re getting 502 Bad Gateway — how do you debug?” — check error.log first; verify upstream is up (curl from nginx box); check it’s listening on the expected port and IP; check Unix socket perms; check SELinux on RHEL.
• “504 vs 502?” — 502: upstream returned an error or refused connection. 504: upstream was reachable but didn’t respond within proxy_read_timeout. Different fixes.
• “Why might client_max_body_size 50M not take effect?” — set in the wrong context (http / server / location override each other; the most specific wins). Reload not done.
• “What’s a 499?” — client closed the connection before nginx finished. Nginx-specific status. Usually a client-side timeout shorter than your endpoint’s processing time, or a CDN/proxy in front giving up.
• “How do you debug a TLS handshake failure?” — openssl s_client -connect host:443 -servername host shows cipher negotiation, cert chain, and any errors. Check error.log for nginx-side details.
• “Logs filling disk — what’s the fix?” — logrotate (rotate daily, keep N days) plus a postrotate that signals nginx to reopen file handles (nginx -s reopen or kill -USR1).