backend / protocols / nginx / 01_nginx_vm_communication.md

How to Configure Communication Between Multiple VMs via Nginx

3 interview angles 9 min read source

How to Configure Communication Between Multiple VMs via Nginx

Overview

Nginx can be configured as a reverse proxy and load balancer to enable communication between multiple Virtual Machines (VMs). This setup allows you to distribute traffic, provide high availability, and manage communication between backend services running on different VMs.

Architecture

Client → Nginx (VM1) → Backend VMs (VM2, VM3, VM4)
           ↓
    Load Balancer
    Reverse Proxy
    SSL Termination

Prerequisites

  • Multiple VMs with network connectivity
  • Nginx installed on the proxy/load balancer VM
  • Backend services running on target VMs
  • Proper firewall rules configured
  • SSH access to all VMs

Basic Setup

1. Install Nginx

# On Ubuntu/Debian
sudo apt update
sudo apt install nginx

# On CentOS/RHEL
sudo yum install nginx
# or
sudo dnf install nginx

# Start and enable Nginx
sudo systemctl start nginx
sudo systemctl enable nginx

2. Basic Reverse Proxy Configuration

Create a basic reverse proxy configuration:

# /etc/nginx/sites-available/backend-proxy
upstream backend_servers {
    server 192.168.1.10:8000;  # VM2
    server 192.168.1.11:8000;  # VM3
    server 192.168.1.12:8000;  # VM4
}

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable the configuration:

sudo ln -s /etc/nginx/sites-available/backend-proxy /etc/nginx/sites-enabled/
sudo nginx -t  # Test configuration
sudo systemctl reload nginx

Load Balancing Methods

1. Round Robin (Default)

upstream backend_servers {
    server 192.168.1.10:8000;
    server 192.168.1.11:8000;
    server 192.168.1.12:8000;
}

2. Least Connections

upstream backend_servers {
    least_conn;
    server 192.168.1.10:8000;
    server 192.168.1.11:8000;
    server 192.168.1.12:8000;
}

3. IP Hash (Session Persistence)

upstream backend_servers {
    ip_hash;
    server 192.168.1.10:8000;
    server 192.168.1.11:8000;
    server 192.168.1.12:8000;
}

4. Weighted Load Balancing

upstream backend_servers {
    server 192.168.1.10:8000 weight=3;  # 3x more traffic
    server 192.168.1.11:8000 weight=2;  # 2x traffic
    server 192.168.1.12:8000 weight=1;   # 1x traffic
}

5. Backup Server

upstream backend_servers {
    server 192.168.1.10:8000;
    server 192.168.1.11:8000;
    server 192.168.1.12:8000 backup;  # Only used if others are down
}

Advanced Configuration

Complete Production Configuration

# /etc/nginx/nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 1024;
    use epoll;
    multi_accept on;
}

http {
    # Basic settings
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    client_max_body_size 20M;

    # Logging
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    # Gzip compression
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    # Upstream backend servers
    upstream backend_servers {
        least_conn;
        server 192.168.1.10:8000 max_fails=3 fail_timeout=30s;
        server 192.168.1.11:8000 max_fails=3 fail_timeout=30s;
        server 192.168.1.12:8000 max_fails=3 fail_timeout=30s;
        
        # Health check (requires nginx-plus or third-party module)
        # keepalive 32;
    }

    # HTTP to HTTPS redirect
    server {
        listen 80;
        server_name example.com www.example.com;
        return 301 https://$server_name$request_uri;
    }

    # HTTPS server
    server {
        listen 443 ssl http2;
        server_name example.com www.example.com;

        # SSL configuration
        ssl_certificate /etc/ssl/certs/example.com.crt;
        ssl_certificate_key /etc/ssl/private/example.com.key;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        # Security headers
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
        add_header X-Frame-Options "SAMEORIGIN" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header X-XSS-Protection "1; mode=block" always;

        # Proxy settings
        location / {
            proxy_pass http://backend_servers;
            
            # Headers
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Port $server_port;

            # Timeouts
            proxy_connect_timeout 60s;
            proxy_send_timeout 60s;
            proxy_read_timeout 60s;

            # Buffering
            proxy_buffering on;
            proxy_buffer_size 4k;
            proxy_buffers 8 4k;
            proxy_busy_buffers_size 8k;

            # Error handling
            proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
            proxy_next_upstream_tries 3;
            proxy_next_upstream_timeout 10s;
        }

        # Health check endpoint
        location /health {
            access_log off;
            return 200 "healthy\n";
            add_header Content-Type text/plain;
        }

        # Static files (if serving directly)
        location /static/ {
            alias /var/www/static/;
            expires 30d;
            add_header Cache-Control "public, immutable";
        }
    }
}

Health Checks and Failover

Basic Health Check Configuration

upstream backend_servers {
    server 192.168.1.10:8000 max_fails=3 fail_timeout=30s;
    server 192.168.1.11:8000 max_fails=3 fail_timeout=30s;
    server 192.168.1.12:8000 max_fails=3 fail_timeout=30s;
}

server {
    location / {
        proxy_pass http://backend_servers;
        proxy_next_upstream error timeout http_500 http_502 http_503 http_504;
        proxy_next_upstream_tries 3;
    }
}

Custom Health Check Script

#!/bin/bash
# /usr/local/bin/nginx-health-check.sh

BACKEND_SERVERS=("192.168.1.10:8000" "192.168.1.11:8000" "192.168.1.12:8000")
NGINX_CONFIG="/etc/nginx/sites-available/backend-proxy"

for server in "${BACKEND_SERVERS[@]}"; do
    IFS=':' read -r ip port <<< "$server"
    if ! timeout 2 bash -c "echo > /dev/tcp/$ip/$port" 2>/dev/null; then
        echo "Server $server is down"
        # Optionally mark as down in nginx config
    fi
done

Add to crontab:

# Check every minute
* * * * * /usr/local/bin/nginx-health-check.sh

SSL/TLS Configuration

Self-Signed Certificate (Development)

# Generate self-signed certificate
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
    -keyout /etc/ssl/private/nginx-selfsigned.key \
    -out /etc/ssl/certs/nginx-selfsigned.crt

Let’s Encrypt Certificate (Production)

# Install certbot
sudo apt install certbot python3-certbot-nginx

# Obtain certificate
sudo certbot --nginx -d example.com -d www.example.com

# Auto-renewal (already configured)
sudo certbot renew --dry-run

Multiple Services Configuration

Routing Based on Path

upstream api_servers {
    server 192.168.1.10:8000;
    server 192.168.1.11:8000;
}

upstream web_servers {
    server 192.168.1.20:8080;
    server 192.168.1.21:8080;
}

upstream db_servers {
    server 192.168.1.30:5432;
}

server {
    listen 80;
    server_name example.com;

    # API requests
    location /api/ {
        proxy_pass http://api_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Web requests
    location / {
        proxy_pass http://web_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Database proxy (if needed)
    location /db/ {
        proxy_pass http://db_servers;
        # Additional security headers
    }
}

Routing Based on Domain

# API subdomain
server {
    listen 80;
    server_name api.example.com;

    location / {
        proxy_pass http://api_servers;
        proxy_set_header Host $host;
    }
}

# Web subdomain
server {
    listen 80;
    server_name www.example.com;

    location / {
        proxy_pass http://web_servers;
        proxy_set_header Host $host;
    }
}

Security Configuration

Rate Limiting

# Define rate limit zones
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=general_limit:10m rate=50r/s;

server {
    location /api/ {
        limit_req zone=api_limit burst=20 nodelay;
        proxy_pass http://api_servers;
    }

    location / {
        limit_req zone=general_limit burst=100;
        proxy_pass http://web_servers;
    }
}

IP Whitelisting

# Allow only specific IPs
location /admin/ {
    allow 192.168.1.0/24;  # Local network
    allow 10.0.0.0/8;      # Private network
    deny all;

    proxy_pass http://backend_servers;
}

Basic Authentication

location /secure/ {
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/.htpasswd;

    proxy_pass http://backend_servers;
}

Create password file:

sudo apt install apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd username

Monitoring and Logging

Custom Log Format

log_format detailed '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time uct="$upstream_connect_time" '
                    'uht="$upstream_header_time" urt="$upstream_response_time"';

server {
    access_log /var/log/nginx/access.log detailed;
    error_log /var/log/nginx/error.log warn;
}

Status Page (Requires stub_status module)

location /nginx_status {
    stub_status on;
    access_log off;
    allow 127.0.0.1;
    allow 192.168.1.0/24;
    deny all;
}

Firewall Configuration

UFW (Ubuntu)

# Allow HTTP and HTTPS
sudo ufw allow 'Nginx Full'
# or
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Allow SSH
sudo ufw allow 22/tcp

# Enable firewall
sudo ufw enable

firewalld (CentOS/RHEL)

# Allow HTTP and HTTPS
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

iptables

# Allow HTTP
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# Allow HTTPS
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Save rules
sudo iptables-save > /etc/iptables/rules.v4

Testing Configuration

Test Nginx Configuration

# Check syntax
sudo nginx -t

# Check configuration with verbose output
sudo nginx -T

# Test and reload
sudo nginx -t && sudo systemctl reload nginx

Test Connectivity

# Test from Nginx VM to backend VMs
curl -I http://192.168.1.10:8000
curl -I http://192.168.1.11:8000
curl -I http://192.168.1.12:8000

# Test through Nginx
curl -I http://nginx-vm-ip/
curl -I https://example.com/

Load Testing

# Install Apache Bench
sudo apt install apache2-utils

# Run load test
ab -n 1000 -c 10 http://example.com/

# Install wrk for more advanced testing
sudo apt install wrk
wrk -t4 -c100 -d30s http://example.com/

Troubleshooting

Common Issues

  1. 502 Bad Gateway

    # Check backend servers are running
curl http://192.168.1.10:8000

# Check firewall rules
sudo ufw status

# Check Nginx error log
sudo tail -f /var/log/nginx/error.log

  2. Connection Timeout

    # Increase timeout values
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;

  3. SSL Certificate Errors

    # Verify certificate
openssl x509 -in /etc/ssl/certs/example.com.crt -text -noout

# Check certificate expiration
openssl x509 -in /etc/ssl/certs/example.com.crt -noout -dates

  4. High Memory Usage

    # Reduce worker connections
worker_connections 512;

# Reduce buffer sizes
proxy_buffer_size 2k;
proxy_buffers 4 2k;

Best Practices

  1. Use HTTPS: Always use SSL/TLS in production
  2. Enable Gzip: Compress responses to reduce bandwidth
  3. Set Proper Timeouts: Configure appropriate timeout values
  4. Monitor Logs: Regularly check access and error logs
  5. Use Health Checks: Implement proper health checking
  6. Limit Connections: Use rate limiting to prevent abuse
  7. Keep Nginx Updated: Regularly update Nginx for security patches
  8. Use Connection Pooling: Enable keepalive connections to upstream
  9. Separate Configurations: Use separate config files for different services
  10. Test Before Deploying: Always test configuration changes

Example: Complete Multi-VM Setup

Architecture

Internet
   ↓
Nginx VM (192.168.1.5) - Load Balancer
   ↓
   ├──→ App VM 1 (192.168.1.10:8000)
   ├──→ App VM 2 (192.168.1.11:8000)
   └──→ App VM 3 (192.168.1.12:8000)

Nginx Configuration

# /etc/nginx/sites-available/multi-vm-setup
upstream app_cluster {
    least_conn;
    server 192.168.1.10:8000 weight=3 max_fails=3 fail_timeout=30s;
    server 192.168.1.11:8000 weight=2 max_fails=3 fail_timeout=30s;
    server 192.168.1.12:8000 weight=1 max_fails=3 fail_timeout=30s;
    
    keepalive 32;
}

server {
    listen 80;
    server_name app.example.com;
    
    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name app.example.com;

    # SSL Configuration
    ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
    ssl_prefer_server_ciphers off;

    # Security Headers
    add_header Strict-Transport-Security "max-age=63072000" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;

    # Rate Limiting
    limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
    limit_req zone=api_limit burst=20 nodelay;

    location / {
        proxy_pass http://app_cluster;
        
        # Headers
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
        
        # Error Handling
        proxy_next_upstream error timeout http_500 http_502 http_503 http_504;
        proxy_next_upstream_tries 3;
    }

    # Health Check
    location /health {
        access_log off;
        return 200 "OK\n";
        add_header Content-Type text/plain;
    }
}

Setup Script

#!/bin/bash
# setup-nginx-proxy.sh

# Install Nginx
sudo apt update
sudo apt install -y nginx

# Create configuration
sudo cp nginx-config /etc/nginx/sites-available/multi-vm-setup
sudo ln -s /etc/nginx/sites-available/multi-vm-setup /etc/nginx/sites-enabled/

# Remove default site
sudo rm /etc/nginx/sites-enabled/default

# Test and reload
sudo nginx -t && sudo systemctl reload nginx

# Configure firewall
sudo ufw allow 'Nginx Full'
sudo ufw enable

echo "Nginx proxy configured successfully!"

Summary

Configuring communication between multiple VMs via Nginx involves:

  1. Installation: Install Nginx on the proxy VM
  2. Configuration: Set up upstream servers and proxy settings
  3. Load Balancing: Choose appropriate load balancing method
  4. SSL/TLS: Configure SSL certificates for HTTPS
  5. Security: Implement rate limiting, IP whitelisting, and security headers
  6. Monitoring: Set up logging and health checks
  7. Testing: Verify connectivity and performance
  8. Maintenance: Regular updates and monitoring

This setup provides a robust, scalable solution for managing communication between multiple VMs with high availability and load distribution.

Interview angle

  • “Why put nginx in front of an application server?” - TLS termination, static file serving, request buffering to protect slow-client exposure, connection limiting, compression and load balancing. The application server does none of these well.
  • “What is proxy buffering and when does it hurt?” - nginx buffers the upstream response before forwarding. That protects the backend from slow clients but breaks streaming: SSE and LLM token streams arrive all at once until you disable it for those routes.
  • “What headers must you set when proxying?” - X-Forwarded-For, X-Forwarded-Proto and Host, or the application sees nginx’s address and the wrong scheme, which breaks redirects, rate limiting and logging.