What is Prometheus?

Definition

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It collects metrics from configured targets at given intervals, stores them in a time-series database, and provides a query language (PromQL) to retrieve and analyze the data.

Key Features

Core Components

1. Time-Series Database: Stores metrics with timestamps
2. Pull Model: Scrapes metrics from targets
3. PromQL: Powerful query language
4. Service Discovery: Automatically discovers targets
5. Alertmanager: Handles alerts
6. Multi-Dimensional: Labels for flexible querying

Architecture

Targets → Prometheus Server → PromQL Queries → Grafana/Alertmanager
(Apps,     (Scrapes, Stores)    (Query Language)  (Visualization,
 Services)                                        Alerts)

How It Works

Pull Model

Prometheus → HTTP Request → Target → Metrics → Prometheus Storage
(Scraper)    (GET /metrics)  (Exports)  (Text)   (Time-Series DB)

Push Model (Pushgateway)

Application → Pushgateway → Prometheus
(Sends metrics)  (Temporary)  (Scrapes)

Basic Setup

Docker Compose Example

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

volumes:
  prometheus-data:

Configuration (prometheus.yml)

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'my-app'
    static_configs:
      - targets: ['app:8000']

Metrics

Metric Types

1. Counter: Monotonically increasing value
2. Gauge: Value that can go up or down
3. Histogram: Distribution of values
4. Summary: Similar to histogram with quantiles

Example Metrics

# Python example with prometheus_client
from prometheus_client import Counter, Gauge, Histogram, start_http_server

# Counter
requests_total = Counter('http_requests_total', 'Total HTTP requests', ['method', 'endpoint'])

# Gauge
cpu_usage = Gauge('cpu_usage_percent', 'CPU usage percentage')

# Histogram
request_duration = Histogram('http_request_duration_seconds', 'Request duration')

# Increment counter
requests_total.labels(method='GET', endpoint='/api').inc()

# Set gauge
cpu_usage.set(75.5)

# Record duration
with request_duration.time():
    process_request()

Exposing Metrics

# Flask example
from flask import Flask
from prometheus_client import make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware

app = Flask(__name__)

# Add metrics endpoint
app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
    '/metrics': make_wsgi_app()
})

PromQL (Prometheus Query Language)

Basic Queries

# Select metric
http_requests_total

# Filter by label
http_requests_total{method="GET"}

# Rate (per second)
rate(http_requests_total[5m])

# Increase over time
increase(http_requests_total[1h])

# Average
avg(cpu_usage)

# Sum
sum(http_requests_total)

# Percentage
100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Common Queries

# Request rate
rate(http_requests_total[5m])

# Error rate
rate(http_requests_total{status="500"}[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# CPU usage
100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Alerting

Alert Rules (alerts.yml)

groups:
  - name: example
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage is above 80% for 5 minutes"

      - alert: HighErrorRate
        expr: rate(http_requests_total{status="500"}[5m]) > 0.1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate"
          description: "Error rate is above 10%"

Common Interview Questions

Q1: What is Prometheus and how does it work?

Prometheus is a monitoring system that:

• Scrapes Metrics: Pulls metrics from targets via HTTP
• Stores Data: Time-series database for metrics
• Queries: PromQL for data retrieval
• Alerts: Alertmanager for notifications

How it works:

1. Targets expose metrics at /metrics endpoint
2. Prometheus scrapes metrics at intervals
3. Stores in time-series database
4. Queries via PromQL
5. Alerts based on rules

Q2: What’s the difference between Prometheus and Grafana?

They Work Together: Prometheus collects/stores, Grafana visualizes.

Q3: What is the pull model in Prometheus?

Pull model means Prometheus actively scrapes metrics from targets:

Advantages:

• Centralized control
• No need to configure each target
• Automatic discovery
• Better for batch jobs

How it works:

Prometheus → HTTP GET /metrics → Target → Returns metrics

Push Model Alternative:

• Targets push metrics to Pushgateway
• Used for short-lived jobs
• Less common

Q4: What are the metric types in Prometheus?

Four metric types:

1. Counter: Monotonically increasing (e.g., total requests)

   http_requests_total

2. Gauge: Can increase or decrease (e.g., CPU usage)

   cpu_usage_percent

3. Histogram: Distribution of values (e.g., request duration)

   http_request_duration_seconds_bucket

4. Summary: Similar to histogram with quantiles

   http_request_duration_seconds{quantile="0.95"}

Q5: How do you expose metrics from a Python application?

Using prometheus_client:

from prometheus_client import Counter, Gauge, start_http_server

# Define metrics
requests_total = Counter('http_requests_total', 'Total requests')
cpu_usage = Gauge('cpu_usage', 'CPU usage')

# Start metrics server
start_http_server(8000)

# Metrics available at http://localhost:8000/metrics

Flask Integration:

from prometheus_client import make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware

app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
    '/metrics': make_wsgi_app()
})

Q6: What is PromQL?

PromQL (Prometheus Query Language) is used to:

• Select Metrics: Filter and aggregate
• Calculate Rates: Per-second rates
• Aggregate: Sum, avg, min, max
• Time Functions: Over time ranges

Examples:

# Rate
rate(http_requests_total[5m])

# Sum
sum(http_requests_total)

# Average
avg(cpu_usage)

# Percentage
100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Best Practices

1. Use Labels Wisely: Don’t create high cardinality
2. Set Scrape Intervals: Balance freshness vs load
3. Organize Jobs: Group related targets
4. Use Service Discovery: Automate target discovery
5. Set Retention: Configure data retention policy
6. Monitor Prometheus: Monitor Prometheus itself
7. Use Recording Rules: Pre-compute expensive queries
8. Set Up Alerts: Configure alerting rules

Summary

Prometheus is:

• Monitoring System: Collects and stores metrics
• Time-Series Database: Stores metrics with timestamps
• Pull-Based: Scrapes metrics from targets
• Query Language: PromQL for flexible queries
• Alerting: Alertmanager for notifications

Key features:

• Pull model for metrics collection
• Multi-dimensional data model (labels)
• Powerful query language (PromQL)
• Service discovery
• Reliable storage

Use Prometheus to:

• Monitor infrastructure
• Track application metrics
• Set up alerting
• Analyze performance
• Store time-series data

Interview angle

• “How does Prometheus collect data?” - it scrapes HTTP endpoints on an interval; targets expose metrics rather than pushing them. Short-lived jobs that can’t be scraped use a Pushgateway, which is the documented exception rather than the norm.
• “Which metric type for what?” - counter for monotonically increasing totals (requests, errors), gauge for values that go up and down (queue depth, memory), histogram for latency distributions so you can compute percentiles. Using a gauge for a counter loses information on restart.
• “How do you compute a request rate?” - rate() over a counter, which handles resets automatically. Alerting on a raw counter value is meaningless; you almost always want a rate or an increase over a window.
• “What’s the failure mode to avoid?” - cardinality. Labels with unbounded values - user id, request id, raw path - create a time series per value and will take the server down. See ../../system_design/05_observability/01_observability_in_practice.md.