Amazon CloudWatch

7 min read index source

Amazon CloudWatch

Overview

Amazon CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers. It provides data and actionable insights to monitor your applications, respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health.

Key Features

Metrics: Collect and track metrics from AWS resources and applications
Logs: Centralized log management and analysis
Alarms: Set up automated monitoring and alerting
Dashboards: Create custom dashboards for visualization
Events: Respond to state changes in AWS resources
Synthetics: Monitor application endpoints and APIs

Interview Topics

1. CloudWatch Metrics

Basic Metrics: CPU, memory, network, disk for EC2 instances
Custom Metrics: Application-specific metrics
Namespace: Logical container for related metrics
Dimensions: Key-value pairs to identify metrics
Resolution: Standard (1-minute) and high-resolution (1-second)

2. CloudWatch Logs

Log Groups: Logical container for log streams
Log Streams: Sequence of log events from a source
Log Events: Individual log entries with timestamp and message
Retention: Configurable log retention periods
Filtering: Search and filter log data

3. CloudWatch Alarms

Alarm States: OK, ALARM, INSUFFICIENT_DATA
Thresholds: Static and dynamic thresholds
Actions: SNS, SQS, Lambda, EC2 actions
Evaluation Periods: Number of periods to evaluate
Datapoints to Alarm: Number of datapoints within threshold

4. CloudWatch Dashboards

Widgets: Different types of visualization widgets
Layout: Customizable dashboard layout
Sharing: Share dashboards across accounts
Auto-refresh: Automatic data refresh
Cross-account: View metrics from multiple accounts

5. CloudWatch Events/EventBridge

Event Patterns: JSON patterns to match events
Targets: Lambda, SQS, SNS, Step Functions
Scheduled Events: Time-based event triggers
Cross-account: Events across AWS accounts
Custom Events: Application-specific events

Common Interview Questions

Basic Questions

What is Amazon CloudWatch and what are its main components?
- Monitoring and observability service
- Components: Metrics, Logs, Alarms, Dashboards, Events
- Provides unified view of operational health
What are the different types of CloudWatch metrics?
- Basic metrics: AWS service metrics (CPU, memory, etc.)
- Custom metrics: Application-specific metrics
- Detailed monitoring: High-resolution metrics
- Standard monitoring: 1-minute resolution

How do you create a CloudWatch alarm?

aws cloudwatch put-metric-alarm \
  --alarm-name "HighCPUAlarm" \
  --alarm-description "Alarm when CPU exceeds 80%" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2

Advanced Questions

How do you send custom metrics to CloudWatch?

import boto3

cloudwatch = boto3.client('cloudwatch')

cloudwatch.put_metric_data(
    Namespace='MyApplication',
    MetricData=[
        {
            'MetricName': 'RequestCount',
            'Value': 100,
            'Unit': 'Count',
            'Dimensions': [
                {
                    'Name': 'Environment',
                    'Value': 'Production'
                }
            ]
        }
    ]
)

What are CloudWatch Logs Insights and how do you use them?
- Query language for log analysis
- Real-time log querying
- Example query:
```
fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)
```
How do you set up cross-account CloudWatch monitoring?
- Use CloudWatch cross-account observability
- Configure IAM roles for cross-account access
- Set up centralized monitoring account
- Use CloudWatch Contributor Insights

Troubleshooting Questions

What if CloudWatch alarms are not triggering?
- Check alarm configuration and thresholds
- Verify metric data is being published
- Check IAM permissions
- Review alarm state and history
How do you handle high CloudWatch costs?
- Use custom metrics sparingly
- Implement log retention policies
- Use metric filters instead of full log ingestion
- Monitor and optimize metric resolution

Best Practices

1. Metric Design

Use meaningful metric names and namespaces
Implement proper dimensions for filtering
Choose appropriate units and statistics
Set up metric retention policies

2. Alarm Configuration

Set realistic thresholds based on baseline
Use multiple evaluation periods
Implement proper alarm actions
Test alarm configurations

3. Log Management

Structure logs consistently
Implement log retention policies
Use metric filters for important events
Centralize log collection

4. Dashboard Design

Create focused, purpose-specific dashboards
Use appropriate visualization types
Implement proper refresh intervals
Share dashboards with relevant teams

5. Cost Optimization

Monitor CloudWatch usage and costs
Use metric filters instead of full logs
Implement proper retention policies
Optimize custom metric publishing

Use Cases

1. Application Monitoring

Monitor application performance
Track business metrics
Alert on application errors
Visualize application health

2. Infrastructure Monitoring

Monitor AWS resource utilization
Track capacity and scaling
Alert on resource issues
Optimize resource usage

3. Security Monitoring

Monitor security events
Track access patterns
Alert on suspicious activity
Maintain audit trails

Integration Patterns

1. AWS Service Integration

EC2, RDS, Lambda, ECS monitoring
Auto Scaling integration
Load balancer monitoring
API Gateway metrics

2. Application Integration

Custom metric publishing
Log forwarding
Health check monitoring
Performance tracking

3. Third-party Integration

External monitoring tools
Log aggregation services
Alerting platforms
Visualization tools

Security Considerations

1. IAM Permissions

Use least privilege principle
Implement proper role-based access
Regular permission reviews
Cross-account access controls

2. Data Protection

Encrypt log data at rest
Secure metric data transmission
Implement proper access controls
Monitor access patterns

3. Compliance

Maintain audit trails
Implement data retention policies
Ensure regulatory compliance
Regular security assessments

Cost Optimization

1. Metric Optimization

Use standard monitoring where possible
Limit custom metric publishing
Implement proper metric retention
Monitor metric costs

2. Log Optimization

Use metric filters for important events
Implement log retention policies
Compress log data
Monitor log ingestion costs

3. Alarm Optimization

Limit number of alarms
Use composite alarms where appropriate
Implement proper evaluation periods
Monitor alarm costs

Performance Optimization

1. Metric Performance

Optimize metric publishing frequency
Use batch publishing for multiple metrics
Implement proper error handling
Monitor publishing performance

2. Query Performance

Optimize CloudWatch Logs Insights queries
Use appropriate time ranges
Implement query caching
Monitor query performance

3. Dashboard Performance

Limit dashboard complexity
Use appropriate refresh intervals
Implement proper widget sizing
Monitor dashboard performance

Disaster Recovery

1. Data Backup

Implement log backup strategies
Use cross-region replication
Maintain metric data backups
Test recovery procedures

2. Monitoring Continuity

Set up cross-region monitoring
Implement failover procedures
Maintain monitoring during disasters
Test disaster recovery procedures

3. Alerting Continuity

Ensure alerting works during disasters
Implement backup notification methods
Test alerting during recovery
Maintain incident response procedures

Migration Strategies

1. From Other Monitoring Tools

Map existing metrics and logs
Implement equivalent monitoring
Migrate dashboards and alerts
Test monitoring during migration

2. Application Migration

Implement CloudWatch monitoring
Migrate existing metrics
Set up new dashboards
Test monitoring thoroughly

3. Infrastructure Migration

Monitor migration progress
Set up CloudWatch for new resources
Migrate existing monitoring
Validate monitoring post-migration

Common Pitfalls

1. Over-monitoring

Too many metrics and alarms
High costs without value
Alert fatigue
Poor signal-to-noise ratio

2. Under-monitoring

Missing critical metrics
No alerting on important events
Poor visibility into issues
Delayed incident response

3. Poor Configuration

Incorrect alarm thresholds
Inadequate log retention
Poor dashboard design
Inefficient queries

Resources

Interview angle

“Metrics, logs, or traces - which do you reach for?” - metrics to know something is wrong and when it started, traces to find which service, logs to find why. Reaching for logs first is the slow path.
“What is a metric filter and why does it matter?” - it turns a log pattern into a metric, so you can alarm on something appearing in logs without paying to query them every minute. It is the cheap way to alert on application-level errors.
“How do you avoid a huge CloudWatch Logs bill?” - set retention on every log group (the default is never expire), do not log at DEBUG in production, sample high-volume events, and export to S3 for anything needed long term. Log ingestion is a common surprise line on the bill.
“What makes a good alarm?” - one that is actionable and fires on user-visible symptoms - error rate, p99 latency, queue age - not on causes like CPU. Alarms nobody acts on train the team to ignore the channel.
“How does this fit with OpenTelemetry?” - instrument with OTel and export to CloudWatch or a third-party backend. That keeps instrumentation portable rather than tied to one vendor’s SDK. See ../../../15_observability/07_opentelemetry.md.