backend / cloud aws / security identity and compliance / README.md

AWS Secrets Manager

7 min read index source

AWS Secrets Manager

Overview

AWS Secrets Manager helps you protect the secrets needed to access your applications, services, and IT resources. The service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.

Key Features

  • Secret Storage: Secure storage for sensitive information
  • Automatic Rotation: Built-in rotation for database credentials
  • Encryption: AES-256 encryption for secrets
  • Access Control: IAM policies for secret access
  • Audit Logging: CloudTrail integration for access tracking
  • Cross-Region Replication: Replicate secrets across regions
  • Lambda Integration: Automatic rotation with Lambda functions
  • RDS Integration: Native integration with RDS databases

Interview Topics

1. Secrets Manager Fundamentals

  • Secrets: Sensitive data stored securely
  • Secret Values: The actual sensitive information
  • Secret Metadata: Information about the secret
  • Secret Versions: Different versions of secret values
  • Secret Rotation: Automatic or manual rotation

2. Secret Types

  • Database Credentials: RDS, Aurora, Redshift credentials
  • API Keys: Third-party service API keys
  • OAuth Tokens: Application OAuth credentials
  • SSH Keys: SSH private keys
  • Custom Secrets: Any sensitive data

3. Rotation Strategies

  • Automatic Rotation: Built-in rotation for supported services
  • Manual Rotation: Custom rotation using Lambda functions
  • Rotation Windows: Scheduled rotation periods
  • Rotation Triggers: Events that trigger rotation
  • Rotation Validation: Ensuring rotation success

4. Security Features

  • Encryption: AES-256 encryption at rest
  • KMS Integration: Customer Master Key (CMK) encryption
  • Access Control: IAM policies and resource policies
  • VPC Endpoints: Private network access
  • Cross-Account Access: Sharing secrets across accounts

5. Integration Patterns

  • Application Integration: SDK and CLI access
  • Lambda Integration: Serverless secret access
  • Container Integration: ECS/EKS secret injection
  • Database Integration: RDS automatic rotation
  • CI/CD Integration: Pipeline secret management

Common Interview Questions

Basic Questions

  1. What is AWS Secrets Manager and when would you use it?

    • Secure storage for sensitive information
    • Automatic rotation of credentials
    • Centralized secret management
    • Compliance and audit requirements

  2. What types of secrets can you store in Secrets Manager?

    • Database credentials
    • API keys and tokens
    • SSH private keys
    • OAuth credentials
    • Custom sensitive data

  3. How do you create a secret in Secrets Manager?

    aws secretsmanager create-secret \
  --name "MyDatabaseSecret" \
  --description "Database credentials" \
  --secret-string '{"username":"admin","password":"mypassword"}'

Advanced Questions

  1. How do you implement automatic rotation for a custom secret?

    import boto3
import json

def lambda_handler(event, context):
    # Get the secret
    client = boto3.client('secretsmanager')
    
    # Generate new secret value
    new_secret_value = generate_new_secret()
    
    # Update the secret
    response = client.update_secret(
        SecretId=event['SecretId'],
        SecretString=json.dumps(new_secret_value)
    )
    
    return {
        'statusCode': 200,
        'body': 'Secret rotated successfully'
    }

def generate_new_secret():
    # Custom logic to generate new secret
    return {
        'username': 'newuser',
        'password': 'newpassword'
    }

  2. How do you retrieve a secret in an application?

    import boto3
import json

def get_secret(secret_name):
    client = boto3.client('secretsmanager')
    
    try:
        response = client.get_secret_value(SecretId=secret_name)
        secret = json.loads(response['SecretString'])
        return secret
    except Exception as e:
        print(f"Error retrieving secret: {e}")
        raise

# Usage
db_credentials = get_secret('MyDatabaseSecret')
username = db_credentials['username']
password = db_credentials['password']

  3. How do you implement cross-account secret sharing?

    • Create secret in source account
    • Configure resource policy for target account
    • Grant necessary permissions to target account
    • Access secret from target account using ARN

Troubleshooting Questions

  1. What if an application can’t retrieve a secret?

    • Check IAM permissions
    • Verify secret name and region
    • Review resource policies
    • Check VPC endpoint configuration

  2. How do you troubleshoot secret rotation issues?

    • Check Lambda function logs
    • Verify rotation permissions
    • Review rotation schedule
    • Test rotation manually

Best Practices

1. Security

  • Use least privilege access
  • Enable encryption with customer CMK
  • Implement proper IAM policies
  • Monitor secret access

2. Rotation

  • Enable automatic rotation where possible
  • Implement custom rotation for unsupported services
  • Test rotation procedures
  • Monitor rotation success

3. Access Control

  • Use IAM roles for applications
  • Implement resource policies
  • Regular access reviews
  • Monitor access patterns

4. Monitoring

  • Enable CloudTrail logging
  • Set up CloudWatch alarms
  • Monitor rotation events
  • Track secret usage

5. Compliance

  • Maintain audit trails
  • Implement access logging
  • Regular security assessments
  • Ensure regulatory compliance

Use Cases

1. Database Credentials

  • RDS database passwords
  • Aurora cluster credentials
  • Redshift connection strings
  • DynamoDB access keys

2. API Management

  • Third-party API keys
  • OAuth tokens
  • Service account credentials
  • Webhook secrets

3. Application Secrets

  • Application configuration
  • Encryption keys
  • Certificate private keys
  • SSH private keys

4. Infrastructure Secrets

  • Load balancer certificates
  • VPN credentials
  • Monitoring API keys
  • Backup credentials

Integration Patterns

1. Application Integration

  • SDK integration
  • CLI access
  • Environment variables
  • Configuration files

2. Container Integration

  • ECS task definitions
  • EKS pod secrets
  • Docker secrets
  • Kubernetes integration

3. Serverless Integration

  • Lambda function secrets
  • API Gateway integration
  • Step Functions secrets
  • Event-driven access

4. Database Integration

  • RDS automatic rotation
  • Aurora cluster rotation
  • Redshift credential management
  • Custom database rotation

Security Considerations

1. Encryption

  • Use customer CMK for encryption
  • Enable encryption in transit
  • Implement proper key management
  • Monitor encryption status

2. Access Control

  • Implement least privilege
  • Use IAM roles for access
  • Regular permission reviews
  • Monitor access patterns

3. Network Security

  • Use VPC endpoints
  • Implement network isolation
  • Monitor network access
  • Secure communication channels

4. Audit and Compliance

  • Enable comprehensive logging
  • Maintain audit trails
  • Regular security assessments
  • Compliance monitoring

Cost Optimization

1. Secret Management

  • Consolidate similar secrets
  • Remove unused secrets
  • Optimize secret names
  • Monitor secret usage

2. Rotation Optimization

  • Use automatic rotation where possible
  • Optimize rotation schedules
  • Monitor rotation costs
  • Implement efficient rotation

3. Access Optimization

  • Optimize IAM policies
  • Use appropriate access patterns
  • Monitor access costs
  • Implement caching where appropriate

4. Storage Optimization

  • Optimize secret values
  • Remove unused versions
  • Monitor storage usage
  • Implement retention policies

Performance Optimization

1. Access Performance

  • Implement caching strategies
  • Optimize API calls
  • Use appropriate regions
  • Monitor response times

2. Rotation Performance

  • Optimize rotation functions
  • Use appropriate timeouts
  • Monitor rotation duration
  • Implement parallel rotation

3. Integration Performance

  • Optimize SDK usage
  • Use connection pooling
  • Monitor integration performance
  • Implement error handling

4. Scalability

  • Design for horizontal scaling
  • Monitor capacity limits
  • Implement auto-scaling
  • Plan for growth

Disaster Recovery

1. Secret Backup

  • Cross-region replication
  • Backup secret configurations
  • Document recovery procedures
  • Test recovery processes

2. Rotation Recovery

  • Backup rotation functions
  • Document rotation procedures
  • Test rotation recovery
  • Maintain rotation documentation

3. Access Recovery

  • Backup access configurations
  • Document access procedures
  • Test access recovery
  • Maintain access documentation

4. Service Continuity

  • Monitor service health
  • Implement failover procedures
  • Maintain operational procedures
  • Test recovery processes

Migration Strategies

1. From Manual Secret Management

  • Map existing secrets
  • Create equivalent secrets
  • Migrate applications gradually
  • Test thoroughly

2. From Other Secret Managers

  • Map existing secrets
  • Create equivalent configurations
  • Migrate secrets and applications
  • Validate functionality

3. Application Migration

  • Plan secret requirements
  • Configure Secrets Manager
  • Migrate applications
  • Test secret access

Common Pitfalls

1. Security Issues

  • Inadequate access control
  • Poor encryption configuration
  • Insufficient monitoring
  • Security misconfigurations

2. Rotation Problems

  • Failed rotation procedures
  • Inadequate testing
  • Poor error handling
  • Rotation misconfigurations

3. Integration Issues

  • Incorrect SDK usage
  • Poor error handling
  • Inadequate testing
  • Performance problems

4. Compliance Issues

  • Inadequate audit trails
  • Poor access logging
  • Insufficient monitoring
  • Compliance violations

Resources

Interview angle

  • “Secrets Manager or Parameter Store?” - Secrets Manager for credentials needing automatic rotation, cross-region replication and native RDS integration. SSM Parameter Store SecureString for configuration and secrets that do not rotate, at a fraction of the cost. Cost per secret is a genuine deciding factor at scale.
  • “How does rotation actually work?” - a Lambda implementing four steps: create the new secret, set it on the service, test it, then finish by moving the AWSCURRENT label. Two versions are valid during the window, which is what makes rotation zero-downtime - and why your application must fetch by label, not pin a version.
  • “How should an application read a secret?” - at startup or through a caching client, never per request. Uncached reads add latency, cost per API call, and hit throttling limits under load.
  • “What is the failure mode people miss?” - a rotated secret with clients holding the old value cached indefinitely. Cache with a TTL, and handle an auth failure by refreshing the secret and retrying once.
  • “How do you keep secrets out of the codebase entirely?” - inject at runtime from the secret store, scan commits with a pre-commit hook, and rotate anything that ever landed in git. A secret committed and then deleted is still in history and must be treated as compromised.