AWS Secrets Manager & Parameter Store: Security Best Practices

Engineers working with AWS face a common dilemma: choosing between Secrets Manager and Parameter Store for secrets management. While both services store sensitive data, they serve different purposes and come with different cost structures. This guide provides technical decision criteria, complete implementation patterns, and real-world lessons learned.

Understanding the Services

Before diving into implementation, let’s establish the technical differences between these services.

Service Comparison

Feature	Parameter Store Standard	Parameter Store Advanced	Secrets Manager
Cost	Free	$0.05/secret/month	$0.40/secret/month
Max Size	4 KB	8 KB	64 KB
Rotation	Manual only	Manual only	Automated with Lambda
Versioning	Single active version	Single active version	Multiple concurrent versions
Cross-Account	Via RAM (since 2024)	Via RAM (since 2024)	Native resource policies
Multi-Region	Manual replication	Manual replication	Automated replication
Encryption	Optional (SecureString)	Optional (SecureString)	Always encrypted (mandatory)
Native Integrations	Basic	Basic	RDS, Redshift, DocumentDB

Key Technical Insight: Parameter Store with SecureString uses KMS encryption and provides free basic secrets management. Secrets Manager adds automatic rotation, native RDS integration, and built-in versioning with staging labels.

Decision Framework

Use this technical decision tree to choose the right service:

Cost Analysis Example:

• 10 static API keys → Parameter Store Standard: $0/month
• 5 RDS passwords with rotation → Secrets Manager: $2.00/month
• 20 configuration values → Parameter Store Standard: $0/month
• Total: $2.00/month vs.$10.00/month if everything was in Secrets Manager

Cross-Account Secret Sharing

One of the most common requirements is sharing secrets between AWS accounts. Here’s the complete implementation pattern.

Architecture Overview

Implementation - Account A (Secret Owner)

// CDK code for Account A
import * as cdk from 'aws-cdk-lib';
import * as kms from 'aws-cdk-lib/aws-kms';
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';
import * as iam from 'aws-cdk-lib/aws-iam';

const kmsKey = new kms.Key(this, 'SecretKey', {
  enableKeyRotation: true,
  description: 'KMS key for cross-account secret sharing',
});

// Grant Account B permission to use the key
kmsKey.addToResourcePolicy(new iam.PolicyStatement({
  sid: 'AllowAccountBDecrypt',
  principals: [new iam.AccountPrincipal('222222222222')], // Account B
  actions: ['kms:Decrypt', 'kms:DescribeKey'],
  resources: ['*'],
  conditions: {
    StringEquals: {
      'kms:ViaService': 'secretsmanager.us-east-1.amazonaws.com',
    },
  },
}));

const secret = new secretsmanager.Secret(this, 'SharedSecret', {
  secretName: 'cross-account/database-credentials',
  encryptionKey: kmsKey,
});

// Add resource policy to secret
secret.addToResourcePolicy(new iam.PolicyStatement({
  sid: 'AllowAccountBAccess',
  principals: [new iam.AccountPrincipal('222222222222')],
  actions: ['secretsmanager:GetSecretValue', 'secretsmanager:DescribeSecret'],
  resources: ['*'],
}));

Implementation - Account B (Secret Consumer)

// CDK code for Account B
const applicationRole = new iam.Role(this, 'ApplicationRole', {
  assumedBy: new iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
  inlinePolicies: {
    'SecretAccess': new iam.PolicyDocument({
      statements: [
        new iam.PolicyStatement({
          effect: iam.Effect.ALLOW,
          actions: ['secretsmanager:GetSecretValue'],
          resources: [
            'arn:aws:secretsmanager:us-east-1:111111111111:secret:cross-account/database-credentials-*'
          ],
        }),
        new iam.PolicyStatement({
          effect: iam.Effect.ALLOW,
          actions: ['kms:Decrypt'],
          resources: [
            'arn:aws:kms:us-east-1:111111111111:key/12345678-1234-1234-1234-123456789012'
          ],
          conditions: {
            StringEquals: {
              'kms:ViaService': 'secretsmanager.us-east-1.amazonaws.com',
            },
          },
        }),
      ],
    }),
  },
});

Warning: Common Pitfall: Forgetting to grant KMS decrypt permission in Account B. The secret retrieval will fail with “AccessDeniedException” even if the Secrets Manager policy is correct.

Tip: Use CloudTrail to check for KMS API calls with error codes. Look for “Decrypt” operations that failed with “AccessDenied”.

Parameter Store Reference Pattern

You can standardize on Parameter Store API while storing actual secrets in Secrets Manager:

# Create a reference parameter
aws ssm put-parameter \
  --name "/app/database/password" \
  --value "{{resolve:secretsmanager:prod/database:SecretString:password}}" \
  --type "String" \
  --description "Reference to Secrets Manager secret"

Application code only needs Parameter Store SDK:

import { SSMClient, GetParameterCommand } from "@aws-sdk/client-ssm";

const client = new SSMClient({ region: "us-east-1" });
const command = new GetParameterCommand({
  Name: "/app/database/password",
  WithDecryption: true,
});

const response = await client.send(command);
console.log(response.Parameter.Value); // Actual secret from Secrets Manager

Benefit: Simplified application code, single API surface area, easier migration path between services.

Container Secrets Injection

There are multiple patterns for injecting secrets into containers, each with different trade-offs.

Pattern A: Environment Variable Injection (ECS)

This is the native ECS approach - secrets are injected at container startup.

// CDK - ECS Task Definition with secrets
const taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDef', {
  cpu: 256,
  memoryLimitMiB: 512,
});

taskDefinition.addContainer('app', {
  image: ecs.ContainerImage.fromRegistry('myapp:latest'),
  secrets: {
    // From Secrets Manager
    DB_PASSWORD: ecs.Secret.fromSecretsManager(dbSecret, 'password'),
    API_KEY: ecs.Secret.fromSecretsManager(apiKeySecret),

    // From Parameter Store
    CONFIG_VALUE: ecs.Secret.fromSsmParameter(configParam),
  },
  logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'app' }),
});

// Grant read permissions
dbSecret.grantRead(taskDefinition.taskRole);
apiKeySecret.grantRead(taskDefinition.taskRole);
configParam.grantRead(taskDefinition.taskRole);

Raw Task Definition JSON:

{
  "family": "myapp",
  "taskRoleArn": "arn:aws:iam::123456789012:role/myapp-task-role",
  "containerDefinitions": [
    {
      "name": "app",
      "image": "myapp:latest",
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/database-AbCdEf:password::"
        },
        {
          "name": "API_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:api-key-XyZaBc"
        },
        {
          "name": "CONFIG_VALUE",
          "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/app/config"
        }
      ]
    }
  ]
}

Warning: Critical Limitation: Secrets are injected ONLY at container startup. Rotated secrets require container restart (new task launch).

Pattern B: Runtime Retrieval with Caching

For applications that need to handle rotation without restarts:

// Application code - retrieve secrets at runtime
import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";

class SecretCache {
  private cache: Map<string, { value: string; expiry: number }> = new Map();
  private client: SecretsManagerClient;
  private ttlMs: number;

  constructor(ttlMs: number = 300000) { // 5 minutes default
    this.client = new SecretsManagerClient({});
    this.ttlMs = ttlMs;
  }

  async getSecret(secretId: string): Promise<string> {
    const now = Date.now();
    const cached = this.cache.get(secretId);

    if (cached && cached.expiry > now) {
      return cached.value;
    }

    const command = new GetSecretValueCommand({ SecretId: secretId });
    const response = await this.client.send(command);
    const value = response.SecretString!;

    this.cache.set(secretId, {
      value,
      expiry: now + this.ttlMs,
    });

    return value;
  }
}

// Lambda handler example
const secretCache = new SecretCache(300000); // Cache for 5 minutes

export const handler = async (event: any) => {
  const dbPassword = await secretCache.getSecret(process.env.DB_SECRET_ARN!);
  // Use password for database connection
};

Cost Analysis:

• Startup injection: 1 API call per container start (~$0.05/10,000 calls)
• Runtime retrieval with 5-min cache: 288 API calls/day per container ($1.44/month per container)
• Runtime retrieval per request: Potentially thousands of API calls (expensive, not recommended)

Pattern C: AWS Parameters and Secrets Lambda Extension

For Lambda functions, use the extension for built-in caching:

// Lambda function using extension
export const handler = async (event: any) => {
  // Extension runs as sidecar, provides local HTTP endpoint
  const response = await fetch(
    `http://localhost:2773/secretsmanager/get?secretId=${process.env.SECRET_ARN}`,
    {
      headers: {
        'X-Aws-Parameters-Secrets-Token': process.env.AWS_SESSION_TOKEN!,
      },
    }
  );

  const secret = await response.json();
  return { dbPassword: JSON.parse(secret.SecretString).password };
};

Benefits:

• Built-in caching (reduces API calls by ~90%)
• No code changes to application logic for caching
• Supports both Secrets Manager and Parameter Store

Deployment:

const lambdaFunction = new lambda.Function(this, 'Function', {
  // ... other config
  layers: [
    lambda.LayerVersion.fromLayerVersionArn(
      this,
      'ParametersAndSecretsLayer',
      `arn:aws:lambda:${this.region}:177933569100:layer:AWS-Parameters-and-Secrets-Lambda-Extension:11`
    ),
  ],
  environment: {
    PARAMETERS_SECRETS_EXTENSION_CACHE_ENABLED: 'true',
    PARAMETERS_SECRETS_EXTENSION_CACHE_SIZE: '1000',
    PARAMETERS_SECRETS_EXTENSION_HTTP_PORT: '2773',
  },
});

Tip: Cost Savings: Lambda extension reduces API calls by 99%. Costs drop from $5/month to$0.05/month for high-traffic functions.

EKS Secrets with CSI Driver

For Kubernetes workloads on EKS, use the Secrets Store CSI Driver for native integration.

Architecture Setup

# 1. Install Secrets Store CSI Driver
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system

# 2. Install AWS Provider
kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

SecretProviderClass Configuration

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/database"
        objectType: "secretsmanager"
        objectAlias: "db-creds"
        jmesPath:
          - path: username
            objectAlias: db-username
          - path: password
            objectAlias: db-password
      - objectName: "/app/config/api-endpoint"
        objectType: "ssmparameter"
        objectAlias: "api-endpoint"
  # Optional: Sync to Kubernetes Secret
  secretObjects:
    - secretName: database-secret
      type: Opaque
      data:
        - objectName: db-username
          key: username
        - objectName: db-password
          key: password

Pod Configuration with IRSA

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/app-secrets-access

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      serviceAccountName: app-service-account
      containers:
        - name: app
          image: myapp:latest
          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets"
              readOnly: true
          env:
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: database-secret
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: database-secret
                  key: password
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets"

IAM Role for Pod Identity

// CDK - Create IAM role for EKS Pod Identity (IRSA)
const podRole = new iam.Role(this, 'PodSecretsRole', {
  assumedBy: new iam.FederatedPrincipal(
    cluster.openIdConnectProvider.openIdConnectProviderArn,
    {
      StringEquals: {
        [`${cluster.openIdConnectProvider.openIdConnectProviderIssuer}:sub`]:
          'system:serviceaccount:production:app-service-account',
        [`${cluster.openIdConnectProvider.openIdConnectProviderIssuer}:aud`]:
          'sts.amazonaws.com',
      },
    },
    'sts:AssumeRoleWithWebIdentity'
  ),
});

// Grant access to secrets
dbSecret.grantRead(podRole);
configParam.grantRead(podRole);

// KMS permissions if using custom key
kmsKey.grant(podRole, 'kms:Decrypt');

Key Difference - IRSA vs Pod Identity:

• IRSA (older method): Requires OIDC provider setup, works with EKS 1.17+
• Pod Identity (newer method, 2024+): Simplified setup, better performance, requires EKS 1.24+

Secret Rotation Implementation

Automated rotation is one of the key benefits of Secrets Manager. Here’s how to implement it correctly.

Rotation Flow

Lambda Rotation Function - RDS MySQL

# Lambda rotation function
import boto3
import json
import pymysql
import os

secrets_client = boto3.client('secretsmanager')

def lambda_handler(event, context):
    secret_arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']

    # Dispatch to appropriate step
    if step == "createSecret":
        create_secret(secrets_client, secret_arn, token)
    elif step == "setSecret":
        set_secret(secrets_client, secret_arn, token)
    elif step == "testSecret":
        test_secret(secrets_client, secret_arn, token)
    elif step == "finishSecret":
        finish_secret(secrets_client, secret_arn, token)
    else:
        raise ValueError(f"Invalid step: {step}")

def create_secret(client, arn, token):
    # Check if version with AWSPENDING label already exists
    try:
        client.get_secret_value(
            SecretId=arn,
            VersionStage="AWSPENDING",
            VersionId=token
        )
        print("Secret version already exists")
        return
    except client.exceptions.ResourceNotFoundException:
        pass

    # Get current secret
    current_secret = client.get_secret_value(
        SecretId=arn,
        VersionStage="AWSCURRENT"
    )
    secret_dict = json.loads(current_secret['SecretString'])

    # Generate new password
    new_password = client.get_random_password(
        ExcludeCharacters='/@"\'\\',
        PasswordLength=32,
        ExcludePunctuation=False,
        RequireEachIncludedType=True
    )

    secret_dict['password'] = new_password['RandomPassword']

    # Store new secret with AWSPENDING label
    client.put_secret_value(
        SecretId=arn,
        ClientRequestToken=token,
        SecretString=json.dumps(secret_dict),
        VersionStages=['AWSPENDING']
    )

def set_secret(client, arn, token):
    # Get pending secret
    pending_secret = client.get_secret_value(
        SecretId=arn,
        VersionStage="AWSPENDING",
        VersionId=token
    )
    pending_dict = json.loads(pending_secret['SecretString'])

    # Get current secret for connection
    current_secret = client.get_secret_value(
        SecretId=arn,
        VersionStage="AWSCURRENT"
    )
    current_dict = json.loads(current_secret['SecretString'])

    # Connect to database with current credentials
    connection = pymysql.connect(
        host=current_dict['host'],
        user=current_dict['username'],
        password=current_dict['password'],
        database=current_dict['dbname'],
        connect_timeout=5
    )

    try:
        with connection.cursor() as cursor:
            # Update password in database
            sql = f"ALTER USER '{pending_dict['username']}'@'%' IDENTIFIED BY %s"
            cursor.execute(sql, (pending_dict['password'],))
        connection.commit()
    finally:
        connection.close()

def test_secret(client, arn, token):
    # Get pending secret
    pending_secret = client.get_secret_value(
        SecretId=arn,
        VersionStage="AWSPENDING",
        VersionId=token
    )
    pending_dict = json.loads(pending_secret['SecretString'])

    # Test connection with new credentials
    connection = pymysql.connect(
        host=pending_dict['host'],
        user=pending_dict['username'],
        password=pending_dict['password'],
        database=pending_dict['dbname'],
        connect_timeout=5
    )

    try:
        with connection.cursor() as cursor:
            # Execute simple query to verify access
            cursor.execute("SELECT 1")
            result = cursor.fetchone()
            if result[0] != 1:
                raise ValueError("Test query failed")
    finally:
        connection.close()

def finish_secret(client, arn, token):
    # Move AWSCURRENT label to new version
    metadata = client.describe_secret(SecretId=arn)
    current_version = None

    for version, stages in metadata['VersionIdsToStages'].items():
        if "AWSCURRENT" in stages:
            if version == token:
                # Already current, nothing to do
                return
            current_version = version
            break

    # Update version stages
    client.update_secret_version_stage(
        SecretId=arn,
        VersionStage="AWSCURRENT",
        MoveToVersionId=token,
        RemoveFromVersionId=current_version
    )

CDK Setup for Rotation

For RDS databases with built-in support:

// RDS database
const dbInstance = new rds.DatabaseInstance(this, 'Database', {
  engine: rds.DatabaseInstanceEngine.mysql({
    version: rds.MysqlEngineVersion.VER_8_0
  }),
  vpc,
  credentials: rds.Credentials.fromGeneratedSecret('admin'),
});

// Attach rotation to the secret
dbInstance.secret!.addRotationSchedule('RotationSchedule', {
  automaticallyAfter: cdk.Duration.days(30),
  hostedRotation: secretsmanager.HostedRotation.mysqlSingleUser(),
});

For custom applications:

// Lambda function for rotation
const rotationLambda = new lambda.Function(this, 'RotationFunction', {
  runtime: lambda.Runtime.PYTHON_3_12,
  handler: 'rotation.lambda_handler',
  code: lambda.Code.fromAsset('lambda/rotation'),
  timeout: cdk.Duration.minutes(5),
  vpc,
  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
});

// Grant permissions
dbSecret.grantRead(rotationLambda);
dbSecret.grantWrite(rotationLambda);

// Attach rotation
dbSecret.addRotationSchedule('CustomRotation', {
  rotationLambda,
  automaticallyAfter: cdk.Duration.days(30),
});

Warning: Common Pitfalls:

1. Network Access: Lambda needs VPC access to reach database. Configure VPC subnets correctly.
2. Timeout: Default 3 seconds is too short. Set to 5 minutes for rotation.
3. Permissions: Lambda needs both read and write to secret, plus KMS decrypt/encrypt.
4. Idempotency: Always check if AWSPENDING version exists before creating new one.
5. Connection Pooling: Open connections using old password won’t automatically get new password. Applications should handle connection refresh.

Alternating Users Strategy

For zero-downtime rotation in high-availability applications:

Architecture:

• Two database users: app_user_a and app_user_b
• Both users have identical permissions
• Rotation alternates which user’s password is updated
• Application always has one valid credential during rotation

Benefits:

• No downtime window
• Active connections continue working during rotation
• Suitable for applications that can’t handle connection refresh

Trade-off: Requires superuser credentials in separate secret to clone users.

Multi-Region Secrets Replication

For disaster recovery scenarios, Secrets Manager supports automatic replication.

Primary Secret with Replication

// Primary region secret with replication
const primarySecret = new secretsmanager.Secret(this, 'PrimarySecret', {
  secretName: 'prod/database-credentials',
  description: 'Production database credentials',
  replicaRegions: [
    {
      region: 'us-west-2',
      encryptionKey: replicaKmsKey, // Optional: use different KMS key
    },
    {
      region: 'eu-west-1',
    },
  ],
});

ARN Structure:

• Primary: arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/database-credentials-AbCdEf
• Replica: arn:aws:secretsmanager:us-west-2:123456789012:secret:prod/database-credentials-AbCdEf

Key Points:

• Secret suffix (-AbCdEf) is identical across regions
• Replication is automatic and near real-time
• Rotation in primary region propagates to replicas
• Each replica is billed as separate secret ($0.40/month each)
• Replicas are read-only, updates must happen in primary region

Disaster Recovery with Failover

// Application code with failover logic
class SecretService {
  private primaryRegion = 'us-east-1';
  private replicaRegion = 'us-west-2';
  private secretName = 'prod/database-credentials';

  async getSecretWithFailover(): Promise<string> {
    try {
      // Try primary region first
      return await this.getSecret(this.primaryRegion);
    } catch (error) {
      console.error('Primary region failed, trying replica', error);
      // Fallback to replica region
      return await this.getSecret(this.replicaRegion);
    }
  }

  private async getSecret(region: string): Promise<string> {
    const client = new SecretsManagerClient({ region });
    const command = new GetSecretValueCommand({
      SecretId: this.secretName,
    });

    const response = await client.send(command);
    return response.SecretString!;
  }
}

Cost Optimization Alternative

Instead of replication, use cross-region secret access (higher latency, lower cost):

// Access secret in us-east-1 from us-west-2 application
const client = new SecretsManagerClient({ region: 'us-east-1' });
const command = new GetSecretValueCommand({
  SecretId: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/database-credentials-AbCdEf',
});

const response = await client.send(command);

Trade-off: Saves $0.40/month per replica but adds cross-region API latency (~50-150ms).

Break-Glass Emergency Access

Emergency access procedures are critical for incident response. Here’s how to implement them securely.

Break-Glass Role Architecture

// CDK - Break-glass IAM role in all accounts
export class BreakGlassStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Break-glass role with broad permissions
    const breakGlassRole = new iam.Role(this, 'BreakGlassRole', {
      roleName: 'BREAK-GLASS-EMERGENCY-ACCESS',
      description: 'Emergency access role for incident response',
      maxSessionDuration: cdk.Duration.hours(2), // Short session
      assumedBy: new iam.AccountPrincipal('111111111111'), // Management account
    });

    // Grant access to all secrets
    breakGlassRole.addToPolicy(new iam.PolicyStatement({
      sid: 'SecretsManagerEmergencyAccess',
      effect: iam.Effect.ALLOW,
      actions: [
        'secretsmanager:GetSecretValue',
        'secretsmanager:DescribeSecret',
        'secretsmanager:ListSecrets',
      ],
      resources: ['*'],
    }));

    // Grant KMS decrypt for all keys
    breakGlassRole.addToPolicy(new iam.PolicyStatement({
      sid: 'KMSDecryptEmergencyAccess',
      effect: iam.Effect.ALLOW,
      actions: ['kms:Decrypt', 'kms:DescribeKey'],
      resources: ['*'],
      conditions: {
        StringEquals: {
          'kms:ViaService': [
            `secretsmanager.${this.region}.amazonaws.com`,
            `ssm.${this.region}.amazonaws.com`,
          ],
        },
      },
    }));

    // Grant Parameter Store access
    breakGlassRole.addToPolicy(new iam.PolicyStatement({
      sid: 'ParameterStoreEmergencyAccess',
      effect: iam.Effect.ALLOW,
      actions: [
        'ssm:GetParameter',
        'ssm:GetParameters',
        'ssm:GetParametersByPath',
      ],
      resources: ['*'],
    }));
  }
}

Monitoring Break-Glass Access

// EventBridge rule to detect break-glass access
const breakGlassAlertRule = new events.Rule(this, 'BreakGlassAlert', {
  eventPattern: {
    source: ['aws.sts'],
    detailType: ['AWS API Call via CloudTrail'],
    detail: {
      eventName: ['AssumeRole'],
      requestParameters: {
        roleArn: [{
          prefix: 'arn:aws:iam::*:role/BREAK-GLASS-EMERGENCY-ACCESS'
        }],
      },
    },
  },
});

// SNS topic for security team
const securityTopic = new sns.Topic(this, 'SecurityAlertTopic', {
  displayName: 'Critical Security Alerts',
});

breakGlassAlertRule.addTarget(new targets.SnsTopic(securityTopic, {
  message: events.RuleTargetInput.fromEventPath(
    '$.detail.userIdentity.principalId has assumed break-glass role'
  ),
}));

Emergency Access Procedure

Activation:

1. Security team retrieves break-glass password from physical safe
2. Second person retrieves YubiKey from separate secure location
3. Both must be present (two-person rule)

Access:

# Configure AWS CLI with break-glass user
aws configure --profile break-glass

# Assume role in target account
aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/BREAK-GLASS-EMERGENCY-ACCESS \
  --role-session-name "incident-2024-11-30-database-outage" \
  --serial-number arn:aws:iam::111111111111:mfa/EMERGENCY-BREAK-GLASS \
  --token-code 123456 \
  --profile break-glass

# Export temporary credentials
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_SESSION_TOKEN="..."

# Access secrets
aws secretsmanager get-secret-value \
  --secret-id prod/database-credentials \
  --query SecretString \
  --output text

Post-Incident:

• Revoke temporary credentials immediately
• Rotate all accessed secrets within 4 hours
• Document all actions taken in incident report
• Review CloudTrail logs for complete audit trail
• Conduct post-mortem on why break-glass was needed

Audit Logging with CloudTrail

Comprehensive audit logging is essential for security and compliance.

CloudTrail Configuration

// S3 bucket for CloudTrail logs
const trailBucket = new s3.Bucket(this, 'CloudTrailBucket', {
  bucketName: `cloudtrail-logs-${this.account}`,
  encryption: s3.BucketEncryption.S3_MANAGED,
  blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
  versioned: true,
  lifecycleRules: [
    {
      transitions: [
        {
          storageClass: s3.StorageClass.INTELLIGENT_TIERING,
          transitionAfter: cdk.Duration.days(30),
        },
        {
          storageClass: s3.StorageClass.GLACIER,
          transitionAfter: cdk.Duration.days(90),
        },
      ],
      expiration: cdk.Duration.days(2555), // 7 years for compliance
    },
  ],
});

// CloudTrail trail
const trail = new cloudtrail.Trail(this, 'SecurityAuditTrail', {
  bucket: trailBucket,
  includeGlobalServiceEvents: true,
  isMultiRegionTrail: true,
  enableFileValidation: true,
  sendToCloudWatchLogs: true,
});

// Data events for Secrets Manager
trail.addEventSelector({
  readWriteType: cloudtrail.ReadWriteType.ALL,
  includeManagementEvents: true,
  dataResources: [
    {
      type: 'AWS::SecretsManager::Secret',
      values: ['arn:aws:secretsmanager:*:*:secret:*'],
    },
  ],
});

Warning: Important: CloudTrail only logs management events by default. Data events (including GetSecretValue) must be explicitly enabled.

Note: ~$0.10/month per 100,000 events. For 10 secrets accessed 1,000 times/month each:$0.01/month.

Athena Queries for Analysis

-- Find all secret access events
SELECT
  eventTime,
  userIdentity.principalId,
  userIdentity.arn,
  eventName,
  json_extract_scalar(requestParameters, '$.secretId') as secretId,
  sourceIPAddress,
  errorCode
FROM cloudtrail_logs
WHERE eventSource = 'secretsmanager.amazonaws.com'
  AND eventName IN ('GetSecretValue', 'PutSecretValue', 'DeleteSecret')
  AND year = '2025' AND month = '12' AND day = '01'
ORDER BY eventTime DESC;

-- Cross-account secret access
SELECT
  eventTime,
  userIdentity.accountId as callerAccount,
  json_extract_scalar(requestParameters, '$.secretId') as secretArn,
  eventName,
  errorCode
FROM cloudtrail_logs
WHERE eventSource = 'secretsmanager.amazonaws.com'
  AND userIdentity.accountId != regexp_extract(
    json_extract_scalar(requestParameters, '$.secretId'),
    'arn:aws:secretsmanager:[^:]+:([^:]+):',
    1
  )
  AND year = '2025' AND month = '12'
ORDER BY eventTime DESC;

Cost Analysis & Optimization

Understanding the cost structure helps you optimize spending without compromising security.

Detailed Cost Scenarios

Scenario 1: 10 Static API Keys (No Rotation)

• Parameter Store Standard: $0/month (free tier)
• Secrets Manager: $4.00/month
• Recommendation: Parameter Store Standard
• Savings: $4.00/month

Scenario 2: 5 RDS Passwords (Monthly Rotation)

• Parameter Store: $0.25/month + manual rotation labor + downtime risk
• Secrets Manager: $2.00/month +$0 rotation = $2.00/month
• Recommendation: Secrets Manager
• ROI: Automation worth the cost

Scenario 3: Lambda with High Traffic

• Without Extension: 1M invocations/month × 1 API call = $5.00/month
• With Extension: API calls reduced by 99% = $0.05/month
• Savings: $4.95/month (99% reduction)

Cost Optimization Strategies

Strategy 1: Hybrid Approach

Use Parameter Store for static configuration, Secrets Manager for rotating credentials:

// Static values in Parameter Store (free)
const apiEndpoint = ssm.StringParameter.fromStringParameterAttributes(
  this, 'ApiEndpoint', {
    parameterName: '/app/api/endpoint',
  }
);

// Rotating credentials in Secrets Manager (paid)
const dbSecret = secretsmanager.Secret.fromSecretNameV2(
  this, 'DbSecret',
  'prod/database'
);

Strategy 2: Consolidate Secrets

Instead of separate secrets for each credential component:

// Wrong: Multiple secrets ($1.20/month)
const dbUsername = new secretsmanager.Secret(this, 'DbUser');
const dbPassword = new secretsmanager.Secret(this, 'DbPass');
const dbHost = new secretsmanager.Secret(this, 'DbHost');

// Right: Single secret ($0.40/month)
const dbCredentials = new secretsmanager.Secret(this, 'DbCreds', {
  secretObjectValue: {
    username: cdk.SecretValue.unsafePlainText('admin'),
    password: cdk.SecretValue.unsafePlainText('generated'),
    host: cdk.SecretValue.unsafePlainText('db.example.com'),
    port: cdk.SecretValue.unsafePlainText('3306'),
  },
});
// Savings: $0.80/month per database

Strategy 3: Selective Replication

Only replicate critical production secrets:

// Only replicate critical production database secrets
if (secretName.includes('/prod/database') || secretName.includes('/prod/auth')) {
  secret.addReplicaRegion('us-west-2', replicaKmsKey);
}

// Non-critical secrets: use cross-region API calls
// Acceptable for: API keys, static tokens, config values

Cost Analysis:

• 20 secrets, 2 replicas: $24/month
• 5 critical replicated + 15 primary only: $10/month
• Savings: $14/month (58% reduction)

Common Pitfalls & Solutions

Here are the technical issues I’ve encountered and how to solve them.

Pitfall 1: Default KMS Key for Cross-Account Access

Problem: Cross-account sharing fails with “AccessDeniedException” when using default aws/secretsmanager key.

Root Cause: AWS-managed keys cannot have their policy modified for cross-account access.

Solution: Always create customer-managed KMS keys:

// Wrong: Uses default key
const secret = new secretsmanager.Secret(this, 'Secret', {
  secretName: 'shared-secret',
});

// Right: Customer-managed key
const kmsKey = new kms.Key(this, 'SharedSecretKey', {
  enableKeyRotation: true,
});
kmsKey.addToResourcePolicy(/* cross-account policy */);

const secret = new secretsmanager.Secret(this, 'Secret', {
  secretName: 'shared-secret',
  encryptionKey: kmsKey,
});

Pitfall 2: Lambda VPC Configuration for Rotation

Problem: Rotation Lambda times out connecting to RDS in VPC.

Root Cause: Lambda not configured with VPC access.

Solution:

const rotationLambda = new lambda.Function(this, 'RotationFunction', {
  // ...
  vpc: database.vpc,
  vpcSubnets: {
    subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
  },
  securityGroups: [rotationSecurityGroup],
  timeout: cdk.Duration.minutes(5), // Not default 3 seconds
});

// Allow Lambda to access RDS
rotationSecurityGroup.addEgressRule(
  database.connections.securityGroups[0],
  ec2.Port.tcp(3306),
  'Allow rotation Lambda to access database'
);

Pitfall 3: ECS Secrets Only Injected at Startup

Problem: After rotation, containers fail with authentication errors.

Root Cause: ECS injects secrets only at startup.

Solution: Implement graceful connection handling:

class DatabaseConnection {
  private pool: any;
  private secretId: string;
  private secretCache: { value: string; expiry: number } | null = null;

  async getConnection() {
    try {
      return await this.pool.getConnection();
    } catch (error) {
      if (this.isAuthError(error)) {
        console.log('Auth error detected, refreshing secret');
        await this.refreshSecret();
        this.pool = this.createPool(this.secretCache!.value);
        return await this.pool.getConnection();
      }
      throw error;
    }
  }

  private async refreshSecret() {
    const command = new GetSecretValueCommand({ SecretId: this.secretId });
    const response = await secretsClient.send(command);
    this.secretCache = {
      value: response.SecretString!,
      expiry: Date.now() + 300000, // 5 minutes
    };
  }
}

Pitfall 4: Excessive Lambda API Calls

Problem: Secrets Manager costs spike to $50+/month.

Root Cause: Fetching secret on every invocation without caching.

Solution: Use Lambda Extension (shown earlier in Pattern C).

Result: 99% cost reduction.

Pitfall 5: Missing CloudTrail Data Events

Problem: No audit trail for GetSecretValue operations.

Root Cause: Data events not enabled by default.

Solution: Enable data event logging (shown in Audit Logging section).

Pitfall 6: Storing Non-Secret Config in Secrets Manager

Problem: Paying $0.40/month for non-sensitive values.

Solution: Use decision framework:

Is it sensitive? (password, API key, token)
├─ YES → Can it rotate?
│  ├─ YES → Secrets Manager ($0.40/month)
│  └─ NO → Parameter Store SecureString (free)
└─ NO → Parameter Store Standard (free)

Key Takeaways

Working with AWS secrets management has taught me these important lessons:

1. Service Selection is About Use Case: Reserve Secrets Manager for rotating credentials. Use Parameter Store for everything else. This simple rule can save 80% on costs.

2. Cross-Account Access Requires Customer-Managed Keys: The default aws/secretsmanager key won’t work. Create customer-managed KMS keys from day one to avoid migration pain.

3. Container Injection is One-Time: Secrets injected at startup don’t update on rotation. Design applications to handle connection refresh or use alternating-users strategy.

4. Lambda Extension Reduces Costs by 99%: For high-traffic Lambda functions, the extension’s built-in caching is essential. It’s a one-line addition that saves significant money.

5. CloudTrail Data Events are Critical: Enable them from day one. The cost is negligible (~$0.10 per 100,000 events) but the audit value is immeasurable.

6. Multi-Region Replication is a Business Decision: Don’t replicate everything. Analyze RTO/RPO requirements and replicate only critical secrets. Cross-region API calls are often acceptable.

7. Break-Glass Procedures Need Testing: Untested emergency access is useless during incidents. Test quarterly to validate both technical and organizational readiness.

8. Automation Beats Process: Manual rotation costs $250/month in engineer time. Automated rotation costs$4/month. ROI is immediate.

The key is balancing security, cost, and operational complexity. Start simple with Parameter Store for static config, migrate sensitive credentials to Secrets Manager, implement rotation for databases, and add cross-region replication only where needed.