2025-12-10
SNS/SQS Cross-Account Fan-Out: Building Multi-Account Event Distribution in AWS
Learn how to implement secure cross-account event distribution using Amazon SNS and SQS. Covers IAM policies, KMS encryption, AWS CDK implementation, and common pitfalls from real-world deployments.
Abstract
Cross-account SNS/SQS fan-out enables secure event distribution across AWS account boundaries. This architecture pattern allows a single SNS topic in one account to deliver messages to multiple SQS queues in different accounts, maintaining administrative isolation while enabling event-driven communication. This guide covers the complete implementation including IAM policies, KMS encryption, AWS CDK setup, and troubleshooting common issues that emerge in production.
Why Cross-Account Fan-Out Matters
Working with multi-account AWS Organizations taught me that proper event distribution is essential for organizational scale. When you have separate accounts for different teams, services, or environments, you need a way to share events without compromising security boundaries.
The SNS/SQS fan-out pattern solves several real problems:
Administrative isolation: Each account maintains independent control over its resources. The billing team can’t accidentally delete the fulfillment team’s infrastructure, even though they both receive events from the same source.
Independent scaling: Consumer accounts scale their SQS processing independently. One slow consumer doesn’t impact others - messages queue up in their account while others continue processing.
Cost efficiency: SNS to SQS delivery is free (you only pay for SNS publishes and SQS operations). Compared to HTTP endpoints or other integration methods, this saves significant costs at scale.
Security boundaries: Each account implements its own encryption, access policies, and compliance controls. The security team can enforce strict key management in their account without requiring changes to the publisher.
Architecture Overview
Here’s how the cross-account fan-out pattern works:
The pattern requires proper configuration at three levels:
- SNS topic policy: Grants cross-account
sns:Subscribepermission - SQS queue policy: Allows SNS service principal to
sqs:SendMessage - KMS key policy (if encrypted): Permits SNS to encrypt/decrypt messages
IAM Policies and Permissions
Getting cross-account permissions right is critical. Here’s what I’ve learned works reliably.
SNS Topic Policy (Publisher Account)
The SNS topic must explicitly grant sns:Subscribe permission to target accounts:
import * as sns from 'aws-cdk-lib/aws-sns';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
export class PublisherStack extends Stack {
public readonly topic: sns.Topic;
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
// Create SNS topic
this.topic = new sns.Topic(this, 'CentralEventTopic', {
topicName: 'central-events',
displayName: 'Central Event Distribution Topic',
});
// Grant cross-account subscribe permissions
this.topic.addToResourcePolicy(
new iam.PolicyStatement({
sid: 'AllowCrossAccountSubscribe',
effect: iam.Effect.ALLOW,
principals: [
new iam.AccountPrincipal('111111111111'), // Account A
new iam.AccountPrincipal('222222222222'), // Account B
new iam.AccountPrincipal('333333333333'), // Account C
],
actions: ['sns:Subscribe'],
resources: [this.topic.topicArn],
})
);
// Optionally allow specific IAM roles instead of entire accounts
// This is more restrictive and follows least-privilege principle
this.topic.addToResourcePolicy(
new iam.PolicyStatement({
sid: 'AllowSpecificRoleSubscribe',
effect: iam.Effect.ALLOW,
principals: [
new iam.ArnPrincipal('arn:aws:iam::111111111111:role/ServiceARole'),
],
actions: ['sns:Subscribe'],
resources: [this.topic.topicArn],
})
);
}
}
Key considerations:
- Use
AccountPrincipalfor organization-wide access orArnPrincipalfor specific roles - The
sns:Subscribeaction is required for creating subscriptions - This policy doesn’t grant message publishing - only subscription creation
- You can add conditions to restrict by source VPC, IP range, or other factors
SQS Queue Policy (Consumer Account)
Each consumer account needs a queue policy allowing the SNS service principal to send messages:
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Duration, Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
interface ConsumerStackProps extends StackProps {
centralTopicArn: string; // ARN from publisher account
}
export class ConsumerStack extends Stack {
constructor(scope: Construct, id: string, props: ConsumerStackProps) {
super(scope, id, props);
// Create dead letter queue for failed messages
const dlq = new sqs.Queue(this, 'EventDLQ', {
queueName: 'events-dlq',
retentionPeriod: Duration.days(14),
});
// Create main event queue
const queue = new sqs.Queue(this, 'EventQueue', {
queueName: 'service-events',
visibilityTimeout: Duration.seconds(30),
receiveMessageWaitTime: Duration.seconds(20), // Enable long polling
deadLetterQueue: {
queue: dlq,
maxReceiveCount: 3,
},
});
// Add queue policy allowing SNS to send messages
queue.addToResourcePolicy(
new iam.PolicyStatement({
sid: 'AllowSNSPublish',
effect: iam.Effect.ALLOW,
principals: [new iam.ServicePrincipal('sns.amazonaws.com')],
actions: ['sqs:SendMessage'],
resources: [queue.queueArn],
conditions: {
ArnEquals: {
'aws:SourceArn': props.centralTopicArn,
},
},
})
);
// Import the cross-account SNS topic
const centralTopic = sns.Topic.fromTopicArn(
this,
'CentralTopic',
props.centralTopicArn
);
// Subscribe queue to topic
centralTopic.addSubscription(
new subscriptions.SqsSubscription(queue, {
rawMessageDelivery: false, // Set to true to receive just the message body
})
);
}
}
Important details:
- The
Conditionwithaws:SourceArnprevents other SNS topics from sending to your queue rawMessageDelivery: falsewraps the message in SNS metadata (recommended for debugging)- Set
rawMessageDelivery: trueif you only want the message body without SNS envelope - Long polling (
receiveMessageWaitTime) reduces empty receives and costs
The Two-Way Handshake
Cross-account subscriptions require both accounts to agree:
- Publisher permits subscription: SNS topic policy grants
sns:Subscribeto consumer account - Consumer accepts messages: SQS queue policy allows SNS service principal to send messages
- Consumer creates subscription: Queue owner calls
sns:Subscribeusing the topic ARN
This two-way handshake is crucial. If either policy is missing, you’ll get “Access Denied” errors. I’ve learned to always check both sides when troubleshooting subscription failures.
KMS Encryption Configuration
Encryption adds complexity to cross-account setups. AWS-managed keys don’t work across account boundaries - you must use customer-managed keys.
Why AWS-Managed Keys Don’t Work
When you create an SQS queue with encryption using the AWS-managed key (alias/aws/sqs), the key policy only grants permissions within that account. The SNS service in the publisher account can’t use a consumer account’s AWS-managed key.
Customer-Managed Key Setup
Here’s a working pattern for encrypted queues:
import * as kms from 'aws-cdk-lib/aws-kms';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as cdk from 'aws-cdk-lib';
import { Duration, Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
interface EncryptedConsumerStackProps extends StackProps {
centralTopicArn: string;
}
export class EncryptedConsumerStack extends Stack {
constructor(scope: Construct, id: string, props: EncryptedConsumerStackProps) {
super(scope, id, props);
// Create customer-managed KMS key
const queueKey = new kms.Key(this, 'QueueEncryptionKey', {
description: 'KMS key for cross-account SQS queue encryption',
enableKeyRotation: true,
removalPolicy: cdk.RemovalPolicy.RETAIN, // Don't delete keys
});
// Grant SNS service permission to use the key
queueKey.addToResourcePolicy(
new iam.PolicyStatement({
sid: 'AllowSNSToUseKey',
effect: iam.Effect.ALLOW,
principals: [new iam.ServicePrincipal('sns.amazonaws.com')],
actions: [
'kms:Decrypt',
'kms:GenerateDataKey',
],
resources: ['*'],
conditions: {
StringEquals: {
// Ensure SNS uses this key only for SQS in this region
'kms:ViaService': `sqs.${this.region}.amazonaws.com`,
},
},
})
);
// Create encrypted dead letter queue
const dlq = new sqs.Queue(this, 'EncryptedEventDLQ', {
queueName: 'encrypted-events-dlq',
encryptionMasterKey: queueKey,
retentionPeriod: Duration.days(14),
});
// Create encrypted main queue
const queue = new sqs.Queue(this, 'EncryptedEventQueue', {
queueName: 'encrypted-service-events',
encryptionMasterKey: queueKey,
visibilityTimeout: Duration.seconds(30),
receiveMessageWaitTime: Duration.seconds(20),
deadLetterQueue: {
queue: dlq,
maxReceiveCount: 3,
},
});
// Queue policy allowing SNS to send messages
queue.addToResourcePolicy(
new iam.PolicyStatement({
sid: 'AllowSNSPublish',
effect: iam.Effect.ALLOW,
principals: [new iam.ServicePrincipal('sns.amazonaws.com')],
actions: ['sqs:SendMessage'],
resources: [queue.queueArn],
conditions: {
ArnEquals: {
'aws:SourceArn': props.centralTopicArn,
},
},
})
);
// Import cross-account topic and subscribe
const centralTopic = sns.Topic.fromTopicArn(
this,
'CentralTopic',
props.centralTopicArn
);
centralTopic.addSubscription(
new subscriptions.SqsSubscription(queue)
);
}
}
Key policy requirements:
kms:Decrypt: SNS needs this to decrypt messages when sending to the queuekms:GenerateDataKey: Required for envelope encryptionkms:ViaServicecondition: Restricts key usage to SQS service in specific region- Enable key rotation for security best practices
Cost Consideration
Customer-managed KMS keys cost 0.03 per 10,000 requests. For cross-account scenarios, this is required - there’s no free alternative.
Message Filtering for Cost Optimization
SNS subscription filters reduce costs by preventing unwanted messages from reaching queues. Filtering happens at the SNS level before SQS charges apply.
Attribute-Based Filtering
Message attributes provide simple, efficient filtering:
// Publisher: Publish with attributes
import { SNSClient, PublishCommand } from '@aws-sdk/client-sns';
const sns = new SNSClient({ region: 'us-east-1' });
await sns.send(
new PublishCommand({
TopicArn: 'arn:aws:sns:us-east-1:999999999999:central-events',
Message: JSON.stringify({
orderId: '12345',
amount: 1500,
region: 'us-east-1',
}),
MessageAttributes: {
eventType: {
DataType: 'String',
StringValue: 'OrderCreated',
},
priority: {
DataType: 'String',
StringValue: 'high',
},
amount: {
DataType: 'Number',
StringValue: '1500',
},
region: {
DataType: 'String',
StringValue: 'us-east-1',
},
},
})
);
// Consumer: Subscribe with filter policy
import * as sns from 'aws-cdk-lib/aws-sns';
import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
centralTopic.addSubscription(
new subscriptions.SqsSubscription(highPriorityQueue, {
filterPolicy: {
// Only high priority events
priority: sns.SubscriptionFilter.stringFilter({
allowlist: ['high', 'critical'],
}),
// Only large orders
amount: sns.SubscriptionFilter.numericFilter({
greaterThan: 1000,
}),
},
})
);
centralTopic.addSubscription(
new subscriptions.SqsSubscription(regionalQueue, {
filterPolicy: {
// Only specific regions
region: sns.SubscriptionFilter.stringFilter({
allowlist: ['us-east-1', 'eu-west-1'],
}),
},
})
);
// Analytics queue receives everything (no filter)
centralTopic.addSubscription(
new subscriptions.SqsSubscription(analyticsQueue)
);
Payload-Based Filtering
Newer payload-based filtering (introduced in 2024) allows filtering on the message body itself:
import { SNSClient, SubscribeCommand } from '@aws-sdk/client-sns';
const sns = new SNSClient({ region: 'us-east-1' });
await sns.send(
new SubscribeCommand({
TopicArn: 'arn:aws:sns:us-east-1:999999999999:central-events',
Protocol: 'sqs',
Endpoint: 'arn:aws:sqs:us-east-1:111111111111:service-events',
Attributes: {
FilterPolicyScope: 'MessageBody',
FilterPolicy: JSON.stringify({
order: {
status: ['completed', 'shipped'],
amount: [{ numeric: ['>', 1000] }],
},
}),
},
})
);
Filter policy benefits:
- Reduces SQS request costs by 50-90% in typical scenarios
- Each subscriber receives only relevant messages
- Filter changes take up to 15 minutes to propagate
- Up to 200 filter policies per topic
FIFO Topics and Queues
FIFO (First-In-First-Out) topics provide strict ordering and exactly-once delivery. Use them when message order matters.
When to Use FIFO
FIFO makes sense for:
- Order processing workflows where sequence matters
- Financial transactions requiring exactly-once processing
- State machine transitions that must occur in order
- Inventory updates where order impacts final state
FIFO Setup Requirements
import * as sns from 'aws-cdk-lib/aws-sns';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
// Create FIFO topic (publisher account)
const fifoTopic = new sns.Topic(this, 'OrderEventTopic', {
topicName: 'order-events.fifo',
fifo: true,
contentBasedDeduplication: true,
// High-throughput mode: 3000+ TPS per message group
fifoThroughputScope: sns.FifoThroughputScope.MESSAGE_GROUP,
});
// Create FIFO queue (consumer account)
const fifoQueue = new sqs.Queue(this, 'OrderQueue', {
queueName: 'orders.fifo',
fifo: true,
contentBasedDeduplication: true,
});
// FIFO topic can only subscribe FIFO queues
fifoTopic.addSubscription(
new subscriptions.SqsSubscription(fifoQueue)
);
Publishing to FIFO topics:
import { SNSClient, PublishCommand } from '@aws-sdk/client-sns';
const sns = new SNSClient({ region: 'us-east-1' });
await sns.send(
new PublishCommand({
TopicArn: 'arn:aws:sns:us-east-1:999999999999:order-events.fifo',
Message: JSON.stringify({ orderId: '12345', status: 'created' }),
MessageGroupId: 'order-region-us-east-1', // Required for ordering
MessageDeduplicationId: 'order-12345-created', // Optional if contentBasedDeduplication enabled
})
);
High-throughput mode considerations:
- Default FIFO: 300 TPS with topic-level deduplication
- High-throughput mode: 30,000 TPS with message-group-level deduplication (increased January 2025)
- Cannot be reversed once enabled
- Use multiple message groups to parallelize processing
Common Pitfalls and Solutions
Here are issues I’ve encountered in production and their solutions.
Pitfall 1: Subscription Shows “PendingConfirmation”
Symptom: Subscription created but stuck in “PendingConfirmation” status. Messages never flow.
Root cause: When the topic owner creates the subscription (rather than the queue owner), SNS sends a confirmation message that must be manually confirmed.
Solution: Always have the queue owner create the subscription:
// PREFERRED: Queue owner subscribes (in consumer account)
const centralTopic = sns.Topic.fromTopicArn(
this,
'CentralTopic',
'arn:aws:sns:us-east-1:999999999999:central-events'
);
centralTopic.addSubscription(
new subscriptions.SqsSubscription(queue)
);
// No confirmation needed - subscription active immediately
If you must have the topic owner create subscriptions, automate confirmation:
# Python script to auto-confirm subscriptions
import boto3
import json
sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/111111111111/service-events'
# Poll for confirmation message
response = sqs.receive_message(
QueueUrl=queue_url,
MaxNumberOfMessages=1,
WaitTimeSeconds=10
)
for message in response.get('Messages', []):
body = json.loads(message['Body'])
if 'SubscribeURL' in body:
# Confirm subscription by visiting URL
import urllib.request
urllib.request.urlopen(body['SubscribeURL'])
print(f"Confirmed subscription: {body['SubscribeURL']}")
# Delete confirmation message
sqs.delete_message(
QueueUrl=queue_url,
ReceiptHandle=message['ReceiptHandle']
)
Pitfall 2: KMS Key Access Denied
Symptom: Messages published to SNS but never appear in encrypted SQS queue. No errors in SNS metrics.
Root cause: SNS service lacks permission to use the KMS key for encryption.
Solution: Verify KMS key policy grants SNS the required permissions:
// Check your KMS key policy includes this
queueKey.addToResourcePolicy(
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
principals: [new iam.ServicePrincipal('sns.amazonaws.com')],
actions: [
'kms:Decrypt',
'kms:GenerateDataKey',
],
resources: ['*'],
conditions: {
StringEquals: {
'kms:ViaService': `sqs.${this.region}.amazonaws.com`,
},
},
})
);
Troubleshooting tip: Check CloudTrail logs for KMS AccessDenied errors:
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
--max-results 50 \
--region us-east-1 \
--query 'Events[?ErrorCode==`AccessDenied`]'
Pitfall 3: Region Mismatch
Symptom: “Access Denied” errors despite correct policies.
Root cause: SNS topic and SQS queue in different regions. Cross-region direct subscriptions aren’t supported for cross-account scenarios.
Solution: Keep SNS topic and SQS queues in the same region. For multi-region requirements, use SNS to Lambda forwarders:
// Region 1 (us-east-1): Original topic
const sourceTopicEast = new sns.Topic(this, 'SourceTopicEast', {
topicName: 'events-us-east-1',
});
// Lambda forwarder publishes to region 2 topic
const forwarder = new lambda.Function(this, 'RegionForwarder', {
runtime: lambda.Runtime.NODEJS_20_X,
handler: 'index.handler',
code: lambda.Code.fromInline(`
const { SNSClient, PublishCommand } = require('@aws-sdk/client-sns');
const sns = new SNSClient({ region: 'eu-west-1' });
exports.handler = async (event) => {
for (const record of event.Records) {
await sns.send(new PublishCommand({
TopicArn: process.env.TARGET_TOPIC_ARN,
Message: record.Sns.Message,
MessageAttributes: record.Sns.MessageAttributes,
}));
}
};
`),
environment: {
TARGET_TOPIC_ARN: 'arn:aws:sns:eu-west-1:999999999999:events-eu-west-1',
},
});
sourceTopicEast.addSubscription(
new subscriptions.LambdaSubscription(forwarder)
);
Pitfall 4: Message Size Limits
Symptom: Some messages delivered successfully, others silently disappear.
Root cause: SNS and SQS both have 256 KB message size limits. Messages exceeding this are dropped without notification.
Solution: Keep messages under 256 KB or use S3 for large payloads:
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { SNSClient, PublishCommand } from '@aws-sdk/client-sns';
const s3 = new S3Client({ region: 'us-east-1' });
const sns = new SNSClient({ region: 'us-east-1' });
async function publishLargeMessage(topicArn: string, payload: any) {
const payloadSize = Buffer.byteLength(JSON.stringify(payload));
if (payloadSize > 200_000) {
// Store large payload in S3
const messageId = crypto.randomUUID();
const s3Key = `messages/${messageId}.json`;
await s3.send(
new PutObjectCommand({
Bucket: 'large-message-payloads',
Key: s3Key,
Body: JSON.stringify(payload),
})
);
// Publish reference to S3 object
await sns.send(
new PublishCommand({
TopicArn: topicArn,
Message: JSON.stringify({
type: 'S3Reference',
bucket: 'large-message-payloads',
key: s3Key,
}),
})
);
} else {
// Direct publish for small messages
await sns.send(
new PublishCommand({
TopicArn: topicArn,
Message: JSON.stringify(payload),
})
);
}
}
Pitfall 5: Filter Policies Not Taking Effect
Symptom: Messages still delivered despite filter policy.
Root cause: Filter policies take up to 15 minutes to propagate, or message attributes don’t match filter format.
Solution: Wait for propagation and verify attribute format:
// Verify message attributes match filter expectations
await sns.send(
new PublishCommand({
TopicArn: topicArn,
Message: JSON.stringify({ orderId: '12345' }),
MessageAttributes: {
eventType: {
DataType: 'String',
StringValue: 'OrderCreated', // Must match filter exactly
},
priority: {
DataType: 'Number', // Use Number type for numeric filters
StringValue: '5',
},
},
})
);
// Monitor filtering effectiveness
const filterMetric = new cloudwatch.Metric({
namespace: 'AWS/SNS',
metricName: 'NumberOfNotificationsFilteredOut',
dimensionsMap: {
TopicName: 'central-events',
},
});
Monitoring and Observability
Effective monitoring is essential for cross-account messaging. You need visibility into both publisher and consumer sides.
Key SNS Metrics
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as cloudwatch_actions from 'aws-cdk-lib/aws-cloudwatch-actions';
import * as sns from 'aws-cdk-lib/aws-sns';
// Monitor SNS delivery failures
new cloudwatch.Alarm(this, 'SNSDeliveryFailures', {
metric: topic.metricNumberOfNotificationsFailed({
statistic: 'Sum',
period: Duration.minutes(5),
}),
threshold: 10,
evaluationPeriods: 2,
treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
alarmDescription: 'Alert when SNS message delivery fails',
});
// Monitor filter effectiveness
new cloudwatch.Alarm(this, 'HighFilterRate', {
metric: topic.metricNumberOfNotificationsFilteredOut({
statistic: 'Sum',
period: Duration.minutes(5),
}),
threshold: 1000,
evaluationPeriods: 1,
comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
alarmDescription: 'Alert when filter rate is unusually high',
});
Key SQS Metrics
// Monitor queue depth
new cloudwatch.Alarm(this, 'QueueDepthAlarm', {
metric: queue.metricApproximateNumberOfMessagesVisible({
statistic: 'Average',
period: Duration.minutes(5),
}),
threshold: 1000,
evaluationPeriods: 2,
alarmDescription: 'Alert when queue depth grows too large',
});
// Monitor processing lag
new cloudwatch.Alarm(this, 'OldMessageAlarm', {
metric: queue.metricApproximateAgeOfOldestMessage({
statistic: 'Maximum',
period: Duration.minutes(1),
}),
threshold: 300, // 5 minutes
evaluationPeriods: 3,
alarmDescription: 'Alert when messages are not processed timely',
});
// Monitor DLQ
new cloudwatch.Alarm(this, 'DLQMessages', {
metric: dlq.metricApproximateNumberOfMessagesVisible({
statistic: 'Sum',
period: Duration.minutes(5),
}),
threshold: 1,
evaluationPeriods: 1,
alarmDescription: 'Alert on any messages in DLQ',
});
Cross-Account CloudWatch Observability
For unified monitoring across accounts, use CloudWatch Observability Access Manager:
import * as oam from 'aws-cdk-lib/aws-oam';
// In monitoring account: Create sink
const sink = new oam.CfnSink(this, 'MonitoringSink', {
name: 'central-monitoring-sink',
policy: {
Version: '2012-10-17',
Statement: [
{
Effect: 'Allow',
Principal: {
AWS: [
'arn:aws:iam::111111111111:root',
'arn:aws:iam::222222222222:root',
'arn:aws:iam::333333333333:root',
],
},
Action: ['oam:CreateLink', 'oam:UpdateLink'],
Resource: '*',
},
],
},
});
// In each source account: Create link to sink
const link = new oam.CfnLink(this, 'MonitoringLink', {
resourceTypes: ['AWS::CloudWatch::Metric', 'AWS::Logs::LogGroup'],
sinkIdentifier: 'arn:aws:oam:us-east-1:999999999999:sink/sink-id',
});
This enables a single dashboard showing metrics from all accounts without data transfer costs (within same region).
Cost Analysis
Understanding costs helps optimize your architecture.
Pricing Breakdown (2025)
SNS costs:
- First 1 million requests/month: FREE
- Beyond free tier: $0.50 per million publishes
- SNS to SQS deliveries: FREE (major cost advantage)
SQS costs:
- First 1 million requests/month: FREE
- Standard queue: $0.40 per million requests
- FIFO queue: $0.50 per million requests
- Each 64 KB chunk = 1 request (256 KB message = 4 requests)
Fan-out cost example (1 million messages to 4 queues):
- SNS publishes: 1M × 0.50
- SNS to SQS delivery: FREE
- SQS receives: 4M × 1.60
- SQS deletes: 4M × 1.60
- Total: $3.70
With 50% message filtering:
- SNS publishes: 1M × 0.50
- Filtered deliveries: 2M messages delivered
- SQS receives: 2M × 0.80
- SQS deletes: 2M × 0.80
- Total: $2.10 (43% cost reduction)
KMS costs (for encrypted queues):
- Customer-managed key: $1/month per key
- KMS requests: $0.03 per 10,000 requests
- Each encrypted message generates 2 KMS requests
Cost Optimization Strategies
- Implement message filtering: 50-70% cost reduction in typical scenarios
- Enable SQS long polling: Reduces empty receives by 90%
- Use batch operations: Up to 10× reduction in API calls
- Keep messages under 64 KB: Avoid multi-request charges
- Use Standard queues when ordering isn’t critical: 20% cheaper than FIFO
Alternative Approaches
SNS/SQS fan-out isn’t always the best choice. Here are alternatives and when to consider them.
EventBridge
When to use:
- Need complex event routing (100+ rules)
- Schema registry and validation required
- Event replay capability essential
- Integration with 30+ AWS services
Trade-offs:
- 0.50)
- More powerful filtering with JSONPath-like syntax
- Built-in schema discovery and validation
- Native cross-account event buses
import * as events from 'aws-cdk-lib/aws-events';
import * as targets from 'aws-cdk-lib/aws-events-targets';
const bus = new events.EventBus(this, 'CentralBus', {
eventBusName: 'central-events',
});
// Grant cross-account access
bus.grantPutEventsTo(new iam.AccountPrincipal('111111111111'));
// Complex filtering
new events.Rule(this, 'HighValueOrders', {
eventBus: bus,
eventPattern: {
source: ['com.myapp.orders'],
detailType: ['OrderCreated'],
detail: {
amount: [{ numeric: ['>', 1000] }],
region: ['us-east-1', 'us-west-2'],
status: ['pending', 'processing'],
},
},
targets: [new targets.SqsQueue(queue)],
});
Kinesis Data Streams
When to use:
- Ordered stream processing required
- Need replay capability (up to 365 days)
- Multiple consumers reading at different speeds
- Real-time analytics use cases
Trade-offs:
- More expensive ($0.015 per shard hour + PUT costs)
- Complex shard management
- Better for streaming analytics than discrete events
- Higher operational overhead
Direct Lambda Invocation
When to use:
- Synchronous processing acceptable
- Event volume under Lambda concurrent execution limits
- No need for queue management
- Simple, fast processing logic
Trade-offs:
- No built-in retry queues
- Cold start considerations
- Limited by Lambda concurrency
- Less flexible than queues for scaling
Real-World Implementation Pattern
Here’s a complete, production-ready multi-account setup:
// publisher-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Construct } from 'constructs';
export class PublisherStack extends cdk.Stack {
public readonly topicArn: string;
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const topic = new sns.Topic(this, 'CentralEvents', {
topicName: 'central-events',
displayName: 'Central Event Distribution',
});
// Allow multiple consumer accounts
topic.addToResourcePolicy(
new iam.PolicyStatement({
sid: 'AllowConsumerSubscribe',
principals: [
new iam.AccountPrincipal('111111111111'),
new iam.AccountPrincipal('222222222222'),
new iam.AccountPrincipal('333333333333'),
],
actions: ['sns:Subscribe'],
resources: [topic.topicArn],
})
);
this.topicArn = topic.topicArn;
// Output for cross-stack references
new cdk.CfnOutput(this, 'TopicArnOutput', {
value: topic.topicArn,
exportName: 'CentralEventTopicArn',
});
}
}
// consumer-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as lambdaEventSources from 'aws-cdk-lib/aws-lambda-event-sources';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as kms from 'aws-cdk-lib/aws-kms';
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import { Construct } from 'constructs';
interface ConsumerStackProps extends cdk.StackProps {
centralTopicArn: string;
serviceName: string;
}
export class ConsumerStack extends cdk.Stack {
constructor(scope: Construct, id: string, props: ConsumerStackProps) {
super(scope, id, props);
// Create KMS key for encryption
const encryptionKey = new kms.Key(this, 'EncryptionKey', {
description: `Encryption key for ${props.serviceName} events`,
enableKeyRotation: true,
});
// Grant SNS permission to use key
encryptionKey.addToResourcePolicy(
new iam.PolicyStatement({
principals: [new iam.ServicePrincipal('sns.amazonaws.com')],
actions: ['kms:Decrypt', 'kms:GenerateDataKey'],
resources: ['*'],
conditions: {
StringEquals: {
'kms:ViaService': `sqs.${this.region}.amazonaws.com`,
},
},
})
);
// Create DLQ
const dlq = new sqs.Queue(this, 'DLQ', {
queueName: `${props.serviceName}-dlq`,
encryptionMasterKey: encryptionKey,
retentionPeriod: cdk.Duration.days(14),
});
// Create main queue
const queue = new sqs.Queue(this, 'EventQueue', {
queueName: `${props.serviceName}-events`,
encryptionMasterKey: encryptionKey,
visibilityTimeout: cdk.Duration.seconds(30),
receiveMessageWaitTime: cdk.Duration.seconds(20),
deadLetterQueue: {
queue: dlq,
maxReceiveCount: 3,
},
});
// Allow SNS to send messages
queue.addToResourcePolicy(
new iam.PolicyStatement({
principals: [new iam.ServicePrincipal('sns.amazonaws.com')],
actions: ['sqs:SendMessage'],
resources: [queue.queueArn],
conditions: {
ArnEquals: {
'aws:SourceArn': props.centralTopicArn,
},
},
})
);
// Subscribe to central topic
const centralTopic = sns.Topic.fromTopicArn(
this,
'CentralTopic',
props.centralTopicArn
);
centralTopic.addSubscription(
new subscriptions.SqsSubscription(queue, {
rawMessageDelivery: false,
})
);
// Create processor Lambda
const processor = new lambda.Function(this, 'EventProcessor', {
runtime: lambda.Runtime.NODEJS_20_X,
handler: 'index.handler',
code: lambda.Code.fromInline(`
exports.handler = async (event) => {
for (const record of event.Records) {
const snsMessage = JSON.parse(record.body);
const message = JSON.parse(snsMessage.Message);
console.log('Processing message:', message);
// Your business logic here
// Message automatically deleted if handler succeeds
}
};
`),
timeout: cdk.Duration.seconds(30),
environment: {
SERVICE_NAME: props.serviceName,
},
});
// Connect queue to Lambda
processor.addEventSource(
new lambdaEventSources.SqsEventSource(queue, {
batchSize: 10,
reportBatchItemFailures: true,
})
);
// CloudWatch alarms
new cloudwatch.Alarm(this, 'QueueDepthAlarm', {
metric: queue.metricApproximateNumberOfMessagesVisible(),
threshold: 1000,
evaluationPeriods: 2,
});
new cloudwatch.Alarm(this, 'DLQMessagesAlarm', {
metric: dlq.metricApproximateNumberOfMessagesVisible(),
threshold: 1,
evaluationPeriods: 1,
});
new cloudwatch.Alarm(this, 'ProcessorErrorsAlarm', {
metric: processor.metricErrors(),
threshold: 10,
evaluationPeriods: 2,
});
}
}
Key Takeaways
Working with cross-account SNS/SQS taught me several important lessons:
Always use customer-managed KMS keys for encrypted cross-account queues. AWS-managed keys simply don’t work across account boundaries. The $1/month per key is unavoidable.
Have the queue owner create subscriptions. This eliminates the manual confirmation step and reduces setup complexity. When the topic owner creates subscriptions, you need automation to handle confirmation messages.
Implement comprehensive monitoring from the start. Cross-account troubleshooting is significantly harder without proper CloudWatch metrics. Set up alarms in both publisher and consumer accounts.
Filter at the SNS level with subscription filters. This reduces costs by 50-90% in typical scenarios. Filtering happens before SQS charges apply, making it highly cost-effective.
Keep SNS topics and SQS queues in the same region. Cross-region subscriptions add significant complexity. If you need multi-region distribution, use Lambda forwarders.
DLQs must be in the subscriber account. You can’t use a DLQ from the publisher account for cross-account subscriptions. Each consumer account needs its own DLQ.
Plan for 15-minute filter policy propagation. Don’t expect immediate changes when updating filter policies. Test filter changes in non-production first.
The SNS/SQS fan-out pattern provides reliable, cost-effective event distribution across AWS accounts. When you need persistent queues with independent consumer scaling and strong administrative boundaries, this architecture delivers excellent results. The implementation complexity is manageable once you understand the permission model and encryption requirements.
Related posts
Stop choosing based on features; choose based on your communication pattern. A practical guide to selecting between SQS, SNS, and EventBridge with working CDK examples and cost analysis.
A deep dive into building SaaS authorization with AWS Cognito and Verified Permissions. Covers Cedar policy language, multi-tenant patterns, JWT token flow, cost analysis, and common mistakes with TypeScript examples.
A vendor-neutral evaluation of external authorization platforms including AWS Verified Permissions, SpiceDB, OpenFGA, Cerbos, and OPA. Covers architecture patterns, cost analysis, and a decision framework for engineering teams.
A practical comparison of TypeScript AI SDKs for building AI agents - Vercel AI SDK, OpenAI Agents SDK, and AWS Bedrock integration. Includes code examples, decision frameworks, and production patterns.
A comprehensive technical guide to Amazon Cognito's advanced features including custom authentication flows, federation patterns, multi-tenancy architectures, migration strategies, and production-grade security implementation.