Introduction
Building a self-reporting real-time processing system enables fast, dynamic feedback loops that can transform your application’s user experience. Whether you’re handling financial transactions, messaging dashboards, or sensor data, the key is to design a system that can ingest volumes of data, process it efficiently, and relay updates back to users without delay.
This post provides a comprehensive, step-by-step guide to creating such a system using Node.js, TypeScript, React, DynamoDB, AWS Lambda, SNS, SQS, Redis, and LocalStack. We’ll walk through each major component—from writing data in DynamoDB to streaming real-time updates back to the client interface. By the end, you’ll have the foundations of a modern architecture that scales, remains fault-tolerant, and is fun to build.
Architecture Overview
Let’s start with a high-level picture of our real-time system:
- DynamoDB + Streams: Write your records or events (such as payments, orders, or sensor readings) to an Amazon DynamoDB table. Each new or modified record triggers an event through DynamoDB Streams.
- AWS Lambda: A serverless function, invoked by changes in DynamoDB Streams, processes event data, updates derived counters or states, and then optionally publishes messages to SNS or SQS for asynchronous workloads.
- SNS and SQS: SNS topics publish notifications to SQS queues for background processing. Separate worker services consume these queues, performing any heavy computations that don’t need to happen on the user-facing path.
- Redis (with concurrency locks): Serves as a cache and concurrency control mechanism to avoid duplicate or race-condition processing. Redis can also enable real-time messaging across services.
- React + WebSockets (or Socket.IO): The frontend receives real-time updates through a persistent socket connection, ensuring instant user feedback as data changes.
Next, let’s dive into how each piece fits together in a more detailed manner.
Step 1: DynamoDB, Streams & AWS Lambda
The first step is to provision a DynamoDB table that stores your events. We’ll enable Streams on that table so any new or updated items trigger a Lambda function. Below is a Node.js/TypeScript example of a simple Lambda function that processes stream events from DynamoDB:
import { DynamoDBStreamEvent } from "aws-lambda";
import { SNS } from "aws-sdk";
const snsClient = new SNS({ region: "us-east-1" });
const TOPIC_ARN = "arn:aws:sns:us-east-1:000000000000:my-topic";
export const handler = async (event: DynamoDBStreamEvent) => {
for (const record of event.Records) {
if (record.eventName === "INSERT") {
// Extract new item data
const newItem = record.dynamodb?.NewImage;
const paymentId = newItem?.PaymentId?.S || "N/A";
// Publish a message to SNS for further processing
await snsClient
.publish({
TopicArn: TOPIC_ARN,
Message: JSON.stringify({ paymentId }),
})
.promise();
}
}
return { statusCode: 200 };
};The function runs whenever a record is inserted (or updated/deleted if configured) in DynamoDB. You can add custom logic such as updating aggregated counters or filtering events before pushing them downstream.
Be sure to configure DynamoDB Streams, your Lambda event source mapping, and the necessary IAM roles so the function can publish to SNS.
Step 2: SNS, SQS & Worker Services
We often want more than a single Lambda function to handle logic. Instead, we can push complex or time-consuming tasks into a queue. AWS SNS can fan out event messages to multiple SQS queues, unlocking specialized worker services. For instance, one queue might handle advanced fraud checks while another deals with email notifications.
import { SQS } from "aws-sdk";
const sqsClient = new SQS({ region: "us-east-1" });
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/000000000000/my-queue";
// A simple loop to poll for new SQS messages
export async function pollQueue() {
while (true) {
const response = await sqsClient.receiveMessage({
QueueUrl: QUEUE_URL,
MaxNumberOfMessages: 10,
WaitTimeSeconds: 20,
}).promise();
if (response.Messages) {
for (const msg of response.Messages) {
console.log("Processing SQS message:", msg.Body);
// TODO: Place domain logic here (e.g. advanced rules, data analytics, etc.)
// Delete from queue after processing
await sqsClient.deleteMessage({
QueueUrl: QUEUE_URL,
ReceiptHandle: msg.ReceiptHandle!,
}).promise();
}
}
}
}Splitting your domain logic into multiple worker nodes offers scalability. As traffic grows, you can spin up more worker containers or instances to handle increased queue throughput.
Step 3: Redis for Locks & Caching
Redis is a natural fit for caching frequently accessed data and preventing duplicate or race-condition processing. By acquiring a lock before critical sections of your code, you avoid concurrency pitfalls. For example, if multiple workers pick up the same payment message, you don’t want them all closing out that payment.
import Redis from "ioredis";
const redis = new Redis();
async function acquireLock(lockKey: string, expirationInSeconds: number): Promise<boolean> {
const lockValue = Date.now().toString();
// NX => only set if not exists, EX => expire in seconds
const result = await redis.set(lockKey, lockValue, "NX", "EX", expirationInSeconds);
return result === "OK";
}
async function releaseLock(lockKey: string) {
await redis.del(lockKey);
}
// Usage in your domain logic
export async function processPayment(paymentId: string) {
const lockKey = `lock:payment:${paymentId}`;
const lockAcquired = await acquireLock(lockKey, 30);
if (!lockAcquired) {
console.log("Lock not acquired, skipping payment processing...");
return;
}
try {
// Proceed safely with domain logic
console.log(`Handling payment #${paymentId}`);
// e.g., update aggregator, publish additional events, etc.
} finally {
// Always release lock
await releaseLock(lockKey);
}
}Redis let’s you handle concurrency with ease while providing lightning-fast data retrieval for your aggregator values or ephemeral caches.
Step 4: React & Real-Time Frontend Updates
A key aspect of a self-reporting system is delivering changes back to users in real time. React can leverage Socket.IO or WebSockets to listen for updates. Whenever a new payment is processed or a counter changes, broadcast a message to all connected clients so they see the update instantly, no refresh needed.
import React, { useEffect, useState } from 'react';
import { io } from 'socket.io-client';
export default function RealTimeDashboard() {
const [updates, setUpdates] = useState<string[]>([]);
const socket = io("http://localhost:3001"); // Your server's socket endpoint
useEffect(() => {
socket.on("payment:update", (msg: string) => {
setUpdates(prev => [msg, ...prev]);
});
return () => {
socket.disconnect();
};
}, []);
return (
<div>
<h1>Real-Time Updates</h1>
<ul>
{updates.map((update, idx) => <li key={idx}>{update}</li>)}
</ul>
</div>
);
}On the server side, simply emit events once an SQS worker or Lambda completes a task (or once your aggregator is updated). This synergy between AWS-backed data pipelines and your Socket.IO channels keeps React components fully in sync.
Step 5: Local Development & Testing with LocalStack
Developing serverless workflows requires frequent iteration.LocalStack offers local emulation of AWS services—like DynamoDB, Lambda, SNS, and SQS—so you can develop and test without constantly deploying to AWS.
# Install LocalStack (via pip)
pip install localstack
# Start LocalStack
localstack start
# Create a dummy DynamoDB table for testing
awslocal dynamodb create-table --table-name Payments \
--attribute-definitions AttributeName=PaymentId,AttributeType=S \
--key-schema AttributeName=PaymentId,KeyType=HASH \
--billing-mode PAY_PER_REQUESTIn your Node.js application, configure AWS SDK clients to point at http://localhost:4566 (LocalStack’s default endpoint). For example:
import { DynamoDB } from "aws-sdk";
const ddbClient = new DynamoDB.DocumentClient({
endpoint: "http://localhost:4566",
region: "us-east-1",
accessKeyId: "test",
secretAccessKey: "test",
});
// Use ddbClient to read/write data from your local environment.
This approach drastically speeds up the development lifecycle—no more waiting for cloud deployments just to test a small code change.
Step 6: Deployment & Monitoring
Once your pipelines are stable in local development, you can containerize services and deploy them to Kubernetes. If you’re using serverless components like Lambda, then containerize only what needs hosting (React frontends, any Node.js workers, Redis, etc.), and let AWS handle the rest.
Tools like k9s simplify Kubernetes operations, letting you monitor logs, scale pods, and debug errors in real time:
# Mac example for installing k9s
brew install derailed/k9s
# Launch k9s to monitor your Kubernetes cluster
k9s
With k9s, you can watch logs from each worker or frontend pod, ensuring everything runs smoothly and diagnosing issues as they arise.
Step 7: Tracking Counters & Observability
Many real-time applications rely on aggregated counts or dashboards (e.g., total payments within the last hour). You can store these counters in a dedicated DynamoDB table or in Redis for fast reads. Whenever the Lambda or a worker finishes processing an event, increment the counters. For example:
async function incrementPaymentCounter() {
// In DynamoDB
await ddbClient.update({
TableName: "Counters",
Key: { counterType: "total-payments" },
UpdateExpression: "ADD #val :incr",
ExpressionAttributeNames: { "#val": "count" },
ExpressionAttributeValues: { ":incr": 1 },
}).promise();
}
// Alternatively, in Redis:
await redis.incr("total-payments");
For deeper observability, consider streaming metrics into Amazon CloudWatch or using OpenTelemetry for distributed traces. This can give you a holistic view of how your system behaves under load, from Lambda failures to real-time aggregator updates.
Step 8: Pitfalls & Best Practices
While building a self-reporting real-time system is exciting, there are common pitfalls to watch out for:
- Hot Partitions in DynamoDB: Overloading a single partition key can cause throttling or latency spikes. Distribute write load carefully, possibly using time-based partition keys or hashed IDs.
- Socket Overuse: Maintaining too many concurrent WebSocket connections can lead to performance issues. Consider scaling horizontally with a messaging middleware like Redis Pub/Sub or a managed service like AWS AppSync if your user base is massive.
- Visibility Timeout Mismatches: For SQS, if the worker’s processing time is longer than the visibility timeout, messages might reappear and get processed multiple times. Always configure visibility timeouts consistently with average or maximum processing durations.
- Inefficient Polling: Using tight polling loops for SQS can rack up unnecessary costs—consider using efficient or event-driven approaches like SQS Extended Client or building an SQS long poll worker.
- Lack of Tests & CI/CD: Real-time systems are especially prone to concurrency bugs. Automated tests, canary deployments, and CI/CD pipelines are crucial for smooth sailing in production.
By being mindful of these areas, you’ll build a more robust, maintainable real-time pipeline.
Conclusion
By combining Node.js, TypeScript, React, AWS services, Redis, and LocalStack, you can deliver fast and responsive applications without getting lost in endless complexity. DynamoDB Streams and AWS Lambda handle real-time event processing, SNS/SQS offloads heavier tasks to asynchronous workers, Redis locks prevent concurrency issues, and Socket.IO broadcasting keeps your React dashboards constantly in sync.
This architecture strikes a great balance between simplicity and scalability, allowing you to expand your system for higher traffic or advanced features (like AI-based analytics) without major rewrites. Hopefully, this guide helps you build next-level real-time applications that are as fun to implement as they are to use!
Further Reading
Below are some curated resources to extend your knowledge on serverless architecture, real-time messaging, and distributed locks.
Key Resources
Official AWS documentation on using DynamoDB Streams.
Simulate AWS services locally for easier development & testing.
Explanation of using Redis for concurrency control.