The Power of Distributed Messages
How multiple consumers can read the same data
A problem as old as the industry itself. We want the same data to be read by different consumers. What we need is a central repository of data (or messages) from which multiple consumers can read.
In the past, this problem was more more technically difficult and required a lot more boilerplate. Today, we can leverage cloud technology to achieve our desired goals in a much more robust and efficient way. Meet the following solution: AWS SQS combined with AWS SNS.
The Application
For a simple application, let’s say we have a user provisioning service, and two consumers which might be interested with user provisioning. Let’s also assume that as part of provisioning, a user enters their billing information, and is asked to set up a few settings on their account.
For this example, let’s assume we have two services; One is the billing consumer (call it Billing), which as the name implies would like to know about new users so that it can insert a new billing record into its database. The second consumer is the settings consumer (call it Settings), which has its own database and is responsible for CRUD operations on account settings. Since both services require to be updated upon new users being provisioned, an approach might be to make two POST API calls, one to Billing, and another two Settings. However, there are some problems with this approach. First, as we gain more users trying to create accounts simultaneously, this will increase the load on the client application, as the provisioning service must make 2 HTTP requests in sequence. If our client application wants to also wait for a response, the wait time will amortize and the consumers need to handle more requests.
What if we instead send a messages indicating that we want new users provisioned, and then we “wait“ for the consumers to receive the messages and perform the processing? This is the beauty of a message queue. Every time we need to create a new user in the system, the provisioning service can place a new message at the back of the queue, and the rest of the system can wait until the message is processed. AWS SQS is a queue technology we can leverage, but there are others out there as well (Redis for example can also be leveraged to create a message queue).
The Missing Piece of Distributed Messaging: The Notification System
We now have the knowledge of message queues. We know that we can use a queue to queue up information that other services might want to know about. The next question we need to answer is how would the queues be populated. Clearly, if we have only one central queue, only one service at a time would be reading from the queue at a time. Once the consumer finishes reading the message, it would either need to pop the message off the queue, or re-queue the message to try again, or to have another consumer try again. This will almost certainly result in a deadlock situation. The way around this is to move the centralization of messages to a different layer of the system. This is were AWS SNS can help. With SNS, we have a general topic which other queues can subscribe to. Each time a new message needs to be processed, SNS can send the message to all queues which subscribe to it. In our working example, both Billing and Settings can create their own queues to read from, and each of those queues would be populated by SNS. This is the power of combining SNS with SQS. With these two pieces of cloud technology, we have successfully distributed the workload of our application across multiple pieces. We also no longer have to be aware about deadlock, since there’s no way the queues could ever be locked (unless of course you create another consumer to read from the existing queue).
Putting It All Together
Img1 contains a crude diagram of the architecture.
Img1: Crude diagram