During the day to day project implementations we see many subsystems which will exchange data to fulfill the business requirement. When we are designing these kind of requirements which involve data exchange with many subsystems, we will try to keep these systems as loosely coupled for better maintainability. To achieve loose coupling we have number of ways like web services, message driven, ESB etc… If we want to go with message driven approach, we can think of messaging services like IBM MQ Series, ActiveMQ, RabbitMQ etc… From Amazon, we have Amazon Simple Queue Service to achieve message oriented approach. In this article, we will see how the Amazon SQS will work and also the important points to keep in mind during the design of application with Amazon SQS.
Amazon Simple Queue Service offers a reliable, highly available, and scalable hosted queue to store the messages. To make the messages highly available, the Amazon distributed SQS will keep the messages into multiple AZ queue servers. The below diagrams depicts the architecture with Amazon SQS.
We need to consider the below points, if you choose to use Amazon SQS.
- The messages will be available up to maximum 14 days in the queue, provided that no delete action performed from our end.
- The order of the messages will not be guaranteed. So, it is better to keep sequence in the message itself if your application is message order dependable.
- The messages are highly available as the SQS will keep the redundant messages across the multiple AZ.
- The message retrievable will depend on the short polling or long polling.
- Short Polling: In the short polling, the Amazon SQS will sample subset of servers from the available SQS servers to retrieve the messages. Due to this, to get all the messages you need to poll multiple times. This technique will give empty responses to the receivers even though the queue doesn’t contain any messages.
- Long Polling: In the long polling, the Amazon SQS will poll all the servers to get the messages. So, with less number of calls to make to SQS to get the messages. In this mechanism, the queue will not send any response back to the receiver if there are no messages available in the queue.
- Some times receivers may get duplicate messages(due to redundant messages across multiple AZ for high availability). So, it is the receivers responsibility to eliminate the duplicates.
- Amazon SQS won’t delete the messages after consuming it. The consumer has to perform delete message operation after consuming it, because the distributed SQS is not guaranteed that the message is consumed by the consumer with out having any problem.
- As SQS won’t delete messages from the queue after consuming, we need to set the “visibility timeout“, which is a time period during which the SQS hide the messages to consume by other consumer(to avoid the same message to consume by another consumer). After successful delivery, the consumer will fire delete message call during the “visibility timeout“. If consumer failed to do so, the same message will be available to other consumers.
In the coming article, we will see how to setup the queue.
Siva Janapati is an Architect with experience in building Cloud Native Microservices architectures, Reactive Systems, Large scale distributed systems, and Serverless Systems. Siva has hands-on in architecture, design, and implementation of scalable systems using Cloud, Java, Go lang, Apache Kafka, Apache Solr, Spring, Spring Boot, Lightbend reactive tech stack, APIGEE edge & on-premise and other open-source, proprietary technologies. Expertise working with and building RESTful, GraphQL APIs. He has successfully delivered multiple applications in retail, telco, and financial services domains. He manages the GitHub(https://github.com/2013techsmarts) where he put the source code of his work related to his blog posts.
Smart Techie 
Leave a Reply