Getting Started with Amazon Simple Queue Service (AWS SQS)

What is Amazon Simple Queue Service

Amazon Simple Queue Service (Amazon SQS) offers reliable and scalable hosted queues for storing messages as they travel between computers.

By using Amazon SQS, you can move data between distributed components of your applications that perform different tasks without losing messages or requiring each component to be always available.

Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing.

Like other Message bus systems using Amazon SQS, you can decouple the components of an application so they run independently, with Amazon SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages programmatically using the Amazon SQS API. Messages larger than 256 KB can be managed using the Amazon SQS Extended Client Library for Java, which uses Amazon S3 for storing larger payloads.

Pricing

As of this post, these applied to all regions except Asia Pacific (Tokyo) and GovCloud (US)

  • First 1 million Amazon SQS Requests per month are free
  • $0.50 per 1 million Amazon SQS Requests per month thereafter ($0.00000050 per SQS Request)
  • All SQS API calls count as a Request
  • A single Request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
  • Each 64KB ‘chunk’ of payload is billed as 1 Request. For example, a single API call with a 256KB payload will be billed as four Requests.

Asia Pacific (Tokyo) region only

  • First 1 million Amazon SQS Requests per month are free
  • $0.476 per 1 million Amazon SQS Requests per month thereafter ($0.000000476 per SQS Request)
  • All SQS API calls count as a Request
  • A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
  • Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.

GovCloud (US) region only

  • No free tier
  • $0.60 per 1 million Amazon SQS Requests per month thereafter ($0.00000060 per SQS Request)
  • All SQS API calls count as a Request
  • A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
  • Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.
Get Started

First thing we need to do is make sure we have a user set up in the Identity and Access Management. We need a user that has access keys see here I cover creating a new user with access keys.

Now in Services search for SQS

AWS_Create_SQS

Select "Create new Queue"

AWS_SQS_CreateQueue

I had already changed my region to EU Ireland( see the top right of the AWS Console you can change the region)

I just left things at the defaults for the moment we'll play with them later. Go ahead and create a queue.

Once your queue is created you will see some information in the panel at the bottom of the screen, mainly the Name followed by a URL, for me it's

https://sqs.eu-west-1.amazonaws.com/54XXXXXXX57/serversncodeq

AWS_SQS_Details

We need that URL. The next part and this caught me first time around is the Permissions, There is no default set so we need to do that, you can click the permission tab or Right click the queue and select "Add a Permissions"

AWS Permission

For the moment I select Everybody and All actions but you can see how we can use user accounts to control access to the queues, who can write and read.

That's it we have a queue and user account so we can access the queue

Let's write some code

The below sample is in C# for the code see Github

AWS SQS Read Console Sample Application

AWS SQS Write Console Sample Application

Write a message to the queue.

AWS SQS Write Console Sample Application

In a C# console app, in Package manager run

PM> Install-Package AWSSDK.SQS

This will install the SDK for SQS so we can get going. Here is the code for that will write a message to the queue.

        string sQueue = "https://sqs.eu-west-1.amazonaws.com/{CODE}/{QUEUENAME}";

        try
        {
            var config = new AmazonSQSConfig()
            {
                ServiceURL = "https://sqs.eu-west-1.amazonaws.com/"
            };

            var _messageRequest = new SendMessageRequest();

            _messageRequest.QueueUrl = sQueue;

            _messageRequest.MessageBody = "This is a test message";

            AmazonSQSClient _client = new AmazonSQSClient("ACCESSKEY", "ACCESSSECRET", config);

            SendMessageResponse sendMessageResponse = _client.SendMessage(_messageRequest);

        }
        catch (Exception ex)
        {
            Console.WriteLine("Error: " + ex.ToString());
        }

So lets take a wander through what we have just done.

First up we have the Queue URL we looked at earlier, I store it off in a string.

We then set the config.

            var config = new AmazonSQSConfig()
            {
                ServiceURL = "https://sqs.eu-west-1.amazonaws.com/" 
            };

The Config Service URL should be the service, region and then amazonaws.com This will tell the SQS Client where we want to connect.

Next we create our MessageRequest

            var _messageRequest = new SendMessageRequest();

We set the Message Request Queue URL to the string we set at the top ("https://sqs.eu-west-1.amazonaws.com/{CODE}/{QUEUENAME}")

            _messageRequest.QueueUrl = sQueue;

            _messageRequest.MessageBody = "This is a test message";

Now we want to set the Amazon SQS client. We pass our Access key and Access Secret we setup in the IAM and then pass the config which will set everything to the region we want.

            AmazonSQSClient _client = new AmazonSQSClient("ACCESSKEY", "ACCESSSECRET", config);

We now send the message.

            SendMessageResponse sendMessageResponse = _client.SendMessage(_messageRequest);
Read a message to the queue.

Sample Code here

In a C# console app, in Package manager run

PM> Install-Package AWSSDK.SQS

To read the message we just created our code will look like this

        string sQueue = "https://sqs.eu-west-1.amazonaws.com/{CODE}/{QUEUENAME}"; 

        try
        {
            var config = new AmazonSQSConfig()
            {
                ServiceURL = "https://sqs.eu-west-1.amazonaws.com/" 
            };
            
            AmazonSQSClient _client = new AmazonSQSClient("ACCESSKEY", "ACCESSSECRET", config);

            ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest();

            receiveMessageRequest.QueueUrl = sQueue;

            ReceiveMessageResponse receiveMessageResponse = _client.ReceiveMessage(receiveMessageRequest);
            
            foreach (var oMessage in receiveMessageResponse.Messages)
            {
                Console.WriteLine(oMessage.Body);

                // Delete the message from the queue
                DeleteMessageRequest deleteMessageRequest = new DeleteMessageRequest();

                deleteMessageRequest.QueueUrl = sQueue;
                deleteMessageRequest.ReceiptHandle = oMessage.ReceiptHandle;

                DeleteMessageResponse response = _client.DeleteMessage(deleteMessageRequest);
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine("Error: " + ex.ToString());
        }

Now lets take a wander through what we just did.

        string sQueue = "https://sqs.eu-west-1.amazonaws.com/{CODE}/{QUEUENAME}"; 

As with the writing a message I like to load the URL into a string.

            var config = new AmazonSQSConfig()
            {
                ServiceURL = "https://sqs.eu-west-1.amazonaws.com/" 
            };

We create the config with a ServiceURL that will point to the SQS in Dublin (eu-west-1)

            AmazonSQSClient _client = new AmazonSQSClient("ACCESSKEY", "ACCESSSECRET", config);

We create the AmazonSQSClient with the access key, Secret and the config.

            ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest();

            receiveMessageRequest.QueueUrl = sQueue;

We set the Queue URL to the URL from our Queue

             ReceiveMessageResponse receiveMessageResponse = _client.ReceiveMessage(receiveMessageRequest);

We create a receive Message response and call the Client and ask for what messages are on the queue.

            foreach (var oMessage in receiveMessageResponse.Messages)
            {
                Console.WriteLine(oMessage.Body);

We can now do a foreach on the ReceiveMessageResponse Messages this will allow us to read each message.

AWS SQS needs you to delete the message once it's read.

                // Delete the message from the queue
                DeleteMessageRequest deleteMessageRequest = new DeleteMessageRequest();

                deleteMessageRequest.QueueUrl = sQueue;
                deleteMessageRequest.ReceiptHandle = oMessage.ReceiptHandle;

Each message has a receipt handle, we set the Queue URL to our Queue we are reading from.

                DeleteMessageResponse response = _client.DeleteMessage(deleteMessageRequest);

That's it we've now created and read a message from the Amazon Simple Queue Service.

What is Amazon Simple Queue Service Amazon Simple Queue Service (Amazon SQS) offers reliable and scalable hosted queues for storing messages as they travel between computers. By using Amazon SQS, you can move data between distributed components of your applications that perform different tasks without losing messages or requiring each…

Read More

AWS Identity and Access Management IAM

AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources for your users. You use IAM to control who can use your AWS resources (authentication) and what resources they can use and in what ways (authorization).

Before you can use any AWS service with the API or form your code you need to setup a user in IAM. You can decide what a user has access to or what they can do. You can create users for spefic actions.

It's pretty straight forward to get a user setup. In the AWS console select services and type IAM

select_aws_IAM

This will show the welcome screen. Followed by a quick summary of users and groups, policies, roles.

For now we want users. On the left you will find "Users" This will display the users on your AWS account,

aws_users

Select "Create New Users" and enter a name for the user. You can create as many users as you want.

For what we are trying to do make sure you tick "Generate an access key for each user" this will allow the user to access the REST API for AWS.

AWS IAM User

Once that's done hit Create

AWS_Access_Keys

Once created you will be shown the Access Key and Secret Access Key. Take these down or download them you won't see the secret again.

That's it your ready to go.

One thing I really like is that once you create the user. In the list you can click and see the activity from that user.

But you can see the activity for the access key aswell. What service was used. You can also create more keys for different things.

AWS IAM User Activity

As you can see this KEY was last used for SQS in EU West yesterday I can turn it off or delete it or create another key for something else if I wanted to.

AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources for your users. You use IAM to control who can use your AWS resources (authentication) and what resources they can use and in what ways (authorization). Before you can use any…

Read More

Amazon Web Services Explained

Amazon Web Services

It can be a bit daunting when you first start looking at AWS and seeing the number of offerings available. I've explained each of the different services and also included some alternatives so you can see what it's like.

What is Amazon Web Services

Amazon Web Services (AWS) is a collection of remote computing services, that make up a cloud computing platform offered by Amazon.com.

Services
Service Description Alternative
CloudServices
EC2 Virtual Machines Azure VMs, Linode, DigitalOcean or Rackspace.
IAM Users, Policies
</td>
S3 File storage, store images and other assets
</td>
VPC Hybrid some on premise and some online.
Lambda Used for storing and then executing changes to your AWS setup or responding to events in S3 or DynamoDB.
Developer Services
API Gateway Works like a proxy on your network.
</td>
	<td> 3Scale</td>
RDS It's basically SQL server.
</td>
	<td> Heroku Postgres, Azure SQL</td>
Route53 It's DNS DNSimple, GoDaddy, Gandi
SES Transactional email service
</td>
	<td> SendGrid, Mandrill, Postmark, MailGun</td>
Cloudfront It's a content delivery network makes it easy to deliver files to your users. MaxCDN, Akamai
CloudSearch Pull in data on S3 or in RDS and then search it. Sphinx, Solr, ElasticSearch
DynamoDB No SQLDB
</td>
	<td> MongoLab, Azure Document DB</td>
Elasticache An open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and click stream analytics. Redis to Go, Memcachier
Elastic Transcoder Video Stuff
</td>
SQS Message Queue
</td>
	<td> RabbitMQ, Sidekiq, Azure Service bus</td>
WAF It's a Firewall
</td>
	<td> Sophos, Kapersky</td>
Cognito Login Service Give end users - (non AWS) - the ability to log in with Google, Facebook, etc.
</td>
	<td> OAuth.io</td>
Device Farm Test your app on a bunch of different IOS and Android devices simultaneously.
</td>
	<td> MobileTest, iOS emulator</td>
Mobile Analytics Track what people are doing inside of your app.
</td>
	<td> Flurry, Azure Application Insights, Newrelic</td>
SNS Send mobile notifications, emails and/or SMS messages
</td>
	<td> UrbanAirship, Twilio</td>
Ops and Code Deployment Services
CodeCommit Version control your code - hosted Git.
</td>
	<td> Github, BitBucket</td>
Code Deploy Deploy and build you code on the cloud. Heroku, Capistrano, Visual studio team services.
CodePipeline Automated tests on code. CircleCI, Travis
EC2 Container Service It's docker on the AWS
</td>
Enterprise / Corporate Services
AppStream Put a copy of a Windows application on a Windows machine that people get remote access to.
</td>
	<td> Citrix, RDP, Azure Remote app</td>
Direct Connect Pay your Telco + AWS to get a dedicated leased line from your data center or network to AWS.
Directory Service Tie together other apps that need a Microsoft Active Directory to control them. Azure Active Directory
WorkDocs Share Word Docs with your colleagues.
</td>
	<td> Dropbox, DataAnywhere</td>
WorkMail Hosted email service. Google Apps for Domains, Office365
Workspaces A windows desktop of your own just without the office.
</td>
Storage Gateway Make automating getting files into S3 from your corporate network easier.
</td>
Big Data Services
Data Pipeline Extract, Transform and Load data from elsewhere in AWS. Schedule when it happens and get alerts when they fail.
</td>
Elastic Map Reduce Iterate over massive text files of raw data that you're keeping in S3.
</td>
Glacier Make backups of your backups that you keep on S3. For long term archiving can be slow to get the data back. Also not cheap.
Kinesis Ingest lots of data very quickly that you then later use other AWS services to analyze.
</td>
	<td> Kafka</td>
RedShift Store a whole bunch of analytics data, do some processing, and dump it out.
</td>
Machine Learning Machine learning on AWS.
</td><td>Azure Machine Learning</td>
SWF Build a service of workers on top of EC2 to accomplish a set task.
</td>
	<td> IronWorker, Azure Web jobs</td>
Snowball Get a bunch of hard drives you can attach to your network to make getting large amounts (Terabytes of Data) into and out of AWS
</td>
AWS Management Services
	</div>
</th>
CloudFormation Set up a bunch of connected AWS services in one go.
</td>
CloudTrail Log who is doing what in your AWS stack (API calls).
</td>
CloudWatch Get alerts about AWS services messing up or disconnecting.
</td>
	<td> PagerDuty, Statuspage</td>
Config Keep from going insane if you have a large AWS setup and changes are happening that you want to track.
</td>
OpsWorks Handle running your application with things like auto-scaling.
</td>
Pricing

Amazon have a pricing calculator that will allow you to estimate and work out how much it will cost to use the services.

Amazon Web Services It can be a bit daunting when you first start looking at AWS and seeing the number of offerings available. I've explained each of the different services and also included some alternatives so you can see what it's like. What is Amazon Web Services Amazon Web Services…

Read More