Getting Started with Amazon Simple Queue Service (AWS SQS)

What is Amazon Simple Queue Service

Amazon Simple Queue Service (Amazon SQS) offers reliable and scalable hosted queues for storing messages as they travel between computers.

By using Amazon SQS, you can move data between distributed components of your applications that perform different tasks without losing messages or requiring each component to be always available.

Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing.

Like other Message bus systems using Amazon SQS, you can decouple the components of an application so they run independently, with Amazon SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages programmatically using the Amazon SQS API. Messages larger than 256 KB can be managed using the Amazon SQS Extended Client Library for Java, which uses Amazon S3 for storing larger payloads.

Pricing

As of this post, these applied to all regions except Asia Pacific (Tokyo) and GovCloud (US)

  • First 1 million Amazon SQS Requests per month are free
  • $0.50 per 1 million Amazon SQS Requests per month thereafter ($0.00000050 per SQS Request)
  • All SQS API calls count as a Request
  • A single Request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
  • Each 64KB ‘chunk’ of payload is billed as 1 Request. For example, a single API call with a 256KB payload will be billed as four Requests.

Asia Pacific (Tokyo) region only

  • First 1 million Amazon SQS Requests per month are free
  • $0.476 per 1 million Amazon SQS Requests per month thereafter ($0.000000476 per SQS Request)
  • All SQS API calls count as a Request
  • A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
  • Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.

GovCloud (US) region only

  • No free tier
  • $0.60 per 1 million Amazon SQS Requests per month thereafter ($0.00000060 per SQS Request)
  • All SQS API calls count as a Request
  • A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
  • Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.
Get Started

First thing we need to do is make sure we have a user set up in the Identity and Access Management. We need a user that has access keys see here I cover creating a new user with access keys.

Now in Services search for SQS

AWS_Create_SQS

Select "Create new Queue"

AWS_SQS_CreateQueue

I had already changed my region to EU Ireland( see the top right of the AWS Console you can change the region)

I just left things at the defaults for the moment we'll play with them later. Go ahead and create a queue.

Once your queue is created you will see some information in the panel at the bottom of the screen, mainly the Name followed by a URL, for me it's

https://sqs.eu-west-1.amazonaws.com/54XXXXXXX57/serversncodeq

AWS_SQS_Details

We need that URL. The next part and this caught me first time around is the Permissions, There is no default set so we need to do that, you can click the permission tab or Right click the queue and select "Add a Permissions"

AWS Permission

For the moment I select Everybody and All actions but you can see how we can use user accounts to control access to the queues, who can write and read.

That's it we have a queue and user account so we can access the queue

Let's write some code

The below sample is in C# for the code see Github

AWS SQS Read Console Sample Application

AWS SQS Write Console Sample Application

Write a message to the queue.

AWS SQS Write Console Sample Application

In a C# console app, in Package manager run

PM> Install-Package AWSSDK.SQS

This will install the SDK for SQS so we can get going. Here is the code for that will write a message to the queue.

        string sQueue = "https://sqs.eu-west-1.amazonaws.com/{CODE}/{QUEUENAME}";

        try
        {
            var config = new AmazonSQSConfig()
            {
                ServiceURL = "https://sqs.eu-west-1.amazonaws.com/"
            };

            var _messageRequest = new SendMessageRequest();

            _messageRequest.QueueUrl = sQueue;

            _messageRequest.MessageBody = "This is a test message";

            AmazonSQSClient _client = new AmazonSQSClient("ACCESSKEY", "ACCESSSECRET", config);

            SendMessageResponse sendMessageResponse = _client.SendMessage(_messageRequest);

        }
        catch (Exception ex)
        {
            Console.WriteLine("Error: " + ex.ToString());
        }

So lets take a wander through what we have just done.

First up we have the Queue URL we looked at earlier, I store it off in a string.

We then set the config.

            var config = new AmazonSQSConfig()
            {
                ServiceURL = "https://sqs.eu-west-1.amazonaws.com/" 
            };

The Config Service URL should be the service, region and then amazonaws.com This will tell the SQS Client where we want to connect.

Next we create our MessageRequest

            var _messageRequest = new SendMessageRequest();

We set the Message Request Queue URL to the string we set at the top ("https://sqs.eu-west-1.amazonaws.com/{CODE}/{QUEUENAME}")

            _messageRequest.QueueUrl = sQueue;

            _messageRequest.MessageBody = "This is a test message";

Now we want to set the Amazon SQS client. We pass our Access key and Access Secret we setup in the IAM and then pass the config which will set everything to the region we want.

            AmazonSQSClient _client = new AmazonSQSClient("ACCESSKEY", "ACCESSSECRET", config);

We now send the message.

            SendMessageResponse sendMessageResponse = _client.SendMessage(_messageRequest);
Read a message to the queue.

Sample Code here

In a C# console app, in Package manager run

PM> Install-Package AWSSDK.SQS

To read the message we just created our code will look like this

        string sQueue = "https://sqs.eu-west-1.amazonaws.com/{CODE}/{QUEUENAME}"; 

        try
        {
            var config = new AmazonSQSConfig()
            {
                ServiceURL = "https://sqs.eu-west-1.amazonaws.com/" 
            };

            AmazonSQSClient _client = new AmazonSQSClient("ACCESSKEY", "ACCESSSECRET", config);

            ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest();

            receiveMessageRequest.QueueUrl = sQueue;

            ReceiveMessageResponse receiveMessageResponse = _client.ReceiveMessage(receiveMessageRequest);

            foreach (var oMessage in receiveMessageResponse.Messages)
            {
                Console.WriteLine(oMessage.Body);

                // Delete the message from the queue
                DeleteMessageRequest deleteMessageRequest = new DeleteMessageRequest();

                deleteMessageRequest.QueueUrl = sQueue;
                deleteMessageRequest.ReceiptHandle = oMessage.ReceiptHandle;

                DeleteMessageResponse response = _client.DeleteMessage(deleteMessageRequest);
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine("Error: " + ex.ToString());
        }

Now lets take a wander through what we just did.

        string sQueue = "https://sqs.eu-west-1.amazonaws.com/{CODE}/{QUEUENAME}"; 

As with the writing a message I like to load the URL into a string.

            var config = new AmazonSQSConfig()
            {
                ServiceURL = "https://sqs.eu-west-1.amazonaws.com/" 
            };

We create the config with a ServiceURL that will point to the SQS in Dublin (eu-west-1)

            AmazonSQSClient _client = new AmazonSQSClient("ACCESSKEY", "ACCESSSECRET", config);

We create the AmazonSQSClient with the access key, Secret and the config.

            ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest();

            receiveMessageRequest.QueueUrl = sQueue;

We set the Queue URL to the URL from our Queue

             ReceiveMessageResponse receiveMessageResponse = _client.ReceiveMessage(receiveMessageRequest);

We create a receive Message response and call the Client and ask for what messages are on the queue.

            foreach (var oMessage in receiveMessageResponse.Messages)
            {
                Console.WriteLine(oMessage.Body);

We can now do a foreach on the ReceiveMessageResponse Messages this will allow us to read each message.

AWS SQS needs you to delete the message once it's read.

                // Delete the message from the queue
                DeleteMessageRequest deleteMessageRequest = new DeleteMessageRequest();

                deleteMessageRequest.QueueUrl = sQueue;
                deleteMessageRequest.ReceiptHandle = oMessage.ReceiptHandle;

Each message has a receipt handle, we set the Queue URL to our Queue we are reading from.

                DeleteMessageResponse response = _client.DeleteMessage(deleteMessageRequest);

That's it we've now created and read a message from the Amazon Simple Queue Service.

What is Amazon Simple Queue Service Amazon Simple Queue Service (Amazon SQS) offers reliable and scalable hosted queues for storing messages as they travel between computers. By using Amazon SQS, you can move data between distributed components of your applications that perform different tasks without losing messages or requiring each…

Read More

AWS Identity and Access Management IAM

AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources for your users. You use IAM to control who can use your AWS resources (authentication) and what resources they can use and in what ways (authorization).

Before you can use any AWS service with the API or form your code you need to setup a user in IAM. You can decide what a user has access to or what they can do. You can create users for spefic actions.

It's pretty straight forward to get a user setup. In the AWS console select services and type IAM

select_aws_IAM

This will show the welcome screen. Followed by a quick summary of users and groups, policies, roles.

For now we want users. On the left you will find "Users" This will display the users on your AWS account,

aws_users

Select "Create New Users" and enter a name for the user. You can create as many users as you want.

For what we are trying to do make sure you tick "Generate an access key for each user" this will allow the user to access the REST API for AWS.

AWS IAM User

Once that's done hit Create

AWS_Access_Keys

Once created you will be shown the Access Key and Secret Access Key. Take these down or download them you won't see the secret again.

That's it your ready to go.

One thing I really like is that once you create the user. In the list you can click and see the activity from that user.

But you can see the activity for the access key aswell. What service was used. You can also create more keys for different things.

AWS IAM User Activity

As you can see this KEY was last used for SQS in EU West yesterday I can turn it off or delete it or create another key for something else if I wanted to.

AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources for your users. You use IAM to control who can use your AWS resources (authentication) and what resources they can use and in what ways (authorization). Before you can use any…

Read More

Getting Started with Azure Service Bus

This post assumes you have already setup an Azure Service Bus if you haven't see here

We're going to create two Console applications in C# to read and write messages to the Azure Service bus.

We will also need shared access keys setup on Azure service bus, if you haven't done that or are unsure where to get them check here

First off we need to add the nuget package to our projects.

PM> Install-Package WindowsAzure.ServiceBus

On nuget: WindowsAzure.ServiceBus

Writing to Azure Service bus

Code for this is here

Create a C# Console app in Visual Studio and add the nuget package

Now we need to add the shared access key. So on the Azure Service bus in the portal open the Shared Access keys and copy the primary Endpoint will look something like this

Endpoint=sb://serversncodeX.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXX

In the app.config

Under appSettings for the Microsoft.ServiceBus.ConnectionString add the end point.

<appSettings>
    <add key="Microsoft.ServiceBus.ConnectionString" value="Endpoint=sb://serversncodeX.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXX"/>
</appSettings>

So back to program.cs

        string connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");

We added a connection string and told it to go to our app.config and get the connection string to our Service Bus Endpoint.

        QueueClient Client = QueueClient.CreateFromConnectionString(connectionString, "serversncodedemoqueue");

We have created a QueueClient with the connectionstring and connected to the queue we created called "serversncodedemoqueue"

If you followed along with the last post we created a queue(note you can do that from code but for this demo I have skipped it) as a side note I think when you can it's better to create the queue yourself in the interface rather then code. It can be costly to check if a queue exists and then create it in code all the time. Also with some of the advanced things you can do with Keys and queues which we'll go into in time it's better to do all the setup up front.

Ok so we have our client created and ready to go lets send a message. For this we're going to play make believe. We have a new user signed up to our great App and we want to send a message to that user. They have just signed up, our app has written it to the database and got the user ID and User Code and wants to tell our email application that it can send an email to this user.

        // Create message, passing a string message for the body.
        BrokeredMessage message = new BrokeredMessage("");

        // Set some addtional custom app-specific properties.
        message.Properties["UserCode"] = "dascdcasas";
        message.Properties["UserId"] = "4550"; 

        // Send message to the queue.
        Client.Send(message);

BrokeredMessage is the class for the message we will send. We create two custom properties for our App and pass the values in from our Database.

Then we send the message.

And that's kinda that. We've now put a message onto the service bus. If you look in the portal on the queue it will show you there is an active message in the queue so lets go ahead and read it.

Reading Messages from Azure Service bus

Code sample is here

The first part of this application is the same as our app 1. We create the connection string and then add it in.

So to recap

  • Create C# Console app
  • Add Nuget package PM> Install-Package WindowsAzure.ServiceBus
  • Add connection string to app.config.

Now lets create the program.cs to read the service bus.

        string connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");

        QueueClient Client = QueueClient.CreateFromConnectionString(connectionString, "serversncodedemoqueue");

        OnMessageOptions options = new OnMessageOptions();
        options.AutoComplete = false;
        options.AutoRenewTimeout = TimeSpan.FromMinutes(1);

The first two parts we've seen before, create our connection string and then create our client with the connection string and call our queue.

The next parts are new, these are the options we're going to allow our client to have as we read the service bus.

  • AutoComplete - Should the message be set to complete automaticlly once we've processed it. Actually this is once we've read it. It's a real fire and forget kind of thing. Your assuming you aren't going to crash.
  • AutoRenewTimeout - the maximum duration within which the lock will be renewed automatically. There are message locks in Service Bus so a message won't be read by someone else but at some stage you need to wonder if the message has crashed out and something else should read it.

Now the bit your waiting for.

        Client.OnMessage((message) =>
        {
            try
            {
                string sMessage = " UserCode: " + message.Properties["UserCode"];

                Console.WriteLine("Found new User - " + sMessage);


                // Remove message from queue.
                message.Complete();


            }
            catch (Exception ex)
            {
                // Indicates a problem, unlock message in queue.
                message.Abandon();
            }
        }, options);

That's it, no really that's it,

Client.OnMessage is a method in the QueueClient, on a message we then pass the message as a parameter into our method and at the end of the call back we set the options to use.

Once in there we then get the message properties we setup in our other app. We passed "UserCode" as a parameter. We read it and it writes the message to our console. It then marks it as Complete and moves on. There is no schema to the properties on the service bus, you can go as mad as you want and have different properties on the same queue just keep in mind only 1 process can read a message at a time.

When you run this code it will remain open and running as it is reading the service bus for any messages. Each time you run the other app and write a message to the service bus it will appear on the screen.

Also note if you run two reader apps then only 1 reads the message, this is the locks in action only 1 process can handle each message.

This post assumes you have already setup an Azure Service Bus if you haven't see here We're going to create two Console applications in C# to read and write messages to the Azure Service bus. We will also need shared access keys setup on Azure service bus, if you haven't…

Read More

Setup Azure Service bus

What is Azure Service Bus messaging?

Azure Service Bus messaging is an asynchronous messaging cloud platform that enables you to send data between decoupled systems. Azure Service bus is developed and provided by Microsoft.

Azure Service bus is hosted on Azure. There are different types of messaging built into Azure Service bus but for now and the purpose of this post I'm focused solely on Queues and the basic messaging we need to get started.

Pricing

There are 3 pricing levels for Azure Service bus. Each level comes with some added features.

Basic - €0.0422 per million operations

Standard - €8.433/mo for the First 12.5M Operations

  • Next 88M Ops (13M - 100M Ops) / Month €0.6746 per million operations
  • Next 2,400M Ops (100M - 2,500M Ops) / Month €0.4217 per million operations
  • Over 2,501M Ops / Month €0.1687 per million operations

Premium

Daily €18.77 fixed rate per Message Unit

As pointed out above we're focusing on Queues at the moment so I'm ignoring brokered connections and topics for the moment.

All prices for North Europe in EURO for more Azure Service Bus Pricing

Get Started

To get started we need to go to Azure Portal

On the left click new and search for "Service Bus" this will open the market place and let you select Service bus
Create Azure Service Bus

Select Service Bus, and then select Create.

Next we need to give it a name and resource group and stuff.

Define Service Bus

Select a name, Subscription and then Set the resource group to use an existing or to create a new one.

Also select the pricing tier you want to use.

And hit Create.

That's it your service bus is now being setup for use.

Service Bus up and running

Setup the Queue and access keys

So now we have our service bus created we need to create the queue. Under General select Queues

Create queue

Select Add Queue, and give it a name.

Create queue

We need to get the access keys so we can use the queue. Above queue select "Shared Access Policies" after a second a default "RootManageSharedAccessKey" will appear,

Open the "RootManageSharedAccessKey"

Azure Shared access keys

These keys you will need next so we can use this service bus queue.

What is Azure Service Bus messaging? Azure Service Bus messaging is an asynchronous messaging cloud platform that enables you to send data between decoupled systems. Azure Service bus is developed and provided by Microsoft. Azure Service bus is hosted on Azure. There are different types of messaging built into Azure…

Read More

Application Services Message bus

As our applications and services grow and we start to break more into Microservices a key part of building applications is Messaging. Being able to pass messages between the applications quickly becomes critical to our service.

Over the next few blog posts I'll look at getting started with different messaging systems and how to use them. It's not a compare it's a getting started in each of them.

Why do we need messaging

Messaging allows communication between applications. For example we create a user in our database and then create a message onto our message bus. Our email service is listening to the message bus. Gets the new message and sends the welcome email. We've now wired two applications to work together over a message bus.

Microservices

Microservices is a particular way of designing software applications as suites of independently deployable services. In a microservices world communication between applications has to be robust and scalable. Message buses provide a flexible solution to this. We can create services and deploy them in different places. You can even have services in different languages, once they can read the same message bus they can do what they need to do.

What next

Over the next few posts I'll introduce different Message bus services and providers.

As our applications and services grow and we start to break more into Microservices a key part of building applications is Messaging. Being able to pass messages between the applications quickly becomes critical to our service. Over the next few blog posts I'll look at getting started with different messaging…

Read More