RabbitMQ and Google Pub/Sub are both powerful and reliable message queue implementations, and if you need to pick one of them for your Google Cloud Platform (GCP) project, the choice may not be simple. We recently started a new project that required a message broker integrated into the solution, and although there are several choices on the market, we quickly narrowed down our options to these two products. This blog post is the first part of the summary of our analysis.

The New Project

The project in question — an apartment rental platform for Swiss clients — had to meet both scalability and geographically restricted hosting requirements. When we began, GCP was the most viable option for meeting these requirements. However, the technology was new to us, which meant we needed to reconsider some of our technology choices.

In this new software, some of our services were going to communicate via message queues, which meant we also needed to choose a message queue implementation that could be deployed to GCP. Additionally, it was important that we decided on something reliable.

In a non-GCP project, RabbitMQ would be the preferred choice because it’s well-known among our team, it has a convenient administrative interface, it has extensively configurable message distribution and routing, and most importantly, its documentation is well-maintained and thorough.

However, RabbitMQ is not a native building block of the GCP infrastructure — we can still install it fairly easily from the GCP Marketplace as a Click to Deploy container, but as a custom install, there’s no service-level agreement (SLA) for it.

Meanwhile, Pub/Sub is GCP’s native implementation for message queues. We hadn’t used it before, but after a quick look at the official developer documentation, our first impressions were positive. Setting up queues requires just a few clicks, and there’s documentation with examples of how to use the queues with some widely used programming languages.

Let’s explore the benefits of each solution and see how easy it is to build up a queue topology with RabbitMQ and Pub/Sub.

Basic Concepts

The general idea of a message broker is to relay messages from publishers to consumers by routing them to different delivery queues. So, we’ll first look at the basic building blocks of RabbitMQ and Pub/Sub to see how much they differ from one another.

RabbitMQ

RabbitMQ is an open source message broker that implements the Advanced Message Queueing Protocol (AMQP) standard, and with its various plugins, it also supports other protocols, including Streaming Text Oriented Messaging Protocol (STOMP) and MQ Telemetry Transport. In this post, we’ll focus on RabbitMQ’s AMQP features.

In RabbitMQ, the topology is built up from exchanges and queues that are connected by bindings.

Exchanges

Exchanges make it possible to define the routing of messages to queues connected to the exchange. Message publishers have to send their messages to exchanges, and from there, they’re routed to queues.

Exchanges can be created on demand by the publisher or the consumer, and they must be declared with a name and a type. The table below shows the four possible types of exchanges, along with their behaviors:

In a topic exchange, wildcards can be used for defining bindings based on routing keys.

Let’s suppose our broker propagates messages with routing keys following this structure: transaction.<product>.<city>.<productcode> — for example, transaction.stock.london.ABC, transaction.bond.frankfurt.DEF, and transaction.stock.newyork.JKL. We can define the following kinds of binding rules for such routing keys:

With header exchanges, you can specify matching rules for header attribute values.

Let’s suppose all our messages have a format and a type header attribute and they look like {format = jpg, type = profilephoto}, {format = pdf, type = workpermit}, or {format = jpg, type = workpermit}. The header exchange may define conjunctive and disjunctive binding rules by listing headers with their expected values, such as:

By default, every RabbitMQ broker installation comes with a set of predeclared exchanges for each exchange type:

The default exchange is the direct exchange with no name (empty string). It has a useful feature that comes in handy for simple applications: Every queue that’s created is bound to this direct exchange with a routing key that’s the same as the queue name.

In addition to types, exchanges have some other important features, outlined below:

Queues

Queues are named resources that store messages consumed by applications. Some important attributes (or arguments, as they’re referred to in RabbitMQ) are outlined below:

Consumers get messages from the queues using a push or a pull mechanism. Push-style delivery happens via consumer channels that should be registered by the consumer application with a basic.consume request. When the broker pushes a message to the consumer (basic.deliver), the consumer has to acknowledge the receipt. This acknowledgement can be positive (basic.ack), negative (basic.nack), or a rejection (basic.reject). The broker doesn’t remove the message from the queue until it gets some kind of acknowledgement. Depending on the acknowledgement type, the message can be deleted, requeued, or rerouted.

The consumer can be registered in auto-acknowledgement mode, but it’s quite risky, because in this case, the queue automatically removes the message from the queue once it’s been pushed to the consumer, and it doesn’t wait for any processing acknowledgement (meaning that if the consumer dies, the message gets lost).

The other possibility for the consumer is to poll the queues directly by using basic.get requests. This is expensive compared to push-style delivery, because the consumer has to both repeatedly check if there are new messages in the queue and wait for a response.

Dead Letter Exchanges

We can use dead letter exchanges to catch messages that weren’t processed by the consumer for some reason. Messages can be republished to a dead letter exchange if:

• The message was negatively acknowledged (basic.nack or basic.reject).
• The message expired due to its TTL.
• The message was dropped because its queue is over the length limit.

RabbitMQ Messages

A message consists of the message body and additional attributes (some of them are known as headers). Some predefined attributes include:

• Content type
• Encoding
• Routing key
• Delivery mode (persistent or not)
• Priority (effective only on queues that have priorities enabled)
• Publishing timestamp
• Expiration period
• Publisher application ID

Sending a Message with RabbitMQ

Messages have to be published to exchanges, and the exchanges route them to the queues. When sending a message, the publisher has to specify the routing key and the exchange name. If the exchange name is specified as an empty string, the message is directly routed to the queue that has the same name as its routing key (see the explanation of default exchanges in the Exchanges section above).

Pub/Sub

Pub/Sub is Google’s message broker implementation. It’s a native building block of the Google Cloud infrastructure and can be used exclusively in Google Cloud. We only have to enable the Pub/Sub Services in our GCP project, and then we can start integrating our application with this service.

In Pub/Sub, we can build our queue topology from topics and subscriptions.

Topics

Topics are named destinations where publishers can send their messages. They play a role similar to exchanges in RabbitMQ. Topics don’t store messages, but instead forward them to subscriptions connected to them. As a result, a topic must have at least one subscription connected to it — otherwise, the messages sent to this topic will be lost.

Pub/Sub doesn’t have any default topics, which means we need to set up at least one named topic and a subscription before using Pub/Sub from any application.

There’s only one type of topic, unlike in RabbitMQ, where there are four different exchanges. Topics simply forward all messages to all subscriptions connected to them. The message flow can be controlled by the filtering rules defined in the subscriptions.

Subscriptions

Subscriptions store messages that are consumed by applications, and they play the same role as queues in RabbitMQ.

Subscriptions can deliver messages to subscriber applications pull or push style. Pull subscribers can use the StreamingPull API, which means they open long-running message listeners to asynchronously receive multiple messages and acknowledge them one at a time. This mechanism is fairly similar to RabbitMQ’s channel mechanism for push-style consumers. If blocking delivery for each message is important, pull subscriptions can also be synchronous.

Push subscribers have to provide a callback endpoint where Pub/Sub delivers the messages. This endpoint must be a publicly accessible HTTPS address with a valid SSL certificate signed by a certificate authority. To make the message delivery more secure, push subscriptions can be configured to add authorization headers to the messages that the subscriber’s callback endpoint can verify.

Messages are delivered at least once, redeliveries may occur, and applications must be ready to handle them.

Subscriptions can filter messages based on their attributes. Filters can be defined for any message attribute. Unlike in RabbitMQ binding expressions, we can use negative conditions for our filters too. Behind the scenes, the subscription with the filtering rule silently acknowledges and drops the messages that don’t match the filter criteria (and this traffic isn’t billed as usage cost).

In the examples below, you can see all the possible operators that can be used for attribute-based filtering:

Dead Letter Topics

Subscriptions can be equipped with a dead letter topic, where undeliverable messages get forwarded after a configurable number of delivery attempts.

Any subscription configured with a dead letter topic adds an additional delivery-attempt field to the messages, and the subscriber can use this to track how many delivery attempts were required to deliver the message.

Pub/Sub Messages

Pub/Sub messages consist of the following elements:

By default, these additional attributes are added by Pub/Sub topics to the messages:

• A message ID unique to the topic
• A UTC timestamp of when Pub/Sub first saw the message

There’s a general size limit imposed by Pub/Sub on message publish requests (and consequently to messages), which is 10 MB.

Sending a Message with Pub/Sub

Messages have to be sent to topics, and from there, they’re routed to subscriptions.

When sending a message, the publisher has to specify the topic, but there’s no other mandatory addressing parameter.

Summary

In this post, we looked at all the basics of RabbitMQ and GCP Pub/Sub. The two message broker platforms offer the same basic functionality, and they’re conceptually very similar in terms of how they propagate messages from publishers to consumers. Despite using different terminology, both allow flexible topology setup and various routing/filtering options. However, Pub/Sub offers somewhat richer and more intuitive filtering features, making advanced use cases easier.

In the second part of this series, we’ll share practical examples of how one should build up typical queue topologies with RabbitMQ and Pub/Sub.