RabbitMQ and Google Pub/Sub are both powerful and reliable message queue implementations, and if you need to pick one of them for your Google Cloud Platform (GCP) project, the choice may not be simple. We recently started a new project that required a message broker integrated into the solution, and although there are several choices on the market, we quickly narrowed down our options to these two products. This blog post is the first part of the summary of our analysis.

The New Project

The project in question — an apartment rental platform for Swiss clients — had to meet both scalability and geographically restricted hosting requirements. When we began, GCP was the most viable option for meeting these requirements. However, the technology was new to us, which meant we needed to reconsider some of our technology choices.

In this new software, some of our services were going to communicate via message queues, which meant we also needed to choose a message queue implementation that could be deployed to GCP. Additionally, it was important that we decided on something reliable.

In a non-GCP project, RabbitMQ would be the preferred choice because it’s well-known among our team, it has a convenient administrative interface, it has extensively configurable message distribution and routing, and most importantly, its documentation is well-maintained and thorough.

However, RabbitMQ is not a native building block of the GCP infrastructure — we can still install it fairly easily from the GCP Marketplace as a Click to Deploy container, but as a custom install, there’s no service-level agreement (SLA) for it.

Meanwhile, Pub/Sub is GCP’s native implementation for message queues. We hadn’t used it before, but after a quick look at the official developer documentation, our first impressions were positive. Setting up queues requires just a few clicks, and there’s documentation with examples of how to use the queues with some widely used programming languages.

Let’s explore the benefits of each solution and see how easy it is to build up a queue topology with RabbitMQ and Pub/Sub.

Basic Concepts

The general idea of a message broker is to relay messages from publishers to consumers by routing them to different delivery queues. So, we’ll first look at the basic building blocks of RabbitMQ and Pub/Sub to see how much they differ from one another.

publishers routing logic queues consumers
The idea of a message broker

RabbitMQ

RabbitMQ is an open source message broker that implements the Advanced Message Queueing Protocol (AMQP) standard, and with its various plugins, it also supports other protocols, including Streaming Text Oriented Messaging Protocol (STOMP) and MQ Telemetry Transport. In this post, we’ll focus on RabbitMQ’s AMQP features.

In RabbitMQ, the topology is built up from exchanges and queues that are connected by bindings.

publishers exchanges bindings queues consumers
The RabbitMQ broker building blocks

Exchanges

Exchanges make it possible to define the routing of messages to queues connected to the exchange. Message publishers have to send their messages to exchanges, and from there, they’re routed to queues.

Exchanges can be created on demand by the publisher or the consumer, and they must be declared with a name and a type. The table below shows the four possible types of exchanges, along with their behaviors:

Exchange TypeBehavior
DirectThe message goes to the queues with a binding key that matches the routing key of the message exactly.
TopicThis routes messages to queues based on a wildcard match between the routing key and the routing pattern.
HeadersThese are similar to Topic exchanges, but they use headers and optional values for routing.
FanoutThis broadcasts all the messages it receives to all the queues it knows. Routing keys are ignored.

In a topic exchange, wildcards can be used for defining bindings based on routing keys.

Let’s suppose our broker propagates messages with routing keys following this structure: transaction.<product>.<city>.<productcode> — for example, transaction.stock.london.ABC, transaction.bond.frankfurt.DEF, and transaction.stock.newyork.JKL. We can define the following kinds of binding rules for such routing keys:

Binding RuleMeaning
transaction.stock.london.#Matches any routing key starting with transaction.stock.london.
transaction.*.london.#Matches routing keys starting with transaction, then with any string as the second element, followed by london

With header exchanges, you can specify matching rules for header attribute values.

Let’s suppose all our messages have a format and a type header attribute and they look like {format = jpg, type = profilephoto}, {format = pdf, type = workpermit}, or {format = jpg, type = workpermit}. The header exchange may define conjunctive and disjunctive binding rules by listing headers with their expected values, such as:

Binding RuleMeaning
{format = jpg, type = workpermit, x-match = all}Matches messages with the arguments format = jpg AND type = workpermit
{format = jpg, type = profilephoto, x-match = any}Matches messages with the argument format = jpg OR type = profilephoto

By default, every RabbitMQ broker installation comes with a set of predeclared exchanges for each exchange type:

Exchange TypeDefault Name
Directamq.direct or empty string
Topicamq.topic
Headersamq.headers (and the AMQP standard amq.match)
Fanoutamq.fanout

The default exchange is the direct exchange with no name (empty string). It has a useful feature that comes in handy for simple applications: Every queue that’s created is bound to this direct exchange with a routing key that’s the same as the queue name.

In addition to types, exchanges have some other important features, outlined below:

FeatureDescription
DurabilityDurable exchanges survive broker restarts, while transient ones don’t.
Auto-deleteAn exchange can be deleted automatically as soon as the last queue is unbound from it.
InternalThis specifies whether or not the exchange is reserved for internal use by the broker only.
ArgumentsThis is a user-definable attribute list where the values can be of type string, list, number, or Boolean. A reserved argument name is alternate-exchange.
Alternate exchangeIf the routing of messages isn’t possible via any of the bindings, messages will be sent to the alternate exchange if specified.

Queues

Queues are named resources that store messages consumed by applications. Some important attributes (or arguments, as they’re referred to in RabbitMQ) are outlined below:

AttributeMeaning
x-max-priorityIf this is set, the queue can handle message priorities.
x-message-ttlAfter the TTL period expires, the message will be discarded from the queue.
x-expiresIf the queue remains unused for the specified period, it’ll automatically be deleted.
x-max-lengthThis is the maximum number of messages in the queue.
x-max-length-bytesThis is the total body size of messages a queue can contain.
x-overflowThis specifies what should happen with messages if the queue is full. It can start dropping the oldest or the newest messages or republish the newest ones to the dead letter queue.
x-dead-letter-exchangeThis is an optional dead letter exchange.
x-dead-letter-routing-keyThe routing key of dead-lettered messages can be optionally replaced.
x-queue-modeIf this is set to lazy, all the messages are written to disk. This causes some performance degradation, but it eliminates unexpected glitches in performance.

Consumers get messages from the queues using a push or a pull mechanism. Push-style delivery happens via consumer channels that should be registered by the consumer application with a basic.consume request. When the broker pushes a message to the consumer (basic.deliver), the consumer has to acknowledge the receipt. This acknowledgement can be positive (basic.ack), negative (basic.nack), or a rejection (basic.reject). The broker doesn’t remove the message from the queue until it gets some kind of acknowledgement. Depending on the acknowledgement type, the message can be deleted, requeued, or rerouted.

The consumer can be registered in auto-acknowledgement mode, but it’s quite risky, because in this case, the queue automatically removes the message from the queue once it’s been pushed to the consumer, and it doesn’t wait for any processing acknowledgement (meaning that if the consumer dies, the message gets lost).

The other possibility for the consumer is to poll the queues directly by using basic.get requests. This is expensive compared to push-style delivery, because the consumer has to both repeatedly check if there are new messages in the queue and wait for a response.

Dead Letter Exchanges

We can use dead letter exchanges to catch messages that weren’t processed by the consumer for some reason. Messages can be republished to a dead letter exchange if:

  • The message was negatively acknowledged (basic.nack or basic.reject).
  • The message expired due to its TTL.
  • The message was dropped because its queue is over the length limit.

RabbitMQ Messages

A message consists of the message body and additional attributes (some of them are known as headers). Some predefined attributes include:

  • Content type
  • Encoding
  • Routing key
  • Delivery mode (persistent or not)
  • Priority (effective only on queues that have priorities enabled)
  • Publishing timestamp
  • Expiration period
  • Publisher application ID

Sending a Message with RabbitMQ

Messages have to be published to exchanges, and the exchanges route them to the queues. When sending a message, the publisher has to specify the routing key and the exchange name. If the exchange name is specified as an empty string, the message is directly routed to the queue that has the same name as its routing key (see the explanation of default exchanges in the Exchanges section above ).

Pub/Sub

Pub/Sub is Google’s message broker implementation. It’s a native building block of the Google Cloud infrastructure and can be used exclusively in Google Cloud. We only have to enable the Pub/Sub Services in our GCP project, and then we can start integrating our application with this service.

In Pub/Sub, we can build our queue topology from topics and subscriptions.

publishers topics subscriptions consumers
The Pub/Sub broker building blocks

Topics

Topics are named destinations where publishers can send their messages. They play a role similar to exchanges in RabbitMQ. Topics don’t store messages, but instead forward them to subscriptions connected to them. As a result, a topic must have at least one subscription connected to it — otherwise, the messages sent to this topic will be lost.

Pub/Sub doesn’t have any default topics, which means we need to set up at least one named topic and a subscription before using Pub/Sub from any application.

There’s only one type of topic, unlike in RabbitMQ, where there are four different exchanges. Topics simply forward all messages to all subscriptions connected to them. The message flow can be controlled by the filtering rules defined in the subscriptions.

Subscriptions

Subscriptions store messages that are consumed by applications, and they play the same role as queues in RabbitMQ.

SettingMeaning / Possible Values
Delivery typeMessages can be delivered in push or pull style. The default is pull delivery.
ExpirationThe subscription expires after x days if there’s no activity (the default is 31 days).
Acknowledgement deadlineMessages have to be acknowledged within this deadline. After the expiration of this deadline, Pub/Sub will attempt to redeliver the message.
FilterSubscriptions can filter messages based on their attributes. You can see in the table below how these filters are defined.
Message retention durationUndelivered messages are retained for this configurable period before they get dropped automatically. The default and maximum duration is 7 days.
Retain acknowledged messagesAcknowledged messages may be retained in the subscription. This is disabled by default.
Message orderingMessages are delivered strictly in the order they arrived. This is disabled by default.
Dead letteringUndelivered messages may be forwarded to dead letter topics. This is disabled by default.
Retry policyWhen a retry policy is configured, Pub/Sub tries to redeliver unacknowledged messages with a configurable backoff time between redelivery attempts.

Subscriptions can deliver messages to subscriber applications pull or push style. Pull subscribers can use the StreamingPull API, which means they open long-running message listeners to asynchronously receive multiple messages and acknowledge them one at a time. This mechanism is fairly similar to RabbitMQ’s channel mechanism for push-style consumers. If blocking delivery for each message is important, pull subscriptions can also be synchronous.

Push subscribers have to provide a callback endpoint where Pub/Sub delivers the messages. This endpoint must be a publicly accessible HTTPS address with a valid SSL certificate signed by a certificate authority. To make the message delivery more secure, push subscriptions can be configured to add authorization headers to the messages that the subscriber’s callback endpoint can verify.

Messages are delivered at least once, redeliveries may occur, and applications must be ready to handle them.

Subscriptions can filter messages based on their attributes. Filters can be defined for any message attribute. Unlike in RabbitMQ binding expressions, we can use negative conditions for our filters too. Behind the scenes, the subscription with the filtering rule silently acknowledges and drops the messages that don’t match the filter criteria (and this traffic isn’t billed as usage cost).

In the examples below, you can see all the possible operators that can be used for attribute-based filtering:

Filtering RuleMeaning
attributes:greenMessages with the green attribute
NOT attributes:greenMessages without the green attribute
attributes.color = "green"Messages with the color attribute and the value green
attributes.color != "green"Messages with the color attribute and without the value green
attributes:green AND (attributes.size = "xtrasmall" OR attributes.size = "small")Messages with the green attribute AND with either of the two size attribute values
hasPrefix(attibutes.domain, "eu")Messages with the domain attribute and starting with eu
NOT hasPrefix(attibutes.domain, "eu")Messages with the domain attribute and NOT starting with eu

Dead Letter Topics

Subscriptions can be equipped with a dead letter topic, where undeliverable messages get forwarded after a configurable number of delivery attempts.

Any subscription configured with a dead letter topic adds an additional delivery-attempt field to the messages, and the subscriber can use this to track how many delivery attempts were required to deliver the message.

Pub/Sub Messages

Pub/Sub messages consist of the following elements:

ContentDescription
Message dataThe message data must be a string. It may be empty, but in that case, the message has to contain at least one custom attribute.
Ordering keyIf this is set, Pub/Sub must respect the order of how messages with the same ordering key have been published and deliver them to subscribers in the same order.
AttributesThese are message attributes with additional metadata and are specified as key-value pairs. A message can have a maximum of 100 attributes.

By default, these additional attributes are added by Pub/Sub topics to the messages:

  • A message ID unique to the topic
  • A UTC timestamp of when Pub/Sub first saw the message

There’s a general size limit imposed by Pub/Sub on message publish requests (and consequently to messages), which is 10 MB.

Sending a Message with Pub/Sub

Messages have to be sent to topics, and from there, they’re routed to subscriptions.

When sending a message, the publisher has to specify the topic, but there’s no other mandatory addressing parameter.

Summary

In this post, we looked at all the basics of RabbitMQ and GCP Pub/Sub. The two message broker platforms offer the same basic functionality, and they’re conceptually very similar in terms of how they propagate messages from publishers to consumers. Despite using different terminology, both allow flexible topology setup and various routing/filtering options. However, Pub/Sub offers somewhat richer and more intuitive filtering features, making advanced use cases easier.

In the second part of this series, we’ll share practical examples of how one should build up typical queue topologies with RabbitMQ and Pub/Sub.