Understanding the key models for messaging systems and the differences between them.
With the explosion in the number and variety of applications and application components that collect or process data, building the connections that move data from where it is created to all of the different places that it is processed is a fundamental part of application development. Messaging is a critical technology for connecting data in this environment.
Messaging is the transmission of data records from one system to another via an intermediate system. In contrast to direct connections, where senders know the receivers to which data needs to be sent and connect directly to each receiver, messaging solutions decouple the sending of data from data consumption. Senders do not know which receivers will see that data or when they will see it.
Messaging solutions simplify development of application by giving architects and developers a standardized, reusable component for robustly handling the flow of data so that they can focus on the core logic in their applications. In addition to routing data, messaging systems can also provide important features such as fault tolerance, logging, and distributed processing to improve manageability and reliability.
Messaging is a broad term that covers several models that differ based on how data is moved from senders to recipients. The two primary messaging models are message queuing and publish-subscribe (often referred to as pub-sub) messaging.
Message queuing is designed for scenarios such as a tasklist or workqueue. A message queue receives incoming messages and ensures that each message for a given topic or channel is delivered to and processed by exactly one consumer. For example, a message queuing scenario could be issuing paychecks–it is critical that every paycheck is issued once and only once, but it does not matter which exact machine does the processing of each check.
Message queues can support high rates of consumption by adding multiple consumers for each topic, but only one consumer will receive each message on the topic. Which consumer receives which message is determined by the implementation of the message queue. To ensure that a message is only processed by one consumer, each message is deleted from the queue once it has been received and processed by a consumer (i.e. once a consumer has acknowledged consumption of the message to the messaging system).
Message queues support use cases where it is important that each message is processed exactly once, but it is not necessary to process messages in order. In the case of network or consumer failures, the message queue will try resending the message at a later time (not necessarily to the same consumer) and as a result that message may be processed out of order.
Like message queuing, publish-subscribe messaging moves information from senders to consumers. However, in contrast to message queuing, publish-subscribe messaging allows multiple consumers to receive each message in a topic. Further, pub-sub messaging ensures that each consumer receives messages in a topic in the exact order in which they were received by the messaging system.
Publish-subscribe messaging systems support use cases that require that multiple consumers receive each message and/or that messages are received in order by each consumer. For example, a stock ticker service can be used by a large number of people and applications, all of whom want to receive all messages on their topics of choice. It is important for these users that they receive messages in order–seeing a high price followed by a low price for a stock means something very different from seeing a low price followed by a high price.
Publish-subscribe use cases are frequently associated with stateful applications. Stateful applications care about the order of messages received because the ordering of messages determines the state of the application and will as a result impact the correctness of whatever processing logic the application needs to apply.
Both message queuing and pub-sub messaging models are necessary in modern applications. There are a number of technologies that support message queuing, publish-subscribe messaging, or both. Technologies such as Apache ActiveMQ, Amazon SQS, IBM Websphere MQ, RabbitMQ, and RocketMQ were developed primarily for message queuing use cases, while systems such as Amazon SNS, Apache Kafka and Google Cloud Pub/Sub were designed primarily for publish-subscribe use cases. There are also solutions such as Apache Pulsar that provide support for both message queuing and pub-sub messaging.