MQTT Retained Messages Explained
Typically, if an MQTT publisher sends a message to a topic and there is no active subscriber to this topic, the MQTT broker simply discards the message. In cases where ensuring the reception of messages is crucial, the ability to persist or retain them may be necessary – and this is where MQTT retained messages come in.
What are MQTT retained messages?
MQTT retained messages are PUBLISH packets with the retain flag set to true (retain flag = 1). If you’re familiar with MQTT, you know that MQTT packets have various properties and flags. One of these available flags is the retain flag, which tells the broker that the message must be stored, or in other words, retained.
It is important to note that retained messages also work with MQTT wildcards, which means that a subscribing client will receive the retained message for the topic that matches the wildcard.
For example, client A stores a retained message for the topic abc/outlet01/alert01, while client B subscribes to all topics matching this pattern: abc/outlet01/#.
Once client B is connected, it will receive the stored message matching the topic outlet01, such as abc/outlet01/alert01.
In the next section, we dive deeper into why retained messages in MQTT might be preferable over normal messages.
Why do you need MQTT retained messages?
While Quality of Service (QoS) in MQTT offers a certain level of guarantee in message delivery, the retained message flag ensures that as soon as an MQTT client (newly subscribed or back online) connects to the broker, it receives at least one message.
This is important because it can be challenging to anticipate when the next message will be published on a topic. In some situations, many downstream applications rely on the timeliness of these messages, as they cannot afford to wait. Retained messages guarantee that newly connected clients receive the messages instantly, which eliminates the need for extended waiting periods.
How MQTT retained messages differ from normal messages?
As you already know, an MQTT retained message is a PUBLISH packet with the retain flag = 1. This flag tells the broker to store the message for a topic and ensures any new client subscribing to that topic will receive the stored message.
When the retain flag in a PUBLISH packet is set to 0, only subscribed clients who are currently active will receive the published message for a particular topic. A disconnected client will not receive the published message from that dropped session.
Please note that each MQTT topic can store only one message at a given time. So when the retain flag in MQTT is true, the MQTT broker only stores the latest message on that topic.
Now that you have a better understanding of retained messages, let’s talk about the structure of an MQTT retained message.
Retained message structure
In an earlier article on MQTT packets, we know that the typical message structure looks something like this:
In the control field, a few flags are available, including the retain flag. An example of a control packet with the retain flag set to true is:
0b00110011 = 0x33 → PUBLISH, DUP=0, QoS=1, Retain=1If you’re keen on learning more about MQTT retained messages, here’s how they work in detail.
MQTT retained message examples
Sending MQTT retained messages
If you are a developer, sending MQTT retained messages is straightforward. Most MQTT client libraries available in the market provide a convenient way to enable the retain flag. By leveraging this feature, you can guarantee that important messages remain accessible, even in the event of unexpected disconnection from subscribing clients.
This is an example of how this feature looks like in the Mosquitto publish client:
mosquitto_pub -t "/test/example" -m "{"state":"off", "value":30}" -rDeleting retained messages
As previously discussed, the broker only stores one message per topic, meaning only the latest message with the retain flag set to true, will be retained. However, this also implies that manually deleting retained messages may not even be necessary, unless there is no longer a need for any retained messages for that topic.
That said, it is possible to explicitly delete a retained message by publishing a zero-byte message with the retain flag set to 1 for the same topic. This will notify the broker to remove the existing stored message, preventing new subscribing clients from receiving the previously retained message.
Let’s use an example to illustrate this.
31 10 000c 72657461696e2f746f706963 31This is what a PUBLISH packet with the retain flag set to 1 looks like, but let’s focus on the important sections of this packet and take a look at what they mean:
- 31: Indicates that this is a PUBLISH packet, with QoS 0, a false
DUPflag, and a trueretainflag - 10: Denotes the decimal 16 in the MQTT packet variable header remaining length
- 72657461696e2f746f706963: Indicates the topic name
retain/topic - 31: Payload of this message, the UTF-8 “1” character
To delete a retained message on the topic retain/topic, you can send a PUBLISH packet with a zero-byte payload, which looks like this:
31 0f 000c 72657461696e2f746f706963When comparing the two packets, it becomes clear that the key difference is that the second message ends with the topic and does not have a payload. By sending this message, the broker will clear the existing retained message on the topic retain/topic.
When should you use MQTT retained messages?
As you already know, MQTT message transmissions use a publish-subscribe messaging model, which effectively decouples subscribers from publishers. However, this also means that subscribers cannot directly get messages from publishers. Instead, they need to actively subscribe to the topic and wait for the next published message.
Retained messages help avoid the delay in time subscribers usually have when subscribing to a new topic. These messages ensure newly subscribed clients get the latest update for that topic, which is particularly helpful when there is uncertainty about the timing of the next message publication.
You should use MQTT retained messages, if:
- Your subscribers need the latest information directly upon connecting.
- Your topic payload remains unchanged and is only published once, but every new subscriber requires information such as the system version number.
Wrap up
In MQTT, a publisher sends a message to a topic, and the broker routes them to the respective subscribing clients. If a topic does not have any active subscribers, the message sent to it will be discarded. This means subscribing clients will never receive the message, even if they reestablish a connection. In such situations, you may need to use retained messages.
MQTT retained messages typically serve to store crucial messages in the event any of your subscribing clients become disconnected. They follow the same format as MQTT PUBLISH packets but with the retain flag set to true. This tells the broker to store the message for this topic and send it to every new subscribing client.
Each topic can store only one retained message. If multiple retained messages are published to a topic, the broker will override the existing message and store the latest one. This process guarantees that only the most recent message remains retained, eliminating the need for manual deletion of older messages. However, if you ever need to remove retained messages manually, you can do so by publishing an empty message with the retain flag set to true.
As mentioned earlier, MQTT retained messages have numerous benefits, including the ability to deliver critical messages to every new subscribing client. That said, it may be challenging to discern whether MQTT retained messages are suitable for your project. You should utilize MQTT retained messages if:
- Your subscribers need to know if the system is online or offline
- Your system does not change, but every new subscriber needs to get information like the system version number.
Keen on trying MQTT retained messages in your next IoT project? Try the Pro Edition for Eclipse Mosquitto. You can test some of the pro features by signing up for a free trial. If you hit any roadblocks while using the open-source version, take advantage of our new professional support service.
About the author
Roger has a background in Electronic Engineering academic research focusing on the design of scientific optical imaging sensors, accompanying control and acquisition systems, and other embedded designs.
He has 6 years of Academic teaching responsibility, teaching semiconductor chip design, embedded design, and programming.
Roger started Mosquitto as a hobby project in 2009, which gained rapid popularity and contributed to the widespread adoption of MQTT as an IoT protocol. He also developed an MQTT client library in Python, and both projects became part of the Eclipse Foundation in 2014/15. In 2018, Roger joined Cedalo to develop Mosquitto further, starting with implementing MQTT v5.0 support.
Subscribe for monthly updates
