MQTT High Availability
Pro Edition for Eclipse Mosquitto leverages MQTT High Availability (HA) that allows your smart automation to keep operating, providing a continuous service at all times and in all cases.
How MQTT High Availability works
Clustering is the key to the MQTT high availability functionality. You require a minimum set of three nodes to create an MQTT broker cluster. All required information coming in and out of the broker is synchronized across the cluster at any moment. In other words, all three brokers possess the same vitals for their functioning data.
When one node fails, the MQTT failover mechanism automatically transitions all MQTT broker operations to another node. However, the Configurable Leader Election feature allows you to manually designate a leader node that best aligns with your system’s unique requirements and operational needs.
Enhanced High Availability Monitoring further improves the reliability of the MQTT broker by providing insights into the HA cluster’s status. It generates output metrics that indicate the current number of voting nodes and their real-time connectivity status to the leader node. Additionally, the feature provides an indication of the current cluster status, accessible only on the leader node. This allows you to effectively assess the health of your HA setup and proactively address any potential issues or risks.
MQTT broker normal operation
MQTT broker normal operation
One or more load balancers funnel all traffic to the current leader MQTT broker node while the others stay in follower mode (“Follower nodes”). At any given moment in time, our MQTT Cluster Management (CM) keeps synchronizing the following data within the cluster: persistent sessions, retained messages, message queues, ACL (Access Control List), and authentication information on all clients, as well as the overall cluster status.
MQTT broker failover operation
MQTT broker failover operation
If the leader node fails, the Configurable Leader Election enables you to select a specific node in the cluster as the leader, or the MQTT broker cluster performs re-organization and assigns the role of the leader node to one of the followers. Due to the constant synchronization process, the new leader node is up-to-date on the communication status and clients’ information. Thus, it can take over seamlessly, ensuring smooth and continuous operation of the cluster. Now, the load balancers route all traffic through the new leader.
MQTT broker back to normal operation
MQTT broker back to normal operation
Once the initial node is restarted, it rejoins the cluster, becomes available, and takes over a follower role. Therefore, the operation is back to normal.
Which types of High Availability cluster modes are there?
Pro Mosquitto supports two MQTT HA cluster modes: Full Sync and Dynamic-Security Sync.
Full Sync acts as an active-passive cluster where only one of the three nodes is active and can have clients connect. This is especially useful in failover cases, allowing the active node to synchronize the MQTT session and authenticate information across the cluster.
The Dynamic-Security Sync was introduced to cover even more use cases, such as failover and performance handling. This cluster mode acts as an active-active cluster, where all three nodes can have clients connect to them simultaneously. However, only dynamic security authentication is synchronized between nodes. Check out the HA cluster modes documentation for more information.
Does my use case require MQTT High Availability?
MQTT High Availability ensures that your clients can always reach the broker. But why is enabling MQTT HA so important?
Ensure continuous communication with your clients
Ensure continuous communication with your clients
Every industrial-grade solution that relies on MQTT as a central piece of its communication should employ a high availability setup. Otherwise, MQTT becomes the single point of failure and jeopardizes your whole solution.
For instance, you used an MQTT broker to implement a smart factory solution that interprets all events as published and received messages and, based on them, performs specific actions. When your broker is offline, it won’t publish data. In other words, your smart solution won’t be functioning at all.
Sometimes a broker node might not be reachable. Often, it is not the broker that causes it but the underlying server hardware, operating systems, network connectivity, etc. Therefore, we recommend using MQTT Broker with High Availability to ensure your solution doesn’t suffer from outages.
Avoid loss of data
Avoid loss of data
Many typical MQTT clients run on constrained devices with limited resources and cannot (really) persist data. Even if a client can store data when connectivity is lost, this configuration becomes too complex to implement since you must consider too many aspects when choosing one for your solution.
For example, you have a sensor that collects data and uses an MQTT broker to publish information on different topics. When a server goes down and the MQTT broker loses connection, all information gathered by a sensor during the server’s downtime period will not be published by the broker and will not be received by clients. As a result, your system will lose data between when the broker lost the connection and when the connection was finally restored.
If your system were equipped with the MQTT High Availability, it would continue to operate seamlessly since switching from one node to another takes only several seconds. As a result, all information between the two mentioned earlier data points would still be available.
Furthermore, you can maintain the stability of your setup by integrating Enhanced High Availability Monitoring. This allows you to gain insights into your HA cluster’s status and take preemptive action before any potential issues or risks arise.
How does MQTT High Availability differ from single-node systems?
Clients see the MQTT High Availability systems as single-node brokers. The clients have no idea whether they are connected to node No. 1 or No. 2. What they know is that they are connected to a broker.
Due to performance reasons, single-node brokers access the hard drive, store the current status, and queue only a few times per hour. If a single-node broker fails, any changes or queue status updates after the last and before a new disk writing cycle will be lost.
The Pro Edition for Eclipse Mosquitto MQTT broker with high availability can perfectly cope with such situations and avoid data loss by performing constant synchronization. It is important to note that data synchronization in the MQTT High Availability setup follows strict conformation to the OASIS MQTT Specification V5.
In particular, single-node systems largely depend on the stability of the underlying server, hardware, and network connectivity. When one of these components fails, the single-node system is no longer reachable. In this case, the communication between clients interrupts and is unavailable until the failure is rectified. Depending on the type of malfunction, the time required for fixing can take seconds to, in extreme cases, days until replacement parts arrive.
Instead, the MQTT High Availability system’s nodes typically operate on different servers. It is even possible to place them in different geographical regions physically. Hence, if there is an outage of network servers, the MQTT broker node itself, or any other component, the operation is automatically switched to one of the follower nodes within less than a second. The above scenario is how the Pro Mosquitto MQTT High Availability works. Moreover, it is precisely how the MQTT High Availability setup dramatically increases the overall system availability and, at the same time, ensures the smooth operation of your IoT infrastructure.
MQTT High Availability extra questions
If you have any further questions, feel free to contact us.
A load balancer performs the crucial monitoring function for the Pro Mosquitto MQTT cluster. It is responsible for checking the availability of servers and closing ports for nodes that are no longer available. When the leader server fails, the cluster re-organizes and defines a new leader.
As soon as a new leader is determined, the load balancer sends all clients there. Although running an MQTT cluster with only one load balancer is possible, we recommend using three. Per default, Pro Edition for Eclipse Mosquitto MQTT High Availability is configured to operate using three load balancers with different IPs to avoid introducing a new single point of failure (SPOF) with one load balancer. Usually, the load balancers’ IPs are summarized in your Domain Name System (DNS) under a common URL. Like this, clients can still be configured for a single access point URL, although the load balancing service is redundant.
Our team can also help you set up your system with a floating IP configuration. In this case, external clients will only contact a single IP address. The floating IP service, in turn, can then internally re-route the request to the three load balancer IPs. To understand whether we can enable this configuration in your environment, we must first check your setup and what company performs the hosting provider’s role.
Yes, you can. However, typically, we need to double-check if such a configuration would work. In this case, we look together with you at the overall system configuration and determine the feasibility. If necessary, our team is ready to support you during the system specification, implementation, and testing stages and provide other professional services on request.
Our full HA mode (Pro Mosquitto version 2.5 onwards) is a minimum three-node active-passive cluster, with only one node active at once – meaning clients only connect to that node. In this case, all clients can communicate with each other. The cluster synchronizes dynsec authentication and authorization information across the cluster.
It also synchronizes persistent MQTT information across the cluster. Therefore, retained messages are present across all nodes. It also means that clients using a clean session value set to “false,” or using session-expiry-interval will have their session synchronized across the cluster. So if the active broker node goes down, when the client reconnects all of its subscriptions, both in-flight and queued messages will be present on the newly active node.
As far as the client is concerned, it is the same node.
Our dynsec HA mode (Pro Mosquitto 2.6 onwards) is a minimum three-node active-active cluster, with all nodes active simultaneously.
Clients can connect to any node. In this case, clients may not be able to communicate with one another. The cluster synchronizes dynsec authentication and authorization information across the cluster. It does not synchronize anything else. This is a highly available setup because a node should always be available for a client to connect to. This mode has the advantage of being able to scale horizontally. It has the disadvantage that it doesn’t meet all usage patterns. It is well suited to the case where you have e.g., sensors reporting back to a central backend service, but the sensors do not communicate with each other.
This mode has limited support for clients using a clean session value set to “false.” If we use HAProxy as the load balancer, it can inspect the CONNECT packets and always direct clients to the node they previously connected to. Therefore, if the client loses connection briefly it will be able to reconnect to its own node, but if the node goes down, the session information is not available.