MQTT basics

The MQTT Broker Handbook: Your Guide to a Connected World

Zakhar Yung06.04.202320.04.2023

The lives of contemporary humans are intertwined with the devices and data they generate or share. Sensors provide valuable information about weather, health, water quality, and more. In some cases, we use a synergy of nature and technology in IoT systems to gather even more refined information. For instance, Warsaw’s water supply is monitored by eight clams that detect pollutants such as heavy metals or pesticides in the city’s drinking water. If the mollusks detect such substances, they close their shells, which sends signals to the server to automatically cut off the water supply. 

The central component of every IoT or monitoring system is a server that manages communication between connected devices. If your system uses the MQTT messaging protocol, this server is referred to as an MQTT broker. This handbook offers a comprehensive introduction to MQTT brokers, including their types, configuration, and other relevant information. Let’s start by gaining a better understanding of what MQTT brokers are.

What is an MQTT broker?

An MQTT broker is a server in an MQTT-based system acting as a central hub for the communication of connected devices. It enables the devices, known as clients, to publish and subscribe to messages. 

As a refresher, MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol used in IoT systems to enable communication between devices.

An MQTT broker is usually implemented as a standalone server application running on a server or a computer. You can install the MQTT broker software on various operating systems, including Windows, Linux, and macOS.

The MQTT broker is also often used in conjunction with gateways or routers. Such an implementation provides higher reliability and performance and is often used in industrial IoT systems and large-scale automation workflows.

MQTT broker’s role in the MQTT architecture

The MQTT broker’s role in MQTT systems is best described in our MQTT connection beginners guide:

An MQTT broker is the centerpiece and orchestrator of the MQTT infrastructure, responsible for establishing and maintaining secure connections with clients and routing messages between them.

Here are the duties the MQTT broker performs:

• Routing messages: The MQTT broker routes messages between clients: from publishers to subscribers.
• Maintaining state: The MQTT broker keeps track of the current state of the MQTT system, including which clients are connected (MQTT Keep Alive), the topics being published and subscribed to, and managing the MQTT Last Will. 
• Authentication and encryption: The MQTT broker can protect the data being transmitted and ensure that only authenticated clients have access to the MQTT system.
• Authorization: The MQTT broker ensures that clients can only access topics where they are authorized. 

The broker filters for:

• IP-address
• port
• username
• password
• client name
• client id
• qos
• retained message
• LWS
• Clean Session
• dup flag
• topic

Overall, the MQTT broker acts as a centralized hub in the MQTT architecture, facilitating communication between clients and maintaining the system’s state.

A plain MQTT broker example in an IoT system

Let’s imagine an IoT system consisting of several temperature sensors that need to publish temperature readings to a central server. The central server will then forward the temperature readings to other applications or devices that request this information. 

On the central server, we would install the Mosquitto broker, which is a popular open-source MQTT broker. The IoT devices (temperature sensors) would connect to the Mosquitto broker and publish temperature readings on a specific topic. For example, "temperature/kitchen" is the topic for the temperature in the kitchen. Other apps or devices that are interested in receiving this information would subscribe to this topic.

The Mosquitto broker would receive the temperature readings from the IoT devices and route the messages to the clients subscribed to this particular topic. If any of the subscribers are temporarily disconnected from the system, the Mosquitto broker would queue the messages and deliver them once the subscribers are reconnected.

In this example, the Mosquitto broker orchestrates the communication between devices and apps to ensure the delivery of temperature readings in a reliable and efficient manner.

What we have shown here is the typical MQTT PUB-SUB model of how an MQTT broker works in an example of a simple IoT system. Let’s check out other ones. 

Common MQTT use cases and MQTT broker’s implementation 

You probably know about smart homes as a major implementation of MQTT brokers. It’s true, however, MQTT brokers are widely used in multiple other IoT and machine-to-machine (M2M) applications, including industrial automation, healthcare monitoring systems, energy management, etc. We’ve prepared a few scenarios explaining how this or that MQTT system works and how the communication between an MQTT broker and clients occurs. 

Smart home automation

To demonstrate using an MQTT broker in smart home automation, we select the implementation of a smart lighting system. It consists of light bulbs equipped with sensors that publish data on their state (on/off) (topic: lightsensor/state) and brightness level (topic: lightsensor/brightness) to the MQTT broker. The broker receives messages from these light bulb sensors and forwards them to a mobile application (for instance) which takes the role of the subscribing client. The client has subscribed to the lightsensor/state and lightsensor/brightness topics. 

From this mobile app, users can control the light bulb status and brightness by sending messages (PUBLISH) to the MQTT broker, which then forwards them to the light bulbs (they become Subscribers). The light bulbs receive messages and update their state or brightness accordingly, publishing new messages to the MQTT broker to reflect the changes.

Healthcare monitoring

Remote patient monitoring can be a good example of demonstrating the usage of MQTT brokers in healthcare automation. The IoT system comprises wearable devices, such as smartwatches or wearable sensors. Their sensors acting as MQTT clients publish data on the patient’s vital signs like heart rate, body temperature, and more to the MQTT broker. The broker forwards messages to the subscriber, e.g. a healthcare monitoring system, for analysis. If any unusual patterns or critical levels are detected in the patient’s vital signs, alerts are sent to healthcare professionals.

This case is perfect to pinpoint the value of the MQTT protocol for sending important or even critical data within constrained networks. Any message loss can directly impact the reliability of your IoT project. With this in mind, MQTT provides three Quality of Service (QoS) levels to ensure the delivery of messages. For example, you can set the QoS level to 2, and the MQTT broker will route the messages from publishers to all subscribers with a guaranteed reach to their destination. For more details on this, read our guide on MQTT QoS.

Industrial automation

MQTT systems can also be used in industrial automations, for example for production line monitoring. In different areas of a production line, multiple sensors and actuators are installed. These MQTT clients publish data on the production process, such as machine performance, product quality, and more. The MQTT broker receives messages from these devices and forwards them to the production monitoring system (IoT platform) where the data is being analyzed. Based on the analysis, the monitoring system provides real-time insights into the production process, e.g. a quality management system could identify any potential bottlenecks or areas for improvement. Operators can also control the production line by publishing control messages to the MQTT broker, which then forwards them to the actuators.

Energy management

Energy management is another interesting case of using MQTT brokers. Smart grid system is a good example to demonstrate this application. An IoT system contains energy meters, the devices that publish data on energy consumption. The MQTT broker receives messages from energy meters and forwards them to the energy management system, which aggregates the data to provide a real-time view of energy consumption across the grid. The system can also use these readings to optimize energy distribution and reduce waste by balancing demand and supply.

Agriculture monitoring and automation

The final use case we’d like to cover is monitoring and automation in agriculture. An MQTT system can include agricultural sensors installed in fields to collect data on soil moisture, temperature, and other environmental factors. This data is published via messages to the MQTT broker, which routes it to the farm management system. Later on, the aggregated data is being analyzed to provide real-time insights into the crop growth conditions. The farm management system can also use the data to optimize irrigation and fertilization, reducing water and fertilizer waste while improving crop yields.

Apart from the introduced cases, MQTT brokers are common for different monitoring systems. In transportation, MQTT brokers can be used to monitor and control vehicles, for example, to navigate automated guided vehicles that collect items around Amazon’s warehouses. In environmental monitoring, MQTT brokers can route data about temperature, humidity, and air quality from environmental sensors to improve sustainability. You can apply brokers in a wide range of IoT setups where real-time communication and monitoring are required. 

Are there many types of MQTT brokers?

Basically, you can differentiate MQTT brokers by their accessibility – public or private, as well as hosting – cloud-based or self-hosted. 

We’ve collected a brief description of these types in one table. Below, you will also find more details of each type and their examples.

• Public MQTT brokers are accessible to the public and can be used by anyone for testing or prototyping purposes. This type of broker provides a shared infrastructure for IoT devices to publish and subscribe to messages. Popular examples of public MQTT brokers are test.mosquitto.org, and mqtt.eclipseprojects.io.
• Private MQTT brokers are not accessible from the public Internet but set up within an organization’s own network. This type of broker provides greater control over data privacy and security compared to public brokers. Private MQTT brokers are typically used for secure and reliable communication between IoT devices within an organization and can be set hosted on-premise or in the cloud.
  • Cloud-based MQTT brokers are hosted and run by a cloud service provider. They offer the benefits of the cloud computing model, such as automatic scalability, high availability, and reduced maintenance overhead. With a cloud-based MQTT broker, organizations can get a ready-to-use solution saving costs and resources on setting up and maintaining an in-house MQTT broker. Some popular cloud-based MQTT brokers include Pro Edition for Eclipse Mosquitto or CloudMQTT.
  • Self-hosted MQTT brokers are installed and run on a server within an organization’s own infrastructure. This type of MQTT brokers provides a high level of security and privacy, and greater control and customization options, as the organization can configure the broker to meet its specific requirements. However, this also requires more technical expertise and resources to set up, configure, and maintain the infrastructure and the very MQTT broker.

What are the popular MQTT brokers?

You can encounter many MQTT brokers from open-source solutions like Mosquitto to platforms-as-a-service like Microsoft Azure IoT Hub. We have singled out the top 3 popular MQTT brokers worth your attention.

Pro Edition for Eclipse Mosquitto

Pro Edition for Eclipse Mosquitto is a more powerful and fully supported version of the open-source Mosquitto MQTT broker. The Pro Edition is designed for commercial use, offering a set of features to ensure reliable and failsafe operation.

• MQTT version supported: v3.1,  v3.1.1, v5.0
• Programming language: C
• Scalability: A single Pro Mosquitto node can handle up to a million of connected devices.
• Security: Supports username and password authentication, X.509 certificate-based authentication, and access control lists (ACLs).
• Core features: High Availability, Mosquitto Management Center, REST API, LMDB persistence.
• Ideal for: Small to medium-sized, as well as Enterprise-grade IoT projects and those requiring a high degree of customization.
• Use cases: Mosquitto Pro can be used for many IoT applications, including smart homes, industrial automation, and more.

Sign up for a free 14-day trial to check out all these benefits of Pro Edition for Eclipse Mosquitto. 

AWS IoT Core

AWS IoT Core is not just an MQTT broker but a cloud service to connect and manage devices and their interaction with each other and cloud apps.

• MQTT version supported: v3.1.1, v5.0
• Programming language: Java, Python, C++, Node.js, .NET, etc.
• Scalability: AWS IoT Core offers highly scalable cloud infrastructure, allowing you to connect and manage millions of devices.
• Security: Supports X.509 certificate-based authentication, Amazon Cognito user pools, and AWS Identity and Access Management (IAM) policies.
• Core features: Device Advisor, Device Gateway, Rules Engine, Registry, integration with other AWS services, such as Amazon Kinesis and Amazon DynamoDB.
• Ideal for: Enterprise-level IoT projects that require a secure and scalable cloud infrastructure.
• Use cases: AWS IoT Core can be used for a wide range of IoT applications, including smart homes, smart energy management, predictive maintenance, and more.

HiveMQ

HiveMQ is an enterprise-ready MQTT broker that provides data movement between connected devices.

• MQTT version supported: v3.1, v3.1.1, v5.0
• Programming language: Java
• Scalability: HiveMQ can handle millions of connected devices and can be easily scaled to accommodate more devices as needed.
• Security: Supports username and password authentication, client certificate authentication, and access control lists (ACLs).
• Core features: High Availability Clustering, Real-time monitoring dashboard, MQTT Client Drill-Down Analysis.
• Ideal for: Enterprise-level IoT projects that require a highly scalable communication infrastructure.
• Use cases: HiveMQ can be used for industrial automation, predictive maintenance, and connected logistics.

These are only the top 3 options in our opinion. However, the list of MQTT brokers available for different tasks can be very long. To choose the ideal solution for your needs, you should follow an actionable checklist that we’ve introduced below.

The checklist on how to choose an MQTT broker

Here are the seven factors or challenges to consider when searching for an MQTT broker for your IoT system:

• Type of MQTT broker (cloud-based, self-hosted)
• Scalability
• Sufficient security support (support of encryption and authentication to secure communication between clients)
• Interoperability (MQTT broker’s ability to work seamlessly with a wide range of devices, platforms, and protocols)
• Administration
• Support
• Cost

Let’s explore what each of these challenges means for making the final decision on which MQTT broker your project requires.

What MQTT broker challenges might you encounter?

Type of MQTT broker

The choice of the type of an MQTT broker mostly depends on the infrastructure your project will be built on. If you have the resources to set up in-house infrastructure, you can go with a self-hosted MQTT broker. This way is, however, rather expensive.

Cloud MQTT brokers are a widespread option since they do not require you to mess with setting up infrastructure from scratch. You can get an out-of-the-box solution and start using it without much effort spent on the setup.

MQTT broker scalability

The main question here is how many devices (clients), you expect your IoT system and hence MQTT broker to handle in short and long-term perspectives. For example, you can start with ten sensors but then increase their number to 1,000. This means the volume of data transferred between clients will increase by 100 times. Your MQTT broker should be ready for this scalability scenario. For this, you should consider the support of specific scalability mechanisms:

• Load balancing that enables distributing the load across multiple broker instances. This increases the overall capacity of the system and reduces the risk of overload due to high traffic.
• Clustering that allows multiple instances to work together as a unified MQTT broker. Clustering is also the key to the MQTT high availability functionality.
• Partitioning that enables dividing a large MQTT broker into smaller units. This improves scalability and reduces latency.
• Caching that allows MQTT broker to store frequently-used data in memory. This reduces the load on the broker and improves its performance.

MQTT broker security

Perhaps, the security factor should have gone before scalability in our checklist. Feel free to tune this if this has a higher priority for your project. Security for MQTT brokers means the support of encryption and authentication to secure communication between clients. 

So, take into account whether the desired MQTT broker provides options for secure communication, such as SSL/TLS, and what options for user authentication are provided. For instance, MQTT brokers can implement user authentication using username+password or digital client certificates. 

Authorization, or access control, is another security mechanism to consider. The MQTT broker checks whether an authenticated client is allowed to publish to a topic. Brokers can restrict access based on different parameters including IP address, username, topic, QoS, etc. 

MQTT broker interoperability

Interoperability means the MQTT broker’s ability to work seamlessly with a wide range of devices, platforms, and protocols. This can have several key benefits including increased compatibility, easier integration, and enhanced reliability. So, interoperability is an important factor that you should consider when choosing an MQTT broker.  

Administration of an MQTT broker

MQTT broker administration determines how easy it is to set up, maintain, and monitor the status of the broker and associated components. In this regard, an MQTT broker should have the following characteristics:

• Ease of use
• Centralized management
• Flexible configuration
• Monitoring and troubleshooting tools available

Good administration capabilities will also help you manage security for the MQTT broker, including access control and security events monitoring.

Support for your MQTT broker 

Support is one of the key factors to consider when choosing between an open-source and commercial solution. The level of community support for the MQTT broker can be high including the availability of online forums and documentation. However, the access and quality of commercial support options are more valuable for projects that cannot afford even a short downtime of their IoT systems. An example of this is a professional support service for Mosquitto.

Costs associated with MQTT broker 

By cost as the factor in choosing an MQTT broker, we mean not only the subscription plan of cloud-based solutions but all the costs associated with the MQTT broker setup and maintenance, including licensing fees, integration with other systems, etc.

Basically, all the crucial factors for choosing an MQTT broker are the challenges you may face. So, you have to foresee that your MQTT broker can:

• Handle a large number of concurrent connections and increased levels of data processing and storage when the number of connected devices goes up.
• Prevent unauthorized access or malicious attacks.
• Guarantee the delivery of messages and ensure that they are not lost or duplicated if that is not intended.
• Support a wide range of devices and platforms.
• Be easy to maintain and upgrade.
• Provide clear logging and monitoring information.

How to set up an MQTT broker

Enough for theory. Are you ready to get your hands dirty? So, let’s demonstrate what the setup of an MQTT broker may look like. Of course, the setup flow will differ depending on the chosen broker, the platform to host it, and some other factors. However, you should have a principal understanding of the steps to complete. 

For this example, we’ve chosen the #1 MQTT broker in the world – Eclipse Mosquitto. It’s available for MS Windows, Linux, and macOS. You can also install it on Raspberry Pi if you plan to build your IoT system using this computer. 

Installing Mosquitto MQTT broker on Windows 

In our case, we’re installing Mosquitton on Windows 11. For this, we downloaded the latest version of Mosquitto for Windows. Alternatively, you can use the official Docker-based installer image. 

• Run the installer file then click Next.

• Select the components to install.

Note: If you install Mosquitto with the Service checkbox checked, the MQTT broker will start automatically with Windows and occupy the default port 1883. So, you won’t have to run it manually every time. 

• If necessary, you can change the install folder. Click Install.

• Click Finish to close the setup dialog window. Congrats, Mosquitto MQTT broker is set up on your computer. However, it’s not run yet. 

. For this, run the terminal (Command Prompt) as admin user, and enter the following command to check the supported command line options. 

mosquitto>mosquitto -h

To start the Mosquitto MQTT broker, navigate to the folder using the following command:

 cd C:\Program Files\mosquitto 

Then enter the following command to run the broker:

mosquitto -v

Note: There can still be an error, so you might need to enter:

mosquitto -h

You will see your MQTT broker running on port 1883.

Mosquitto as an application or a service

If you get an error Only one usage of each socket address (protocol/network address/port) is normally permitted, you might have another application already running, listening on port 1883. This is likely that the Mosquitto broker is already running as a service. 

To stop Mosquitto MQTT broker as a service, use the following command:

net stop mosquitto

How to configure an MQTT broker

Configuration of an MQTT broker usually includes setting up authentication and authorization, enabling SSL/TLS encryption, configuring MQTT topics, and much more.

In the case of Mosquitto MQTT broker, you can configure it using its configuration file, mosquitto.conf, located in the installation folder.

This file contains everything that you can possibly do with Mosquitto MQTT broker configuration. The file contains the following configuration sections:

• General configuration
• Listeners
• Certificate-based SSL/TLS support
• Pre-shared-key based SSL/TLS support
• Persistence
• Logging
• Security
• Default authentication and topic access control
• External authentication and topic access plugin options
• Bridges (to connect multiple MQTT brokers together)

To configure your Mosquitto MQTT broker, open mosquitto.conf in a textual editor like Notepad and specify the desired configuration parameters. 

Then you need to remove # for the configuration line that starts with a variable name and specify the variable value separated by a single space. For example, by default, Mosquitto MQTT broker does not allow anonymous connections.

To enable anonymous connection, change this variable to TRUE like this:

Once you have updated the mosquitto configuration file, always make sure to save your edits (Ctrl+S) and reload the MQTT broker. Use the command 

net stop mosquitto

then 

net start  mosquitto

How do you know that this edit has come into effect? Let’s try to connect to the MQTT broker!

How to connect to MQTT broker

To connect a client to an MQTT broker you need to know the broker’s connection details: 

• hostname or IP address
• port number
• required authentication details (such as a username and password)

Then configure the client so it can send a CONNECT request to establish an MQTT connection. We’ve explained the details in our MQTT Connection Guide for Beginners. Let’s see how this works in real life in the example of our freshly setup Mosquitto MQTT broker. 

We’re going to test the connection using the MQTT Explorer, a comprehensive MQTT client. We’ve downloaded and installed the latest version for Windows from here. 

And these are the connection parameters for our Mosquitto MQTT broker:

• Host: localhost
• Alternatively, we may use the IP address, which is 127.0.0.1.
• Port number: 1883
• Username and password can be left empty since we did not configure authentication.

Click Connect and there you go!

And here is what the connection looks like in the terminal.

Note: For this example, we’ve run Mosquitto as an application, not a service.

As for the PINGREQ and PINGRESP communication between the client and the broker – they allow keeping the MQTT connection alive.

Let’s disconnect and make a new connection to demonstrate the PUB-SUB model in action: how clients can subscribe to topics and publish messages.

And here is what it looks like in the terminal.

How a client disconnects from an MQTT broker

Although you’ve already seen the disconnection on the screenshot above, it would be logical to wrap up this MQTT broker setup section by explaining how clients disconnect from the broker. 

It’s quite easy – a client sends a DISCONNECT message to the broker thus saying goodbye 🙂 This disconnects the connection between the client and the broker in a proper manner.

However, the connection can also be broken without the client’s sending the DISCONNECT package, for example, if a device is out of order. In this case, the MQTT broker sends the MQTT Last Will message to all the subscribers of this client’s topic according to the specified rules. 

You can also check out this guide dedicated to installing Mosquitto on Windows for more information.

Final question: Why is a server in MQTT called a broker?

We decided to sum up our MQTT broker guide with an interesting question that some rookies may have. Why is it called an MQTT broker not an MQTT server when calling it a server would be quite logical? For instance, clients post and get data via the server in HTTP. But HTTP and MQTT are made for different purposes as we’ve blogged about in our HTTP vs MQTT for IoT guide, so, are the brokers in MQTT. 

Well, a broker in real life is an individual or company that acts as an intermediary between an investor and a stock exchange. In MQTT, a server also acts as an intermediary or middleman between clients: it receives messages from publishers and routes them to subscribers. It does not store any data, still it queues data that hasn’t been sent or received yet – it’s intention is to route messages. So, the server acts as a broker. Besides, in computer networking, brokers refer to software that facilitates communication between entities.

Using the term MQTT server instead of MQTT broker isn’t a huge mistake. However, if you even google ‘mqtt server’, you won’t find this exact keyword among the top 10 results. Google will return ‘mqtt broker’ instead.