Dev guides

How to Test a Mosquitto MQTT Broker in JMeter: Detailed Guide

Andreas Schiffler11.10.202316.12.2023

MQTT is a widely used protocol for IoT (Internet of Things) communication, where an MQTT broker’s role is to connect devices and transfer messages using the publish/subscribe model. 

As a matter of fact, professional MQTT broker setups often require handling many client connections and high data rates. This makes an MQTT broker’s scalability and performance crucial for any IoT solution. 

In this article, I will  demonstrate how to test the functionality and performance of the Mosquitto MQTT broker using various tools.

Choosing the metrics for Mosquitto MQTT broker testing

The aim of designing a test plan for network or data-based services is to verify or derive the performance of the MQTT broker or, generally, the digital service. 

• One performance metric could be the average response delay time under specific load conditions defined by the number of client connections and message transport in bytes/second. 
• Another metric could be the time it takes – including the time for the transport over the internet – for one client to connect to the broker, publish a message, and receive the acknowledgment (PUBACK). 
• Furthermore, another testing criterion could be to verify access to different topics for different client credentials.

Software tools to test Mosquitto

To perform this testing, you need software tools to simulate many client connections and messages at random points in time and possibly connect to the broker from different WAN regions.

• Emqtt_bench is an open-source command line application that enables you to easily simulate connections and loads for thousands of clients. It can be a basic building block for custom test scripts running on multiple real or virtual computer instances. Simulating thousands of client connections is possible, but it is not as easily handled as with JMeter.
• JMeter is an open-source performance testing tool with a GUI written in Java and is highly cross-compatible. It can simulate thousands of client connections and is widely used for load testing of various web services.

There are also fully managed web tools or test suites that use either Emqtt bench or JMeter as core components.

However, this article focuses on the usage and capabilities of JMeter. In the next section, I will explain how to install and use it on a Windows PC.

In this article, I focus on testing the Mosquitto open-source version. However, there is also a Pro version of Mosquitto with commercial features and support, which you can explore and test by signing up for either a cloud or on-premises trial.

JMeter installation and usage on Windows

To begin with, install JMeter on Windows, follow these steps: 

1. Download the zip archive (a higher version may be available in the future).
2. Unpack the archive and browse to the folder.
3. Download the MQTT protocol extension (a higher version may be available in the future).
4. Move or copy the MQTT extension to the lib\ext folder within the JMeter directory (see Step 2.)
5. In the bin folder of the JMeter directory, double-click the jmeterw.cmd file and wait a few seconds. A window will pop up, and the application will be ready for use.

MQTT extension loading verification

To verify if the MQTT extension is loaded correctly, download a JMeter test plan file by right-clicking on the link: first_test.jmx (the file extension is jmx). Then, select Save as. 

In the JMeter GUI, select File from the menu, choose Open, and then select the downloaded file.

If the MQTT extension is successfully loaded in JMeter, you will see a screen similar to Figure 1.

If the MQTT plugin is not found or not correctly loaded, the following message (Figure 2) will appear after opening the test file.

To solve this issue, check the following:

• There is the mqtt-xmeter-fuse-2.0.2-jar-with-dependencies.jar file in the %JMETER_HOME%\lib\ext directory. It can be downloaded here. 
• When you double-click on the jmeterw.cmd file in the %JMETER_HOME%\bin directory (Windows OS), it will launch JMeter.

Creating a simple Mosquitto test plan

Overall, to get familiar with the JMeter interface and develop a simple Mosquitto test plan, start with an easy procedure.

Our plan includes the following steps:

• Connect to an MQTT broker with enabled TLS encryption for the data transport (use MQTT protocol version 3.1.1)
• Authenticate with username and password
• Publish a message on any topic
• Disconnect from the broker

Opening the JMeter GUI for the first time automatically creates an empty test plan. 

You must first create a thread group to define the steps noted above. A thread group is a fundamental element of a test plan in JMeter. It controls the number of threads that JMeter uses to execute a test. Multiple threads are used to simulate concurrent broker connections. To add a thread group, right-click the test plan in the tree view on the left, choose Add, and select Thread group. For simplicity, do not change the default parameters, so that only one thread will be started.

According to the simple test plan from above, you must first add the MQTT Connect action. In JMeter, these types of action are called samplers. To add a sampler, right-click the thread group, choose Add, and select the MQTT Connect sampler. Then, configure the connect parameters in the dialog that opens on the right. Edit the following parameters according to your setup or needs: the broker address, port number, username, password, protocol version, and the setting for the TLS encryption. 

In the same way, add an MQTT Pub Sampler, which will publish a message defined by the parameters in the dialog. Here, you can set the topic to publish the message to, the quality of service (QoS) level, and the message/payload itself.

Now, add the MQTT Disconnect sampler to ensure disconnection from the broker after running the test. It is important to note that the order of the samplers in the tree view will determine the order in which the actions are executed.

As a final component, you will need a listener to collect and save the test results. To add such a listener to our test plan, right-click the test plan, choose Add, then Listener, and add the View Results Tree listener. In the component’s parameters, specify the filename and location on your local disk to save results. For example, set the filename and location to C:\users\admin\documents\test-plan-report.jtl (the JMeter result file has the extension jtl.)

The screen of the JMeter GUI will look like the example shown in Figure 3. If you need help creating this test plan, you can download it here and load it to the JMeter GUI via the menu file and an open file dialog.

You can now run the test plan by clicking the green play button in the GUI. If this is not the first test run, the application will ask if you want to append or overwrite the result file. After the test finishes, if the green Play button in the menu bar is visible, it indicates that the test has produced some results. Click/open the result listener to see an overview of the executed steps (see Figure 3) – red for fail and green for success. To create a featured test plan report with dashboards, graphs, and metrics from the result file, rerun JMeter with two options from the command line.

To create an HTML report from the result file, run the following command line:

jmeter -g C:\users\admin\documents\test-plan-report.jtl -o C:\users\admin\documents\reports

In this command line, the parameter after the -g  option is the filename of your result file set in the result listener. The -o option defines the location where the created HTML report will be stored on disk. After the command line’s execution, open the location specified after the -o option, and you will see some files and folders. Now, open the index.html file in a browser and see the report page similar to the one shown in Figures 8 and 9.

Sign up for a free Pro Mosquitto clour or on-premises trial and test all broker features.

JMeter scripting and automation

Consider the following actions when testing an MQTT broker using JMeter:

• Test different user credentials,
• Test connections with or without TLS-encrypted transport,
• Simulate loads,
• Test access control lists for users/clients subscribing or publishing on specific topics.

Two elements that make the test plan more programmatic and flexible are:

• User Parameters that define one to many different credentials to establish a connection.
• User Defined Variables that define general variables such as the broker address or port number.

You can add these two elements to the thread group by right-clicking the group and selecting Add, then choosing the pre-processor to add user parameters or the config to add the user-defined parameters.

For example, in the user-defined parameters, you set the broker address and the port number. Then, you can use these parameters in the MQTT connect sampler by referring to them in a special syntax with the following pattern: use ${<NAME>} in the settings dialog and replace <NAME> with the name of the parameter. See Figure 4 for a visual representation of this.

You can use these or other user-defined parameters in any setting dialog. For example, to specify the topic, the message payload, or the MQTT client ID.

The other element, also visible in Figure 4, is the user parameters element, which holds the information about which user – which credentials – shall be used in the thread group. 

JMeter can start multiple threads in a thread group, with each thread running with a different user. To do this, first, increase the number of threads to 500. Then, uncheck the thread group option “Same user on each iteration” and define five users (username and password). JMeter ramps up 500 threads and uses the specified credentials. After starting the fifth thread, JMeter will use the first user credentials for thread six because there are more threads than users defined. To see an example, refer to Figure 5.

As a matter of fact, there are two other elements used to control the test sequence:

• The Loop controller repeats a set of samplers or requests for the number of times specified in the test step.
• The timer waits for a specified constant or a random time within a defined time range.

These two elements are also basic building blocks to simulate a stochastic load resulting from many MQTT connections publishing messages on different topics, as shown in the tree view of such a test plan in Figure 6.

In more advanced test automation use cases, you can start the test plan from the command line and set the properties by providing them as arguments on the command line. 

For example, the command line starts the test plan defined by the my-advanced-plan.jmx file and sets the property TARGET_PORT to 1884:

jmeter -n -t c:\users\admin\documents\my-advanced-plan.jmx -JTARGET_PORT=1884

To use this property, define the variables in the test plan, as shown in Figure 7. The pattern is ${__P(<PROPERTY_NAME>,<DEFAULT>)}. Then, replace the <PROPERTY_NAME> with a name value. This name will be used on the command line with the -J option. I recommend using only capital letters to denote that this information is set via command line arguments.

Using JMeter with MQTT extension on Docker to test Mosquitto

The sections above describe the basic steps to create and automate a test plan. 

If you want to start testing immediately, you can use ready test plans that can be customized via the command line. One such test plan, shown in Figure 6, can be downloaded here. 

Alternatively, it can be executed in the virtual environment (github codespace). This git repository also includes a Dockerfile that can be used to build a container to call and run JMeter from the command line.

If you have an installed JMeter with the MQTT extension, you can perform the following steps:

• Clone the repository with the ready test plan and test environment. This will create a “docker-jmeter-server” directory, which will contain a test plan:

git clone https://github.com/aschiffler/docker-jmeter-server

• Navigate to the created directory:

cd docker-jmeter-server

• Start the default test plan (requirement: JMeter is locally available on the Windows machine where you run the following command line):

jmeter -n -t tests\mqtt\sub_pub.jmx -e -l test-plan.jtl -o .\report-dir

• Start the test with custom variables (Windows cmd command). Note that the property values in the command below are just example placeholders. To run these tests, you will have to have a running broker (locally or on the cloud) with two configured users (more about this test plan configuration is explained below):

jmeter -n -t tests\mqtt\sub_pub.jmx -e -l test-plan.jtl -o .\report-dir ^
-JTARGET_HOST=mybroker.cedalo.cloud ^
-JTARGET_PORT=1883 ^
-JUSER1=foo -JPASS1=secret1 ^
-JUSER2=bar -JPASS2=secret2 ^
-JTHREADS_SUB=20 -JTHREADS_PUB=20 -JTOPIC=test

• Start the test with custom variables (if run it in this github codespace):

jmeter -n -t tests/mqtt/sub_pub.jmx -e -l test-plan.jtl -o ./report-dir \
-JTARGET_HOST=mybroker.cedalo.cloud \
-JTARGET_PORT=1883 \
-JUSER1=foo -JPASS1=secret1 \
-JUSER2=bar -JPASS2=secret2 \
-JTHREADS_SUB=20 -JTHREADS_PUB=20 -JTOPIC=test

Alternatively, you can launch JMeter as a Docker container by calling the run.sh script file that starts the roschi/jmeter-mqtt:5.6 Docker container from dockerhub.com. This container version already holds the required JMeter application and the MQTT extension (using: bash shell and Linux OS):

./run.sh -n -t tests/mqtt/sub_pub.jmx -e -l test-plan.jtl -o ./report-dir \
-JTARGET_HOST=mybroker.cedalo.cloud \
-JTARGET_PORT=1883 \
-JUSER1=foo -JPASS1=secret1 \
-JUSER2=bar -JPASS2=secret2 \
-JTHREADS_SUB=5 -JTHREADS_PUB=500 -JTOPIC=test

The repository’s sub_pub.jmx Mosquitto test plan can be used out of the box or modified. Two thread groups are defined: one where the subscriptions to a topic are executed, and the other where messages are published to a topic in a loop with a random wait time. The number of threads in each group can be defined via command line arguments THREADS_SUB and THREADS_PUB. The specific topic is defined via TOPIC. 

Three user credentials are used, independently of the number of threads. Two of them can be passed via the arguments USER1/PASS1 and USER2/PASS2. The third is left empty to test whether anonymous access is possible.

The command line call above also defines the report directory. The filename test-plan.jtl defines the result file.

The test plan uses two known client credentials and one anonymous client to make requests to Mosquitto. Before running the plan, these two clients should be created on the broker, and anonymous access should be disabled. Additionally, publish and subscribe access to one specific topic (defined by -JTOPIC) should be granted only for one client account. The other client should only be able to subscribe. The last steps can be easily achieved with the help of the Mosquitto dynamic security plugin.

After a test run, a report in the form of an HTML file is created in the report directory. If you use the ready-to-run codespace, you can start an HTML live server by clicking the Go Live button in the lower information pane of the codespace environment to open the report in your browser.

Sample results for an MQTT broker test plan

The sub_pub.jmx test plan for Mosquitto from this repository shows that if everything is correctly configured, both clients can connect to the broker. But only one client can subscribe or publish. 

In Figures 8 and 9, you can see two sample screenshots from the JMeter HTML report page. The figures present summarized information. Some portion of the test cases fail. This is due to the configuration where anonymous access is disabled, and publishing on the specified topic for one set of user credentials is impossible.

Wrap up

In this article, I used JMeter to demonstrate the Mosquitto testing process with multiple client connections, different credentials, and actions (subscribe/publish).

In addition, I provided instructions on how to use a ready test plan in a local environment or any virtual instance by using JMeter in a Docker container.

Ensuring productive quality assurance and validation requires multiple test plans to address different scopes and cases. Therefore, the Cedalo team can support you with expert knowledge on Mosquitto profound testing and validating your IoT setup performance requirements.

About the author

Andreas Schiffler

Research Professor at the Technical University of Wuerzburg-Schweinfurt

Dr. Andreas Schiffler is a research professor at the Technical University of Wuerzburg-Schweinfurt in the field of production and data technology in mechanical engineering. In addition to research topics related to 3D metal printing, Dr. Schiffler developed a Kubernetes cluster for the practice-oriented basics of IoT and Industry 4.0 as part of the student training. Before joining the university, he worked in different product development-related positions for Siemens AG and Schaeffler AG.

His hobbies and private activities are mixed to share practical knowledge on topics like home automation driven by MQTT protocol or using open-source CNC controllers for machine tools.