Dev guides

Step-by-Step Tutorial on Enabling MQTT on ESP32 Module

Andreas Schiffler10.11.202216.12.2023

Using small and smart IoT devices is common nowadays. Technologies such as Bluetooth Low Energy (BLE) and Long Range radio technology (LoRa, LoRaWAN) combined with internet access enable applications like e-scooter location advertising, parking slot monitoring, and indoor navigation. Such applications commonly use highly integrated energy-efficient microcontrollers (µC) or systems on a chip (SoC). Candidates for such tasks are the ESP32 series microcontrollers with integrated Wifi/Bluetooth modules on the chip.

Besides using ESP32 µCs as hardware, another common approach is to use an MQTT broker with high availability for communication and information exchange, like Pro Edition for Eclipse Mosquitto.

Therefore, this article describes the use of professional and reliable MQTT communication between an ESP32 module (ESP32 IoT board with ethernet) and the MQTT broker. It addresses:

• MQTT protocol v5.0
• TLS encryption
• WebSocket transport
• Auto reconnect functionality

Quickstart using MQTT on ESP32 with WebSocket and TLS

To begin with, a sample application is available as a pre-built firmware and can be directly flashed on your ESP32 module. It runs perfectly in combination with the 14-day-trial of Pro Edition for Eclipse Mosquitto.

1. Head to https://aschiffler.github.io/esp32-mqtt-node/ 
2. Connect your ESP32 module via a USB cable to the PC running the internet browser showing the webpage from 1.
3. Click the button “flash the firmware to your device.”
4. Choose the correct serial port (mostly showing text like: “USB to UART bridge”) and click connect.
5. Click “INSTALL…” and confirm the installation.
6. After installation, click “Next” and then (same dialog as above) click “LOGS & CONSOLE.”

To illustrate, now you see the application running and waiting for your input.

At this stage, you need to input the following information:

1. Wifi connection: Enter the wifi SSID and password in one line like “mywifi mypass” (only WPA2 PSK is supported with the pre-built firmware).
2. After enabling the connection to wifi/internet, the application tries to contact an NTP time server to get a synchronized date/time.
3. MQTT Broker connection string: Enter the MQTT broker in one of the options, as you can find in the table below.
4. MQTT protocol version: Type ‘y’ for using protocol version v5.0.
5. Client/User credentials: Enter username and password (This client shall be able to publish and subscribe on the topics: “topic/qos1” and “topic/qos0” on the MQTT broker).

Possible MQTT broker connection strings

The output looks like the following image. If all goes right, the ESP32 module publishes a message “Cedalo is awesome” on the topic “topic/qos0.”

Now you can use any other MQTT client and publish the message “time” on “topic/qos1” to view the response from your ESP32 module on the topic “topic/qos0” and also on the console log. This prints/sends the actual date and time from the ESP32.

In the picture above, you can also see that the MQTT client automatically tries to reconnect after a connection loss. This was provoked by disabling the wifi connection on the access point.

If this quick start fails, navigate to the Possible Issues section. Alternatively, read the following section on the details about the firmware and the MQTT client on ESP32.

Software needed to use MQTT on ESP32

The nature of microcontrollers is that you need a firmware that will be flashed on the device and, therefore, realize the application itself. In most cases, the ESP32 modules you can find on the internet have already installed bootloader firmware. The bootloader lets the user flash the custom firmware in the memory over a USB connection with a PC.

As you can see in the table below, several options exist for building the application firmware for ESP32.

Because this article focuses on MQTT, we need to look more closely at the existing MQTT libraries or options for using ESP32 µCs.

Using micro Python on ESP32 is becoming increasingly popular, but the well-known PAHO MQTT client can’t run on embedded micro Python.

As I mentioned in the introduction to this blog post, ESP32 µCs is often used in mobile applications. Since network connectivity is a challenge, one should consider unknown network routes and firewall rules when implementing this type of a project. Therefore, the favorite approach is choosing a “bidirectional HTTP protocol” known as WebSocket. This is because, in most cases, the transport of the MQTT payloads over HTTP/WebSocket will show the most incredible robustness in transmitting information on unknown routes. Check out this blog post to learn more about enabling WebSockets for Mosquitto.

Reading the above information and facing the requirements also clarifies that we have to choose the Espressif IoT Development Framework (ESP-IDF) for the development. In short, the ESP-IDF is comprehensive, covers all technological topics, and uses C/C++, which makes the development more challenging from the initial point of view. However, do not worry, as there are many examples in the public repository that will help you a lot.

As you can see, you can use this article, together with the prepared source code and pre-built firmware, to get a demonstrative MQTT Application. Therefore, you can follow this link (GitHub account needed) and start a ready-to-use development environment in your internet browser.

Example Application for MQTT on ESP32

The following sections explain building a basic program that enables the ESP32 to connect securely to the Mosquitto MQTT broker to publish and subscribe to topics.

Additionally, the sample application will have a simple functionality. Every time a message is published to a specific topic, the ESP32 will publish the actual time, which is synchronized via NTP to another topic.

Requirements for the MQTT Broker

As a result, to make use of this functionality, you will require the following:

• Firstly, a running MQTT broker and the address (ipv4 or DNS name).
• Secondly, the credentials (username/password) to publish and subscribe to topics.
• Lastly, the X509 certificate of the broker (Only needed if the issuer CA is other than the well-known R3 Let’s encrypt).

Overall, all the above requirements can be easily fulfilled by Pro Mosquitto from Cedalo. Furthermore, see the tutorial here on how to set it up. Sign up for a 14-day free trial and, within a few minutes, get a free version of a professional and secure Eclipse Mosquitto MQTT broker with an intuitive WebUI setup.

The source code for an MQTT client on ESP32

The source code for the sample application I described in this article is here. Also, it uses the ESP-IDF SDK for building the firmware.

In brief, editing and adjusting the code will start mostly in the file main/app_main.c

The relevant structure of app_main.c is as follows. The code is presented as a simplified excerpt, showing the most important parts that relate to MQTT.

Includes (excerpt, MQTT, TLS, NTP)

#include "mqtt_client.h"
#include "esp_tls.h"
#include "esp_sntp.h"

The function app_main will start the network connection, the NTP synchronization, and finally, in line 4, the MQTT client.

1 | void app_main(void){
..
2 |     ESP_ERROR_CHECK(network_connect());
3 |     ESP_ERROR_CHECK(sntp());
4 |     ESP_ERROR_CHECK(mqtt_start());
...
5 | }

After that, the function sntp sets the time servers and starts the synchronization task.

6 | esp_err_t sntp(void){
...
7 |  sntp_setservername(0, "0.de.pool.ntp.org");
8 |  sntp_setservername(1, "1.de.pool.ntp.org");
9 |  sntp_setservername(2, "2.de.pool.ntp.org");
10|  sntp_init();
...
11| }

As a matter of fact, this function – mqtt_start – is one core component in this sample application. The struct in lines 13-17 defines some mandatory settings for the clients (see here for reference).

Furthermore, in lines 18-23, some of the MQTT v5.0 specific values are set. These settings apply only if, in line 14, the value will be set to MQTT_PROTOCOL_V_5. Moreover, you can read the article here for some more information on using protocol v5.0.

Later, in line 25, the client instance handle is created, and the settings are applied. The function call registers the callback function in line 30, and finally, the MQTT loop task is started in line 31.

12| esp_err_t mqtt_start(void){
...
13|   esp_mqtt_client_config_t mqtt_cfg = {
14|    .session.protocol_ver = MQTT_PROTOCOL_V_3_1_1
15|    .broker.address.uri = “mqtts://your-borker.cedalo.cloud”
16|    .credentials.username = “user”
16|    .credentials.authentication.password = “password”
17|   };
18|   esp_mqtt5_connection_property_config_t connect_property = {
19|    .session_expiry_interval = 10,
20|    .request_resp_info = true,
21|    .request_problem_info = true,
22|    .message_expiry_interval = 10,
23|    .response_topic = "topic/response",
24|   };
...
25|   esp_mqtt_client_handle_t client = esp_mqtt_client_init(&mqtt_cfg);
26|    if (mqtt_v5){
27|     esp_mqtt5_client_set_connect_property(client, &connect_property);
28|     ESP_LOGI(TAG,"Using MQTT protocol version 5.0");
29|   }
30|   esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID,...);
31|   esp_mqtt_client_start(client);
...
32| }

The mqtt_event_handler function implements the main functionality of the sample application. In the first place, the function handles different events by looking at the event id (line 37).

Therefore, there are two events in the code excerpt. After the client has established a connection to the broker, the client subscribes to the topic “topic/qos1” with the Quality of Service level 1 (line 40). Read this article to learn more about MQTT QoS levels.

If the broker successfully registers the subscription and reports to the client, the event in line 42 will be handled. In the sample application, this event will also trigger the client to publish a message to the topic “topic/qos0” with Quality of Service level 0, lines 43 and 44 (see this reference about the function call parameters).

Finally, every time a message is received on one of the subscribed topics – wildcards like “topic/#” are possible – in lines 46 and 47, the message (= event->data) is printed on the console.

33| static void mqtt_event_handler(...){
...
34|  esp_mqtt_event_handle_t event = event_data;
35|  esp_mqtt_client_handle_t client = event->client;
36|  int msg_id;
37|  switch ((esp_mqtt_event_id_t)event_id) {
38|   case MQTT_EVENT_CONNECTED:
39|     ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");
40|     msg_id = esp_mqtt_client_subscribe(client, "topic/qos1", 1);
41|     break;
42|   case MQTT_EVENT_SUBSCRIBED:
43|     msg_id = esp_mqtt_client_publish(
44|          client, "topic/qos0", "Cedalo is awesome", 0, 0, 0);
45|     break;
46|   case MQTT_EVENT_DATA:
47|     ESP_LOGI(TAG,"Received message: %s",event->data);
48|     break;
49|  }
...
50| }

Build your own ESP32 MQTT application

The easiest way to customize or create your application is to use the ready-to-code environment provided here. Also, this will start a gitpod instance (GitHub account needed) with all the tools and libraries already setup.

By executing the command idf.py menuconfig in the terminal, a menu configuration dialog opens, and you can set various options for the application or the components. For example, you can disable the certificate verification by default for testing purposes. Therefore, navigate the menu to Component Config > ESP-TLS > Allow insecure options > Skip verification and enable this option. Then hit “Q” and “Y” to exit and save.

To compile the code and build the application (firmware), type idf.py build in the terminal and wait until finished.

You can now flash the built firmware by opening the web-flash tool. Therefore click the tab “Ports” in the lower window part (see image below) and click the URL on the page.

In addition, you can find the procedure with the modified firmware in the Quickstart section.

However, if you would like to code not in the browser and use a local IDE setup, check out the docker file in the repository, which shows the list of mandatory packages. Alternatively, follow the official documentation for ESP-IDF.

Possible Issues and Troubleshooting

Quick-Start, Using the MQTT Application on ESP32 issues

Issue No. 1: The console shows connection timeout for wifi or MQTT broker.

Reason:

• The ESP32 module can not access the broker via wifi, or the wifi router does not assign an IP address via dhcp; wifi router has no route to the broker (no internet connection).

Solutions:

• Restart the ESP32 by clicking restart in the console, or make a power cycle by unplugging and plugging in the USB cable. 
• Type in the correct credentials for wifi and MQTT broker.
• Establish a network route to the MQTT broker.

Issue No. 2: The console shows esp-tls-errors.

Reasons: 

• The provided certificate from the MQTT broker is not valid.
• The provided certificate from the MQTT broker can not be verified.

Solutions:

• Check with other tools if the certificate is valid (self-signed certificates are always invalid). If not, disable verification (see section before).
• Check the certificate in the file main/mqtt_broker_cert.pem if this matches the issuer CA (default R3 Let’s encrypt) of your MQTT broker. If not, replace the mqtt_broker_cert.pem with the correct one.
• Restart ESP32 module by power cycle.

Issue No. 3: MQTT connection errors/timeout.

Reasons:

• Wrong user credentials.
• Protocol Version not supported.
• Port (e.g., 8883, 1883) can not be reached by the ESP32 client.

Solutions:

• Check/adjust credentials.
• Test connection to specific ports with other wifi clients in the same network.
• Use Protocol version 3.1.1.
• Restart ESP32 module by power cycle.

Build the application/firmware issues

Issue No. 1: Compilation fails.

Reason:

• Syntax errors, file not found, type cast…

Solutions:

• Investigate on given lines.
• Search the internet for similar problems.

Issue No. 2: Command idf.py was not found.

Reason:

• Rate limits and idle throttling of gitpod instance.

Solution:

• Restart the gitpod instance (Code changes will be preserved).

Issue No.3: Web Flash Tool is not available, shows “manifest not found.”

Reasons:

• The binary files (firmware) are not available.
• The web-flash-tool is not served.

Solutions:

• Build the firmware with the command idf.py build.
• Download the firmware from the build folder and use another flash tool.

About the author

Andreas Schiffler

Research Professor at the Technical University of Wuerzburg-Schweinfurt

Dr. Andreas Schiffler is a research professor at the Technical University of Wuerzburg-Schweinfurt in the field of production and data technology in mechanical engineering. In addition to research topics related to 3D metal printing, Dr. Schiffler developed a Kubernetes cluster for the practice-oriented basics of IoT and Industry 4.0 as part of the student training. Before joining the university, he worked in different product development-related positions for Siemens AG and Schaeffler AG.

His hobbies and private activities are mixed to share practical knowledge on topics like home automation driven by MQTT protocol or using open-source CNC controllers for machine tools.