This post offers an introduction to the MQTT (Message Queuing Telemetry Transport) protocol [1] and also demonstrates its usage with an example in Python (Just for info: telemetry means the collection of measurement data from a remote location and its transmission wiki-link.
What is MQTT?
- The MQTT protocol is lightweight, efficient, and contains only a small footprint. This makes it a perfect match for IoT devices and scenarios which are often running within unstable environments.
- I have to write a python program using MQTT protocol to subscribe topics on thingspeak.com and display it on the screen of a Raspberry Pi. I found official help/documentation about publishing messages and field feeds to thingspeak.com, but they don't provide any information about how to subscribe a topic, e.g. What is the form of a topic string, what are the inclusions of that topic string etc.
- Umqtt.simple2 is a MQTT client for MicroPython. (Note that it uses some MicroPython shortcuts and doesn't work with CPython). Support MQTT Version 3.1.1 only. It certainly works with micropython ports: esp8266 and esp32.
If you dig into MQTT, there's not all that much there — it's a very lightweight transfer protocol. The 'M' is 'minimal' after all. But it is one more layer on top of raw UDP (and. MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks.
MQTT is a communication protocol which is actually developed for low-energy devices to transmit data with the low-bandwidth network.
It is a lightweight protocol built on the top of TCP/IP.
Lightweight: In software/programs, lightweight specify the characteristic of low memory and CPU usage.
MQTT allows simple and efficient data transmission from sensors to IoT (Internet of Things) networks. Additionally, it also enables the easy integration of new sensors to the IoT network.
MQTT Terminology
Eclipse Paho
In MQTT, there are five main terminology
- Broker: It is the middleware between data sender (publisher) and data receiver (subscriber). You can consider it as a server which receives data from sender and forward it to the receiver.
- Publisher: In simple terms, you can think of a device which needs to send data to other parties. This device could be a sensor, laptop, and Raspberry pi board.
- Subscriber: It is the receiver of data sent by the publisher.
- Topic: Publisher and Subscriber do not know each other directly. When the publisher sends some data to Broker, it associates that data with a
topic. The topic is a string in the format of hierarchy e.g.sensor/room-1/temprature(it is just an example) where ‘/' is level separator and each string represents a level. You can define it as per your requirement. When the subscriber connects to the Broker, it specifies the topic in which it is interested in. Then, broker forward published messages to their corresponding subscribers on the basis of the topic. Here, you need to understand two operators+and#. Before jumping to these operators, let's assume a scenario where we have four temperature sensors in four rooms. These sensors are sending their data with the following topics
- home/bedroom/temperature
- home/kitchen/temperature
- home/dining-room/temperature
- home/study-room/temperatureNow, in order to receive data from all four rooms, subscribers need to subscribe to the above topics. One way of doing that is to subscribe to each topic separately. Another way is to use
+operator and simply subscribe to the topichome+temprature.+operator here matches any single level (+is known as a single-level wildcard). Therefore,home+tempraturewill match any topic with three-level hierarchy starting withhomeand ending withtemprature.
Coming to second operator#which matches multi-level in the topic. For instance,home#will match all the aforementioned four topics.Some examples
Let's say we have multiple sensors which are transmitting data with following topics
- schoolclass-1group-1audio
- schoolclass-1group-2audio
- schoolclass-2group-1audio
- schoolclass-2group-2audioWhich topic to subscribe to receive all data from class-1 ?
schoolclass-1#Which topic to subscribe to receive data from group-1?school+group-1audioWhich topic to subscribe to receive data from entire school?school#
# operator can appear only once in a topic expression e.g. Connect lg tv to laptop wireless. school#group-1# is invalid.
A topic expression # matches every topic hence subscriber to this topic will receive every message.
- Message: It is simply the data which needs to be sent.
Demonstration of MQTT in Python
Now, we will see a python example of using MQTT protocol for transmitting data. We will use a python package [paho-mqtt](https://pypi.org/project/paho-mqtt/ ' target = '_blank) for creating publisher and subcriber. Anime drawing. You can install it using following command
Next, we need to set up a Broker. There are numerous option of this. You can either use a cloud-based broker or you can install a broker on a server in your network. There are multiple cloud services are available with can be used ([Complete list of broker servers](https://github.com/mqtt/mqtt.github.io/wiki/servers ' target = '_blank)). I would like to mention [MaqiaTTo](https://www.maqiatto.com ' target = '_blank) which is a free cloud-based MQTT broker. It can be used for testing purpose. In this tutorial, however, we are going to set-up a broker in the network. We will use [Mosquitto](https://mosquitto.org/ ' target = '_blank) broker server.
Setting up Mosquitto Broker
Mosquitto is an open-source MQTT message broker. Following are the instructions for installing it on your systems.
Madame butterfly story. Madama Butterfly, opera in three acts (originally two acts) by Italian composer Giacomo Puccini (Italian libretto by Luigi Illica and Giuseppe Giacosa) that premiered at La Scala opera house in Milan on February 17, 1904. The work is one of the most frequently performed of all operas. Since Madama Butterfly has more stage time than other sopranos, it is imperative that the singing actress captures extensive dramatic variety in addition to singing with exquisite tone and an affinity to Puccini. Expect a superlative rendering (at an unbelievably low price) of Puccini's Madama Butterfly. 7 people found this helpful. Amazon Customer. 5.0 out of 5 stars Five Stars. Reviewed in the United States on December 19, 2017. Verified Purchase. Arrived on time and is a very good recording of excellent voices.
- Windows users: Download and install 64bit-version32bit-version (If you are not sure then install 32bit-version)
- Ubuntu Users: Run following commands
Python Mqtt Ssl
- Mac Users: First install brew package manager using following command
Next, install Mosquitto
Starting Mosquitto Server (Mac Users)Now, we are going to start our MQTT broker.
Run the following command
On ubuntu following command can be used to start or stop Mosquitto
sudo systemctl (start|stop) mosquitto
Writing Publisher and Subscriber codes
Now, we will develop our publisher and subscriber. Following are the steps for developing publisher and subscriber in Python.
- Create client
- Connect to broker
- Publish/subscribe message
- Connect callback function
- Run the loop
- Disconnect
Let's understand these steps in details. The first step is to create a client. For this, we will create a Client object from paho-MQTT python package. The second step connects to the broker. In our case, we are running the broker on the same machine, therefore, we specify the broker's IP as 127.0.0.1 and port as 1883 (default port for Mosquitto broker). In third step, you can publish or subcribe using publish() and subscribe() function. Next, we need to write callback functions.
It needs a bit of explanation. Callback functions are functions which are executed on the occurrence of particular events. Followings are the table showing the event and their corresponding callback function
Let's take one example to understand it. When a client connects to the broker, the broker sends an ACK (acknowledgment) to the client. This event triggers the execution of a callback function on_connect.
Running a loop: Why we need to start a loop?
When a publisher sends messages to the broker or subscriber receive messages from the broker, these messages are first stored in the buffer. Now, in order to process all messages (either for sending or receiving), we need to write a loop manually. Thanks to paho-MQTT, it provides three functions for the same purpose, therefore, we don't need to write message processing loop. These functions are as following
- loop(): When you call this function, it will process any pending message sending or receiving action. This function waits for a particular time (you can specify using
timeoutparameter) for processing buffer for reading or sending a message. After, that its execution completes. Therefore, if you plan to use this function you need to call it regularly. - loop_forever(): This function call results in indefinite execution of your program. This function automatically reconnects to the broker in case of disconnection. This function is blocking type function (you can understand it as an infinite for loop) and it returns when you
disconnectwith the broker. - loop_start() & loop_stop():
loop_start()function starts a new background thread and that thread regularly executeloop()function. You can stop this background thread usingloop_stop()function.
As we have setup our broker server, now we move towards writing publisher code.
If you dig into MQTT, there's not all that much there — it's a very lightweight transfer protocol. The 'M' is 'minimal' after all. But it is one more layer on top of raw UDP (and. MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks.
MQTT is a communication protocol which is actually developed for low-energy devices to transmit data with the low-bandwidth network.
It is a lightweight protocol built on the top of TCP/IP.
Lightweight: In software/programs, lightweight specify the characteristic of low memory and CPU usage.
MQTT allows simple and efficient data transmission from sensors to IoT (Internet of Things) networks. Additionally, it also enables the easy integration of new sensors to the IoT network.
MQTT Terminology
Eclipse Paho
In MQTT, there are five main terminology
- Broker: It is the middleware between data sender (publisher) and data receiver (subscriber). You can consider it as a server which receives data from sender and forward it to the receiver.
- Publisher: In simple terms, you can think of a device which needs to send data to other parties. This device could be a sensor, laptop, and Raspberry pi board.
- Subscriber: It is the receiver of data sent by the publisher.
- Topic: Publisher and Subscriber do not know each other directly. When the publisher sends some data to Broker, it associates that data with a
topic. The topic is a string in the format of hierarchy e.g.sensor/room-1/temprature(it is just an example) where ‘/' is level separator and each string represents a level. You can define it as per your requirement. When the subscriber connects to the Broker, it specifies the topic in which it is interested in. Then, broker forward published messages to their corresponding subscribers on the basis of the topic. Here, you need to understand two operators+and#. Before jumping to these operators, let's assume a scenario where we have four temperature sensors in four rooms. These sensors are sending their data with the following topics
- home/bedroom/temperature
- home/kitchen/temperature
- home/dining-room/temperature
- home/study-room/temperatureNow, in order to receive data from all four rooms, subscribers need to subscribe to the above topics. One way of doing that is to subscribe to each topic separately. Another way is to use
+operator and simply subscribe to the topichome+temprature.+operator here matches any single level (+is known as a single-level wildcard). Therefore,home+tempraturewill match any topic with three-level hierarchy starting withhomeand ending withtemprature.
Coming to second operator#which matches multi-level in the topic. For instance,home#will match all the aforementioned four topics.Some examples
Let's say we have multiple sensors which are transmitting data with following topics
- schoolclass-1group-1audio
- schoolclass-1group-2audio
- schoolclass-2group-1audio
- schoolclass-2group-2audioWhich topic to subscribe to receive all data from class-1 ?
schoolclass-1#Which topic to subscribe to receive data from group-1?school+group-1audioWhich topic to subscribe to receive data from entire school?school#
# operator can appear only once in a topic expression e.g. Connect lg tv to laptop wireless. school#group-1# is invalid.
A topic expression # matches every topic hence subscriber to this topic will receive every message.
- Message: It is simply the data which needs to be sent.
Demonstration of MQTT in Python
Now, we will see a python example of using MQTT protocol for transmitting data. We will use a python package [paho-mqtt](https://pypi.org/project/paho-mqtt/ ' target = '_blank) for creating publisher and subcriber. Anime drawing. You can install it using following command
Next, we need to set up a Broker. There are numerous option of this. You can either use a cloud-based broker or you can install a broker on a server in your network. There are multiple cloud services are available with can be used ([Complete list of broker servers](https://github.com/mqtt/mqtt.github.io/wiki/servers ' target = '_blank)). I would like to mention [MaqiaTTo](https://www.maqiatto.com ' target = '_blank) which is a free cloud-based MQTT broker. It can be used for testing purpose. In this tutorial, however, we are going to set-up a broker in the network. We will use [Mosquitto](https://mosquitto.org/ ' target = '_blank) broker server.
Setting up Mosquitto Broker
Mosquitto is an open-source MQTT message broker. Following are the instructions for installing it on your systems.
Madame butterfly story. Madama Butterfly, opera in three acts (originally two acts) by Italian composer Giacomo Puccini (Italian libretto by Luigi Illica and Giuseppe Giacosa) that premiered at La Scala opera house in Milan on February 17, 1904. The work is one of the most frequently performed of all operas. Since Madama Butterfly has more stage time than other sopranos, it is imperative that the singing actress captures extensive dramatic variety in addition to singing with exquisite tone and an affinity to Puccini. Expect a superlative rendering (at an unbelievably low price) of Puccini's Madama Butterfly. 7 people found this helpful. Amazon Customer. 5.0 out of 5 stars Five Stars. Reviewed in the United States on December 19, 2017. Verified Purchase. Arrived on time and is a very good recording of excellent voices.
- Windows users: Download and install 64bit-version32bit-version (If you are not sure then install 32bit-version)
- Ubuntu Users: Run following commands
Python Mqtt Ssl
- Mac Users: First install brew package manager using following command
Next, install Mosquitto
Starting Mosquitto Server (Mac Users)Now, we are going to start our MQTT broker.
Run the following command
On ubuntu following command can be used to start or stop Mosquitto
sudo systemctl (start|stop) mosquitto
Writing Publisher and Subscriber codes
Now, we will develop our publisher and subscriber. Following are the steps for developing publisher and subscriber in Python.
- Create client
- Connect to broker
- Publish/subscribe message
- Connect callback function
- Run the loop
- Disconnect
Let's understand these steps in details. The first step is to create a client. For this, we will create a Client object from paho-MQTT python package. The second step connects to the broker. In our case, we are running the broker on the same machine, therefore, we specify the broker's IP as 127.0.0.1 and port as 1883 (default port for Mosquitto broker). In third step, you can publish or subcribe using publish() and subscribe() function. Next, we need to write callback functions.
It needs a bit of explanation. Callback functions are functions which are executed on the occurrence of particular events. Followings are the table showing the event and their corresponding callback function
Let's take one example to understand it. When a client connects to the broker, the broker sends an ACK (acknowledgment) to the client. This event triggers the execution of a callback function on_connect.
Running a loop: Why we need to start a loop?
When a publisher sends messages to the broker or subscriber receive messages from the broker, these messages are first stored in the buffer. Now, in order to process all messages (either for sending or receiving), we need to write a loop manually. Thanks to paho-MQTT, it provides three functions for the same purpose, therefore, we don't need to write message processing loop. These functions are as following
- loop(): When you call this function, it will process any pending message sending or receiving action. This function waits for a particular time (you can specify using
timeoutparameter) for processing buffer for reading or sending a message. After, that its execution completes. Therefore, if you plan to use this function you need to call it regularly. - loop_forever(): This function call results in indefinite execution of your program. This function automatically reconnects to the broker in case of disconnection. This function is blocking type function (you can understand it as an infinite for loop) and it returns when you
disconnectwith the broker. - loop_start() & loop_stop():
loop_start()function starts a new background thread and that thread regularly executeloop()function. You can stop this background thread usingloop_stop()function.
As we have setup our broker server, now we move towards writing publisher code.
The above code publishes a message on with topic houselight. In this code, we have written two functions connect_msg and publish_msg (you can use any names for functions). We connected these functions to callback functions using client.on_connect = connect_msg. We specified that connect_msg function is callback function and it will be called when the Connection ACK event occurs (events and their callbacks are given in the above table). In this program, we used the loop function which process pending action (sending or receiving) and then returns.
As we have our publisher with topic houseligh topic, we now develop our subscriber for the same topic.
Execution Results
References
Python Mqtt Api
- MQTT Version 5.0. Edited by Andrew Banks, Ed Briggs, Ken Borgendale, and Rahul Gupta. 07 March 2019. OASIS Standard. https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html. Latest version: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html.
