If you are new to IoT or have encountered MQTT for the first time,
this blog post will help you
understand what MQTT is, how it works, its components,
and how it compares to better known protocols like HTTP.
By Vit Prajzler
MQTT is a lightweight networking protocol, designed to efficiently transport telemetry data between multiple producers and multiple consumers.
A protocol, like MQTT, is a set of well-defined rules on how to format messages over a computer network. The rules include both the format and the meaning of the messages, and how to use those messages to communicate information.
In simple terms, MQTT is a recipe for writing and reading telemetry messages over a network.
MQTT is an application layer protocol (in the OSI stack), so let's start with a comparison to another famous application layer protocol: HTTP.
Unlike HTTP (Hyper-Text Transport Protocol) which was designed to transport hyper-text (aka, web pages), MQTT (Message Queueing Telemetry Transport) was designed to handle small queue-able packets of telemetry data. What is a little confusing is that both protocols are called “transport protocols”, but they are actually not transport layer protocols in the sense of the layer stack of the OSI model. They are application layer protocols. Naming is hard.
So while HTTP was built for a request-response mode of interaction (a web browser retrieving a web page from a server), MQTT was built to report telemetry from multiple sources, and receive telemetry at multiple destinations in a publish-subscribe model.
In practice, the major difference is that an MQTT client can receive notifications (telemetry) from the server (broker) by only asking once, whereas an HTTP client has to ask the server if there is any data, periodically. An HTTP client would be polling (asking the server continuously for more data), whereas an MQTT client would subscribe to a topic once, and then receive all subsequent data without polling (asking).
In HTTP, a client makes a request to a URL (such as GET /index.html) on a server (such as
www.avoguard.com), and waits until it gets a response (such as 200 OK, along
with the page content).
In MQTT, there are no requests and no responses. Instead, a client connects to a broker (such as
broker.avoguard.com), subscribes to a set of topics (such as
/avoguard/device/id-123/configuration), and once it has collected some telemetry from its
“physical” sensors, it publishes the telemetry to another set of topics (such as
/avoguard/device/id-123/status).
A HTTP client operates in (outgoing) transactions.
An MQTT client operates in incoming and outgoing events.
Since I mentioned topics, let's dive into what they are and how they compare to URLs.
The role of an URL in HTTP is very similar to the role of a topic in MQTT.
Without a topic, every MQTT subscriber would receive all the data from every MQTT publisher. Without a URL, an HTTP client could not ask for a specific resource, it would always get a dump of all the server data. While there might be cases in which that would be good enough, most systems do benefit from the extra layer of separation.
You can think of a topic as a virtual network that only selected publishers and subscribers are part of.
Having addressed the topics, I ran out of similarities between the protocols, so let's shift focus to the components of an MQTT architecture.
To accommodate the multiple sources and destinations of telemetry data within a single client (imagine an embedded device with a bunch of different sensors), an MQTT client can assume one of two roles: a publisher or a subscriber. It can also be both.
All MQTT clients connect to the same central entity (“server”) that manages the distribution of the telemetry data. That is the MQTT broker.
To facilitate the data flow from the MQTT Publishers to the MQTT Subscribers, a lot of coordination is required. In order to avoid having all the publishers keeping a list of all of their subscribers, and all subscribers having to tell each individual publisher which topics they are interested in, MQTT delegates the coordination role to the MQTT broker.
The broker is the key component that limits the complexity of the clients (both subscribers and the publishers), by completely separating their respective domains. An MQTT client only needs to know the broker address, and the names of the topics it is interested in. It doesn't need to know anything about publishers or subscribers.
The main (and pretty much one and only) role of the broker is making sure all the data from the publishers gets delivered to the subscribers. That includes for example duplicating messages if there are multiple subscribers to the same topic.
A publisher is a creator of messages. It has telemetry data that it places into an MQTT packet, and sends it to a broker. It doesn't care where the message is going to eventually end up. It kind-of says, FYI, this is my latest data, if you're interested. If not, that's fine, just throw the data away.
Unlike an HTTP client, an MQTT publisher does not wait for a “response” to the MQTT packet (“request”) it just sent. And since it doesn't have this burden, its software can be much simpler, and the rate at which it can publish messages can be higher.
Note that not waiting for responses does not mean the messages get lost along the way. There are mechanisms (QoS) that allow for confirmed reception on the MQTT layer. But even without the confirmations, MQTT sits on top of TCP, which is responsible for reliable delivery.
A subscriber is a silent listener. It connects to a broker and tells the broker which type of packets (topics) it is interested in. The broker maintains a list of the topics of interest, and when a packet arrives from an MQTT Publisher which topic matches one on the list of interests, it forwards it to the subscriber.
Subscriber just sits there and listens.
Why don't we just use HTTP everywhere if it's already used all across the Internet for delivery of web pages? Fair point. HTTP is a successful, versatile protocol that has been adapted for many use-cases beyond its initial scope of purpose.
MQTT is not as versatile, and therefore not as widely used. But it is a great fit for a specific purpose. MQTT has become the go-to protocol for Internet of Things applications, and beyond. It is also used in cloud infrastructure as a well-defined low-latency interface between services.
In short, what MQTT can do better than HTTP is low-latency delivery of large volumes of small messages, with very little overhead.
If you are interested in more technical advantages of MQTT over HTTP, continue reading.
Unlike the transactional short-lived retrieval of a web page, typical MQTT sessions last hours, days, or weeks. On a reliable network with reliable devices, an MQTT session could last forever. The session itself establishes a pipe (much like TCP) over which packets can be sent with very little overhead.
An MQTT publish packet (one that a publisher would use to send data to subscribers) has 2 bytes of fixed header, topic name + its length, and then the packet payload. For short topic names, the overhead is 4 bytes. HTTP needs 4 bytes just to tell the other side it is an HTTP connection (by literally sending the letters “HTTP” to the other side), where the complete HTTP header is at least 20 bytes long.
If you have a lot of small packets to send, the low MQTT overhead makes a big difference.
Having to wait for a response to a previous message before sending another one or re-establishing a connection for every request adds delays. In use-cases where latency is critical, MQTT is a great choice. The non-blocking nature of the protocol, a push architecture rather than pull, along with long lived connections limits delays in delivery of MQTT packets.
Unlike the request-response model of HTTP, the publish-subscribe model of MQTT is designed for asynchronous communication. That means that subscribers do not need to provide immediate response to publishers, which in turn enables more flexible communication patterns.
Subscribers also don't need to actively request incoming packets, they can just subscribe and wait for the delivery to happen.
Not all of your data needs to be acknowledged, sometimes you simply might not care if one message gets lost. Other times, you might want absolute guarantee that a message has reached its destination once, and only once (no duplicates e.g. from retransmissions).
MQTT features adjustable Quality of Service (QoS) levels, ranging from fire-and-forget, at least once, to exactly once delivery, in increasing order of complexity and overhead.
MQTT is not a complex protocol. It is simple enough to be implemented in resource constrained devices, and the client implementation can easily fit in the memory of even the smallest chips (think firmware rather than embedded linux).
It wouldn't be me if I didn't touch upon the security of MQTT. Just like HTTP, MQTT was conceived when the Internet was a safer place. Encryption was not omnipresent, and if you wanted to make it a little harder for unauthorized people to access your system, you'd make them use a username and password - over a plaintext connection.
We have come a long way since then, but you see the foundation is still similar. Nowadays, MQTT is typically used with TLS (Transport Layer Security, the successor of SSL) in the same way that most web pages are using it. In other words, MQTTS is just like HTTPS.
The one difference I see in the field is that more and more MQTT deployments are not blindly trusting the secure connections, but are actually using client authentication.
The client authentication in MQTT can come in different forms. Some people opt for using usernames (device serial numbers) and passwords (basically pre-shared keys), while others delegate authentication to the transport layer (TLS), where they can use generic identity providers. On the TLS layer, the client authentication can be done using PSK (pre-shared keys) or using client certificates (private / public keys).
In summary, MQTT on top of TLS is using the same level of security as a usual web application. The minor difference is that in a web application, typically the client (browser) authenticates the server, but the server does not authenticate the client. In MQTT, it is more common for the server to require client authentication.
At Avoguard, we realize that the worlds of the cloud / web and the Internet of Things are very different. While MQTT is very popular in IoT applications, the cloud and web infrastructure is built with HTTP.
Avoguard's Gateway makes it possible to bridge the two worlds, without forcing cloud engineers to learn MQTT or firmware engineers to use HTTP. With MQTT gateway, both teams can leverage their know-how and protocols that are best suited for the problem they are solving for your business.
MQTT and HTTP are both application layer protocols. They are both used to send messages between two endpoints over the internet. But they are used in different scenarios, and have different strengths and weaknesses.
Protocol Comparison |
HTTP |
MQTT |
|---|---|---|
| Primary purpose | Fetching of web pages | Transport of telemetry |
| Typical session duration | Short (seconds) | Long (days or weeks) |
| Protocol design | Text-stream oriented | Event oriented |
| Usage pattern | Request / Response | Publish / Subscribe |
| Push notifications | Complicated, inefficient | Simple, efficient |
| Basic security | Username / Password | Username / Password |
| Security | TLS | TLS |
If you have more questions about how to use MQTT in your IoT projects, please contact us.
If you're ready to use or evaluate the Avoguard Gateway, please contact us to get a quote or a demo.
A typical MQTT usage model / architecture is very different from HTTP. MQTT is not focused on requests and responses. There is no single “server” serving requests of clients. You might still be inclined to call the broker a server, fair enough - it's close. But the packets from the clients are merely passed through the broker, more like a proxy than a server.
In the end, you can call the broker a server, and all the subscribers and publishers clients.
MQTT is just a mechanism for delivery of telemetry. It needs to be adapted to your business logic. Most frequently, that means you need to design your own topic schema, which will define how effectively your data will flow from publishers to subscribers.
Security in MQTT is limited to usernames and passwords, so I warmly recommend using TLS to secure your connections.
Scaling out an MQTT infrastructure is something that the protocol is ready for, but it doesn't prescribe how you should do it. So if you're planning on horizontally scaling an MQTT deployment, you need to choose the right client and broker implementation.
Designing an efficient topic hierarchy is one of the most important aspects of a successful MQTT deployment.
It is similar to picking the right directory structure for organizing your files, or picking the right database schema to represent your data. You can certainly change it later, but the applications that depend on it will need to be updated, too.
A good schema will be future proof enough to accommodate for changes in functionality without a need for change in existing topic schema.
You would certainly want MQTT to work over any kind of network - Wi-Fi, Ethernet, Cellular. Or maybe avian carriers like ravens or pigeons. In order to accommodate the complexity of all the options and combinations, network protocols are built in layers (called OSI model).
Each layer is responsible for a limited, well-defined part of the networking system. The lowest layer is the physical layer (which includes an ethernet cable, telephone cable, or a radio frequency band). On top of this foundation, there are a number of other layers that serve various purposes (like converting bits to the physical layer format, avoiding crosstalk, congestion, guaranteeing delivery, order of messages, etc…). They all stack on top of each other. Frequently you can change the protocols you use within a specific layer, while keeping the other layers in place. It's neat.
Contact us to get a quote or book a demo.