Avoguard blog

Real power of MQTT - wildcards

How can MQTT be optimally used to implement a Unified Namespace (UNS)?
Discover the difference between an MQTT topic, MQTT topic level, and MQTT topic filter.
Learn how they are used by MQTT publishers and subscribers.

by Vit Prajzler on September 26th, 2023

The concept of the Unified Namespace (“UNS”) is frequently at the core of discussions of digital transformation in the industrial market segment, IIoT and Industry 4.0.

In this article, I will explore how that high-level concept should guide a real-world usage of MQTT. The specific focus will be the optimal usage of MQTT topics by publishers and subscribers within a Unified Namespace.

Scroll to the bottom if you want the TL;DR version.

MQTT wildcard

Photo by Surendran MP on Unsplash

Publisher - Subscriber asymmetry

A common misconception in MQTT is that publishers and subscribers are basically the same, except that they are on the opposite side of an MQTT "pipe". You publish to a topic and subscribe to a topic. But that is not true. Let me elaborate.

When a publisher sends a packet to an MQTT broker, it sends it to one specific topic, such as site/facility/location/appliance/value-type for a generic topic format or spBv1.0/group id/message type/edge node id/device id if it uses Sparkplug B. (yes, spaces are valid characters in MQTT)

You've likely noticed the topic is a string, separated by forward slashes into topic levels. But why are the slashes used as separators, and not some other character? And do you even need topic levels? Can't you just use any string as an MQTT topic?

If publishers and subscribers were symmetric, you would be right. You could use any level separator - a backslash, a comma, or any other character, and just configure the subscriber in the same way. A topic would be just a string after all, right? Feel free to give it a try on your MQTT broker. At first glance, it will work. But it has one major drawback.

You might think you need topic levels because an MQTT broker somehow needs them to function properly, or because the standard requires it. But that's also not true.

The real reason you should use topic levels is because publishers and subscribers are not symmetric in their handling of MQTT topics.

Subscribers can use wildcards

In MQTT, there is one feature that publishers cannot use. It can only be used by subscribers. That feature is topic filters.

When a subscriber establishes a new subscription with a broker, it does not have to pick a specific topic. The subscription request it makes does not carry a topic, but a topic filter.

A topic filter can be identical to a topic, but it has two additional super-powers. Any topic level of a topic can be replaced with a + wildcard, and multiple topic levels can be replaced with a # wildcard.

From the site/facility/location/appliance/value-type example above, if you wanted to subscribe to all appliances in any location within a specific site and facility, you can use site/facility/+/+/value-type. That way you'll get all the specific value-type (e.g. pressure or temperature) across all the appliances within the site and the facility. If you wanted all values of all appliances, you can simply replace the last three topic levels with a #, resulting in site/facility/#.

Similarly, if you are interested in all values of a specific type of appliance no matter its location, you can subscribe to +/+/+/appliance/+.

The + wildcard can only replace one topic level, while the # wildcard can replace multiple topic levels. The # wildcard can only be used at the end of a topic filter, never in the middle.

Useful trick:
Discover existing MQTT broker topics

If you don't know what topics are in use in your MQTT broker, you can subscribe to # and check the topics in the packets you receive. Be careful though, because you are subscribing to all the topics the broker has. On a busy broker, that can be a lot of data, and a lot of additional load on the broker if it is not configured properly.

Why are wildcards so powerful?

While publishers are limited to publishing to a specific topic, subscribers have much more freedom and flexibility.

Why are topic filters and their wildcards such a critical feature of MQTT?

  • Auto-discovery: subscribers do not need to keep inventory of individual devices. They can focus on the type of data they need.
  • Portability: subscribers do not need to know the exact topic of a device, they can subscribe to a topic filter and discover the topic of a device at runtime
  • Scalability: subscribers can subscribe to a single topic filter and get data from all the devices that match the filter. No need to go one-by-one.
  • Extensibility: new devices can be added to the system without having to reconfigure subscribers

Why should you care?

Why can't you simply use an arbitrary string as an MQTT topic? For small deployments, you can. Pick whatever topic scheme fits your needs, as long as you are dealing with only a handful of devices.

For larger deployments, basically anything that spans two locations or more, I strongly recommend you spend a few hours with your stakeholders to do an inventory of everything that could ever be connected to your MQTT infrastructure, and decide on a hierarchical namespace for your topics. It is unlikely that you will know today what kind of subscribers you might have in the future, but is likely you can capture the producers. Start with those.

Consider the following when creating the topic structure

  • Reusability - could new data sources (devices) plug into the topics you designed
  • Semantics - how easy it is for a human to read a topic and know what it means without additional data / documentation / training
  • Security - who will be allowed to subscribe to what wildcards
  • Scalability - how easily can topics be divided / separated / sharded

TL;DR

An MQTT publisher generates data by sending publish packets to an MQTT broker. A publish packet is always addressed to one specific topic. That topic cannot have any wildcards. But it should have topic levels.

MQTT topic levels are segments of a topic, separated by the forward slash character.

An MQTT subscriber does not have to subscribe to a specific topic. A subscribe packet subscribes to one or more topic filters, which can be topics, but can also contain wildcards instead of one or more topic levels.

A single subscription request can ask for single, multiple, or even all topics of the broker with a single topic filter. With a smart design of your MQTT topic namespace, you can leverage this feature to your advantage.

A well-designed MQTT namespace can

  • make your infrastructure scale out more easily
  • make new devices work out-of-the-box
  • make it straightforward for subscribers to get to all the data they need without having to do lengthy manual configuration
  • make it easy for users to understand and user your system without lengthy training
  • make your solution simpler to secure and audit

If you have more questions about how to use MQTT optimally in your IoT projects, please contact us.

Ready to get started with MQTT on the Avoguard Gateway?

Contact Avoguard

Contact us to get a quote or book a demo.