Real power of MQTT - wildcards

Publisher - Subscriber asymmetry

A common misconception in MQTT is that publishers and subscribers are basically the same, except that they are on the opposite side of an MQTT "pipe". You publish to a topic and subscribe to a topic. But that is not true. Let me elaborate.

When a publisher sends a packet to an MQTT broker, it sends it to one specific topic, such as site/facility/location/appliance/value-type for a generic topic format or spBv1.0/group id/message type/edge node id/device id if it uses Sparkplug B. (yes, spaces are valid characters in MQTT)

You've likely noticed the topic is a string, separated by forward slashes into topic levels. But why are the slashes used as separators, and not some other character? And do you even need topic levels? Can't you just use any string as an MQTT topic?

If publishers and subscribers were symmetric, you would be right. You could use any level separator - a backslash, a comma, or any other character, and just configure the subscriber in the same way. A topic would be just a string after all, right? Feel free to give it a try on your MQTT broker. At first glance, it will work. But it has one major drawback.

You might think you need topic levels because an MQTT broker somehow needs them to function properly, or because the standard requires it. But that's also not true.

The real reason you should use topic levels is because publishers and subscribers are not symmetric in their handling of MQTT topics.

Subscribers can use wildcards

In MQTT, there is one feature that publishers cannot use. It can only be used by subscribers. That feature is topic filters.

When a subscriber establishes a new subscription with a broker, it does not have to pick a specific topic. The subscription request it makes does not carry a topic, but a topic filter.

A topic filter can be identical to a topic, but it has two additional super-powers. Any topic level of a topic can be replaced with a + wildcard, and multiple topic levels can be replaced with a # wildcard.

From the site/facility/location/appliance/value-type example above, if you wanted to subscribe to all appliances in any location within a specific site and facility, you can use site/facility/+/+/value-type. That way you'll get all the specific value-type (e.g. pressure or temperature) across all the appliances within the site and the facility. If you wanted all values of all appliances, you can simply replace the last three topic levels with a #, resulting in site/facility/#.

Similarly, if you are interested in all values of a specific type of appliance no matter its location, you can subscribe to +/+/+/appliance/+.

The + wildcard can only replace one topic level, while the # wildcard can replace multiple topic levels. The # wildcard can only be used at the end of a topic filter, never in the middle.

Useful trick:
Discover existing MQTT broker topics

If you don't know what topics are in use in your MQTT broker, you can subscribe to # and check the topics in the packets you receive. Be careful though, because you are subscribing to all the topics the broker has. On a busy broker, that can be a lot of data, and a lot of additional load on the broker if it is not configured properly.

Why are wildcards so powerful?

While publishers are limited to publishing to a specific topic, subscribers have much more freedom and flexibility.

Why are topic filters and their wildcards such a critical feature of MQTT?

• Auto-discovery: subscribers do not need to keep inventory of individual devices. They can focus on the type of data they need.
• Portability: subscribers do not need to know the exact topic of a device, they can subscribe to a topic filter and discover the topic of a device at runtime
• Scalability: subscribers can subscribe to a single topic filter and get data from all the devices that match the filter. No need to go one-by-one.
• Extensibility: new devices can be added to the system without having to reconfigure subscribers

Why should you care?

Why can't you simply use an arbitrary string as an MQTT topic? For small deployments, you can. Pick whatever topic scheme fits your needs, as long as you are dealing with only a handful of devices.

For larger deployments, basically anything that spans two locations or more, I strongly recommend you spend a few hours with your stakeholders to do an inventory of everything that could ever be connected to your MQTT infrastructure, and decide on a hierarchical namespace for your topics. It is unlikely that you will know today what kind of subscribers you might have in the future, but is likely you can capture the producers. Start with those.

Consider the following when creating the topic structure

• Reusability - could new data sources (devices) plug into the topics you designed
• Semantics - how easy it is for a human to read a topic and know what it means without additional data / documentation / training
• Security - who will be allowed to subscribe to what wildcards
• Scalability - how easily can topics be divided / separated / sharded