Why is MQTT not a scalable protocol

Dominik Obermaier 1

In January 2018, the makers released the IoT protocol MQTT in version 5, which, compared to MQTT 3.x, above all offers more convenience and better usability.

The popular communication protocol MQTT has become an integral part of communication for the Internet of Things. Since its public availability in 2010, it has developed into the standard protocol for scalable, lightweight and bandwidth-saving push communication between mobile devices, connected cars, backend systems and IoT devices (Internet of Things) in general.

The version of MQTT currently used by most users is 3.1.1. The successor MQTT 5, which has been in work since the end of 2015, has been available since January 2018. In addition to many small improvements in the protocol, the makers of MQTT 5 have donated many new functions that should make working with the protocol easier and more comfortable. So it is time to take a look at the new features in the protocol and answer the question of whether it is already worthwhile to rely on the new version in projects.

Finally of legal age

Although MQTT began its triumphant advance in IoT projects mainly from 2012 - at that time the open source broker Mosquitto was already available and the first commercial MQTT broker HiveMQ was released - it is interesting to understand that the protocol is significantly older. The first version of MQTT was created in 1999 - at that time under a different name, but many of the popular functions were already included. Shortly before the release of MQTT 5, the protocol came of age, as it was 18 years old in 2017.

It may be surprising that the successor to MQTT 3.1.1 has version number 5 instead of 4. A look at a technical detail reveals the reason for this: The version in the CONNECT package of MQTT 3.1.1 has the number 4. Since the protocol only allows whole numbers in the header, the 5 had to follow at this point. It was therefore only logical to match the name and the technical versioning in order to avoid future confusion.

Use cases for MQTT

The new protocol version offers numerous new functions. To understand the motivation for this, a look at the use cases for MQTT helps. Most of the new features arose from requirements and challenges that the designers observed in large projects with MQTT. The protocol is particularly suitable for the following applications, among others:

  • Push-based communication,
  • Networking of backend systems and IoT devices,
  • Communication over unreliable channels such as cellular networks,
  • Decoupled communication of all participants via publish / subscribe,
  • Networking several million devices and
  • Transmission of data that conserves bandwidth.

Figure 1 shows schematically the decoupling of the subscriber communication via a central distributor, the MQTT broker. The participants do not know each other directly, but send messages to a so-called topic. Other participants can subscribe to this so that the broker can send the message to several participants. In contrast to classic messaging protocols, the topics are highly dynamic. Since the broker takes care of the creation, no manual creation is necessary. MQTT systems with hundreds of thousands or even millions of topics are not uncommon. For a more in-depth introduction to MQTT, the "MQTT Essentials" series from the HiveMQ blog and the introductory article on MQTT from heise Developer are suitable.

MQTT 5 at a glance

The Technical Committee for MQTT at OASIS, which is responsible for specifying MQTT, had set itself the following goals for MQTT 5, among others:

  • Simplifications of the protocol for highly scalable systems,
  • improved error handling and reporting,
  • Extensibility of the protocol,
  • Simplifications of state transitions,
  • Identification and provision of functions for common communication patterns with MQTT and
  • improved authentication and authorization mechanisms.

Although MQTT 5 can clearly be seen as the successor to MQTT 3.1.1, the new protocol is not backwards compatible due to the abundance of innovations. For the interaction, however, there are brokers that guarantee downward compatibility, so that MQTT-3.1.1- can interact with MQTT-5 clients in the same system.

This article concentrates on the most interesting new features in MQTT 5. The blog series "MQTT 5 - New Features and Hidden Gems" offers a more detailed, English-language overview of all features.

Best before date for messages

MQTT has always been stateful: The broker saves the state of the client, the so-called client session. The box "Data of an MQTT session" shows which data a session contains. By saving the status on the broker, a client can reconnect at any time and continue its session. The broker is able to deliver messages to the client that the client missed when he was not connected.

Data of an MQTT session

An MQTT session of a specific client at the broker usually consists of the following data:

  • Subscriptions: All topic subscriptions of the client.
  • Session data: metadata such as client identifier and other broker-specific session data.
  • Queued Offline Messages: All MQTT messages that the client should receive while it was offline. The client receives the data as soon as it reconnects.
  • Outgoing Message Flow: All Quality of Service 1 and 2 flows that have not yet been completed. This means that brokers and clients can guarantee message delivery guarantees in the event of disconnections.

The broker saves the data even if a client is not connected. The latter can therefore continue to work with the stored data at any time as soon as he reconnects.

However, there are cases in which a client never reconnects to the broker and it would be legitimate for the latter to delete the client's session. MQTT 3.1.1 did not provide for this function, even if some MQTT brokers such as HiveMQ and mosquitto allow a session to be deleted after a certain time interval.

MQTT 5 now allows each client to report the time to live (TTL) when the connection is established, i.e. the lifetime of the session when the client is offline. The broker is thus able to delete the session individually for each client if it does not log on to the broker again in the specified interval. This saves resources on the server.

In addition to the MQTT sessions, the new version of the protocol enables MQTT-PUBLISH- To add an expiration date to messages that contain user data. As soon as the time is exceeded, a broker is no longer allowed to deliver a message. This is particularly interesting for scenarios in which many clients are often not connected and the broker should hold the messages until the clients connect again. If the messages have expired, the broker can delete them prematurely. Even if the function is not included in MQTT 3.1 and MQTT 3.1.1, some brokers allow a TTL for MQTT-PUBLISH-Messages to use.

Description for errors

With negative acknowledgments, a function that has long been desired by the community has found its way into the MQTT standard. This means that numerous MQTT packets may contain error codes. For most error cases, MQTT 3.1.1 intended to disconnect the clients in the event of errors caused by the client.

In addition to numerous standardized error codes, MQTT 5 now offers the option of returning a description of the error as text to the client. The broker issues the error message as a response package such as PUBACK or SUBACK back to the client.

The broker is free not to use the error codes: The previous behavior of disconnecting MQTT clients directly without giving any further reasons is still permitted and may be desired in certain scenarios for security reasons in order not to offer attackers unnecessary information .

While MQTT 3.1.1 only uses DISCONNECTPackets were sent from the client to the server, the server is now allowed to enter before disconnecting DISCONNECT-Send the packet to the client to indicate that it will subsequently disconnect. The package may contain an error code to inform the client of the reason. This makes it easier for software developers to find out why the broker disconnected from their client in the event of an error.

Beyond binary data

MQTT sees itself as an application transport protocol and is data-agnostic. This means that the protocol does not in principle care about the type of data transmitted: Until now, all of them were considered binary data by MQTT. In project practice, however, it is often necessary for the recipient of a message to know exactly what type of data it is and how it is to be interpreted. With MQTT 5, clients can send a MQTT-PUBLISH-Add a so-called content type to the message, which defines the type of data. You can use both standardized MIME types, as are common with HTTP, and you can also specify your own interpretation aids.

In addition to the new header for defining a content type, a client can use an additional header to specify whether the user data is binary data or UTF-8 coded text. Since MQTT treats all user data sent as binary data by default, the option is useful, for example, to inform the recipient that he can simply process the data as text or should interpret it as binary data. Even if the headers for Content Type and Payload Format Indicator are used together in many cases, it is not necessary to set one or even both headers when sending a message. In projects, it can be assumed that developers will use both frequently in the future for reasons of maintainability, as the specification of content types has established itself among other things with REST APIs.

Direct answers

MQTT is a protocol that is based on the publish / subscribe pattern and uses it to decouple the sender and recipient of a message. The principle is in contrast to the request / response pattern, which is used in HTTP, among other things: the client sends a request to the server, which returns a response. With MQTT, a client subscribes to topics at the broker and receives the relevant messages for the subscription in a push procedure as soon as they arrive at the broker. In addition, each client can send messages to the broker.

In practice, it is common for a message recipient to want to send a direct reply to the original sender of the message. This is not initially provided for by the protocol, but can still be achieved with the on-board resources of MQTT 3.1.1. For example, many projects work with so-called feedback topics: the original sender of a message has a subscription to a specific reply topic, which the recipient of the message can use to reply.

Since the pattern has now become established, MQTT 5 offers the convention directly in the protocol: The new "Response Topic" header can be used directly in a PUBLISH-Package and informs the recipient of the message that he should send a reply about the defined topic. Developers can use another new header field in connection with the request / response pattern: They can use the correlation data field to assign messages to general response topics used by several clients. This is unambiguous information that is part of the communication between the parties.

Shared subscriptions

Many MQTT brokers already enable shared subscriptions for MQTT 3.1.1, as explained in a post on the HiveMQ blog. The function has met with a great response in many projects, so that it is now officially part of the standard with MQTT 5. Shared subscriptions are a type of client load balancing. Usually a single client orders a subscription and receives every message on the topic from the broker. Shared subscriptions offer an option to split the messages for a subscription among different clients so that each client only receives a subset of the messages.

The procedure is particularly interesting for distributed microservice backend systems that have to process a large stream of messages. If the volume on a topic is higher than the system can process, it can be distributed over several instances with shared subscriptions: the broker alternately sends the messages to one of the clients sharing a subscription. One advantage of the procedure is that the number of clients with a shared subscription can be scaled at runtime. If the load becomes too high, a new client can subscribe to the shared subscription, and if the load is lower, the client can cancel the subscription again via unsubscription.

User-related metadata

Many users faced a challenge with MQTT 3.1.1 when they wanted to transmit metadata to the recipient of a message in addition to the pure user data. With user-specific metadata, MQTT 5 offers a function desired by many users. The new standard defines any number of headers that can be used in most MQTT packages. This enables MQTT clients to enrich their packets with their own key-value pairs. The recipient of the message is free to decide whether or not to interpret the additional metadata. The user properties are mainly intended for application and scenario-specific properties that the broker should not interpret. There is no limit to the maximum number of user-property pairs, but MQTT packages can be enriched with any number of properties. The only requirement is that the values ​​can be encoded in UTF-8. Base64 coding, for example, is suitable for sending binary data in headers.

Further improvements

In addition, the new version of the IoT protocol offers many small and useful innovations. The following additions are worth mentioning:

  • Topic Alias: The function saves bandwidth because it supplements a short numeric alias that can replace the long topic name during transmission.
  • Flow Control: Clients and servers can determine how many messages can be processed in parallel.
  • Maximum packet size per client: Each communication participant can indicate the maximum size of a processable message.
  • AUTH Packet: The makers have introduced a new MQTT package that is used for complex authentication procedures such as the procedures specified in the SASL framework (Simple Authentication and Security Layer).
  • Forwarding to other brokers: MQTT brokers can tell a client that it should connect to another MQTT server.

Migration Considerations

MQTT 5 is a big step forward, and the multitude of new functions makes the fresh version attractive for new projects. This raises the question of whether it is already worthwhile to rely on the new version of the protocol or whether one should stick with MQTT 3.1.1 for now.

This question can not be answered generally. At the time of the article (early 2018), however, it can be stated that there is no production-ready implementation of an MQTT 5 client library and no broker supports MQTT 5. Interested parties can rely on MQTT 5 with the first implementations and make their first steps with the new protocol. In production environments, on the other hand, experience has shown that it makes sense to wait for the first wave of early adopters until the teething problems in the implementations have been eliminated.

The professional MQTT broker HiveMQ will be available in the second quarter of 2018 as an early adopter release with MQTT 5 support, and production-ready releases are planned for the third quarter. The makers of the open source MQTT broker mosquitto are also working on MQTT 5 support.

In order to remain flexible in projects, it is important when selecting the broker for the project to ensure that it can work in parallel with MQTT 3.1.1 and 5 in the future so that the change to a newer protocol version can be carried out easily.

Conclusion

MQTT came of age with version 5 and took the opportunity to leave behind criticized weaknesses of the older versions. The transport protocol now offers more flexibility than before, but is not noticeably more complex. MQTT sees itself as a lightweight, scalable, simple and slim protocol - and version 5 proves this again. With the additional functions and an ever larger group of users, MQTT is also armed for the next 18 years. (rme)

Dominik Obermaier
is the managing director of dc-square GmbH, the company behind HiveMQ. His main interests are MQTT, IoT protocols and highly scalable software for the JVM. He helped develop the protocol standards MQTT 3.1.1 and 5 at OASIS.

1 comment