MQTT – How to consume a snapshot value published in the past.

Introduction

MQTT, or Message Queuing Telemetry Transport, is a lightweight messaging protocol designed for devices with limited processing power and bandwidth. To understand MQTT, it’s essential to grasp its fundamental principles, such as the publish/subscribe messaging pattern and Quality of Service levels. For beginners looking to delve into MQTT, resources like the official MQTT website (mqtt.org), the Eclipse Mosquitto broker, and online tutorials on platforms like YouTube and Udemy can be incredibly helpful in getting started.

In this post, I’ll explore design considerations for transporting time series data typically generated by analog sensors and distributed to various subsystems such as displays, databases, supervision systems, cloud systems for machine learning, predictive maintenance, and more.

Use Case 1 -Monitor and share the temperature inside a freezer.

In this configuration, a temperature sensor has been included in a freezer. A component is now in place to change the information gathered by the sensor into an mqtt topic, structured like this:

Topic: freezer/<guid>/temperature

Payload: {unix_time_stamp:<ts> , value:<int> }

The layout of the component design is illustrated in the image provided:

Consider this scenario: what if the section that displays the temperature information is refreshed every minute? If users refresh the page immediately after receiving the latest update, they will have to wait approximately a minute to receive new data. While this delay may be suitable in some cases, it may not be ideal for all situations. In the next part, we will investigate several alternatives to address this issue.

Use Case 2 – Extend use case 1 to display the current temperature

The use case 1 as a small draw back, the update of the subscribes depend on the publish event.

If the publishing period is set to 2 minutes, restarting the display subscriber could result in a delay of up to 2 minutes for updates. To prevent this, consider adding 1 or 2 extra topics depending on the expected data updates. One topic can be used to request current data, which the temperature can subscribe to. Then, a response topic can be published or the existing one can simply be updated. If a response is published, the requester needs to subscribe to it. Overall, this approach increases design complexity and necessitates careful attention to prevent errors.

Let’s design this 2 options:

Use case 2 – Option 1 – Force publisher

Is this case we can design this topic:

Topic: freezer/<guid>/temperature.request

Payload: None

Pros: No need to create a response topic neither some additional handling that might be common to the event data

Cons: All components that subscribe to the data provided by the periodic event will also updated, this might not be desired is some cases.

Use case 2 – Option 2 – Add request/response pattern

The second option is to use a topic to request the data and another to publish the response, this is also known as a MQTT request/response design pattern.

There are two additional topics

Request topic: freezer/<guid>/temperature.request

Request payload: None

Response topic: freezer/<guid>/temperature.response

Response payload: Same payload as the topic freezer/<guid>/temperature

Pros:

• Selective usage of the request/reply pattern
• simple way to use a pseudo data synchronization
• Alternative way to retrieve data on demand

Cons:

• Additional design and implementation complexity

Use case 3 – Ensure a response using the request/response pattern and the retain flag

One option is to implement a retry mechanism in the system. When a request is made to the component and it is not yet ready or available, the system can automatically retry the request at regular intervals until a response is received. This approach ensures that the component eventually receives the request, but it may lead to delays in receiving the information.

Another approach is to implement a notification system for the component. When the component is ready to receive requests, it can send a notification to the system indicating its availability. The system can then resend the request to the component. This approach reduces the chances of delays in receiving the information, as the component only receives the request once it is fully ready to process it.

Each of these design choices has its own set of advantages and disadvantages. The retry mechanism ensures that the component eventually receives the request, but it may result in delays. On the other hand, the notification system reduces the chances of delays but requires additional implementation for the notification mechanism. Ultimately, the choice between these two approaches will depend on the specific requirements and constraints of the system.

As previously mentioned, there are two choices available. The first option is to post the request using the retain flag, which ensures that the topic remains active during operation. Subsequent components that sign up for it will then be sent the published event along with the topic information.

Topic: washish_machine/+/schedule.resquest
Payload: {request_id:<unique id>, unix_time_stamp:<ts> , start_date:<int> }

Topic: washish_machine/+/schedule.response
Payload: {request_id:<unique id>, unix_time_stamp:<ts> , status:<OK|NOK>, status_message:<String> }

Another choice is to designate a subject for the publisher’s status update. This method also incorporates the Last Will and Testament (LWT) framework from MQTT. The LWT is established when the client connects and is broadcasted by the broker upon disconnection.

pros:

• No need for the retain flag, this can be problematic for bridged brokers (!)
• No need to clear retained topics

cons:

• There is a need to manage the scheduled requests, this can be a topic if there are many schedules and there could be a need to store it persistently if the Home Automation shutdowns before the request reach the washing machine

• More topics and respective subscriptions are required for the component status

Last Thoughts

In considering the design options for implementing a robust messaging system for edge and remote devices, it is crucial to account for the unique challenges presented by these devices. Given that they may not always be connected due to the nature of the technology or network constraints, it becomes imperative to create a system that can effectively handle intermittent connectivity and ensure reliable communication.

One possible design option is to incorporate a decentralized messaging architecture, where messages are distributed across multiple nodes in the network. This approach can help to ensure that messages are delivered even if some nodes are offline, reducing the risk of message loss or delays. Additionally, implementing message queuing systems can help to buffer messages during periods of network instability, ensuring that they are delivered once connectivity is restored.

Another design consideration is the use of lightweight communication protocols that are optimized for low-bandwidth and high-latency environments. By optimizing the communication protocols used for messaging, it is possible to reduce the overhead and latency associated with message transmission, improving the overall efficiency and reliability of the messaging system.

In addition to these design options, practical examples can be provided to illustrate how these principles can be applied in real-world scenarios. By drawing on experiences from the past seven years of working with global distributed IoT systems, valuable insights can be shared to inform the development of robust messaging systems for edge and remote devices. Through a combination of theoretical design principles and practical examples, a comprehensive understanding of effective messaging system design can be achieved.