Skip to main content

Azure IoT Edge device data on the Power BI dashboard

I wanted to present the IoT sensor data on the Power BI dashboard as near real time as possible. And it was not as easy as I thought initially. I will compare two methods of doing this with Azure and how that affects the latency.

I'm using Azure IoT Hub and Azure SaaS as much as possible with as little bespoke code as possible.

So I would like to show the sensor status on the Power BI dashboard as close to real time as it could be done.


In my scenario I have a bespoke IoT Edge module that collects data from a custom built motion sensor. The data then is sent to the $upstream and available in IoT Hub.


The structure of the data is simple it either sends information that there is motion detected or that there is no motion detected - the val field. And the time when it happened.
{ "val": 1, timeCreated:"2020/03/31 19:20"}

Two methods

I've found two methods to do it. One with Stream Analytics and another one with Azure Function and Power BI dataset raw endpoint. Let's compare them.

Solution 1: Stream Analytics


That worked well, had SaaS, no bespoke code required. I could add some additional logic inside the Stream Analytics query which was brilliant. 
But unfortunately it did not work as fast as I wanted. It was taking approx 11 seconds from the edge module to appear on the Power BI dashboard that was not acceptable for me. After a bit of digging I found that there is a metric called Wateremark delay. And indeed it was showing 11s. 

In Stream Analytics Event Ordering section there is a setting accepting late events set to 5 seconds by default.
I changed that to 0 and also handling other events action to Drop. That reduced the delay to 6 seconds. Still not acceptable for me. If you know how to speed this up let me know.
Having no other ideas I started to look for an alternative solution.

Solution 2: Azure Function and Power BI raw dataset endpoint


You can find more details how to set it up in https://www.youtube.com/watch?v=o4RwJ8b6N40 and a sample code of the Azure Function pushing to Power BI in https://github.com/yujhongmicrosoft/eventhub-demo.

This was not the ideal solution because it required me to write bespoke code for the function, which I wanted to avoid and use just the SaaS. I could not make use of the queries like in Stream Analytics so it was just the raw data, unless I could do more bespoke code processing inside the function.

However from the latency point of view it worked brilliantly. I did not measure precisely enough to say in milliseconds but it was below 1s. So exactly what I wanted.

Summary

Both solutions technically work. With Stream Analytics I have SaaS, Queries but 6 seconds. With Function I have bespoke code that I would have to maintain, any transformations or aggregations have to be done in the function code, but it is below 1 second.

Naturally in your solution you may need to consider other requirements and features of both services. For me important was to have it (in priority order) near real time, with no bespoke code and with option to do some modifications. So from my point of view I have not found an ideal solution.

If you have other ideas how to approach it or have any other comments let me know.

Labels:

Comments

Post a Comment