As we head deeper into spring in the Northern Hemisphere, our thoughts are drifting far away from the hanging icicles that decorate rooflines in colder regions. But winter will come back around before we know it, which means that property owners, especially those that are responsible for a large number of properties, should be considering how they are going to manage the removal of those icicles. While they might seem picturesque, hanging icicles can pose significant dangers and practical challenges for property owners.
As they grow in size, these sharp ice formations can become heavy and may fall unpredictably, posing a risk to anyone walking nearby. This risk is particularly concerning around areas with high pedestrian traffic, such as entrances and exits to buildings. Falling icicles can cause serious injuries, and in some cases, damage vehicles and other property.
Furthermore, hanging icicles might be signaling that the building is being damaged. Frequently, icicles form as a result of ice dams — accumulations of ice that form along the edge of a roof. These ice dams can trap water on the roof, which can lead to leaks and water damage within the structure of the building. Over time, this can lead to costly repairs and compromise the integrity of the roof and the building's walls.
Fortunately, all of these problems can be avoided by removing hanging icicles, which is a pretty simple task in most cases. The trick is knowing that there are icicles that need to be removed so that action can be taken before problems have a chance to occur. Considering that icicles can form very quickly — at rates of up to 1 centimeter per second under certain conditions — it can be very challenging to respond quickly.
A vision-based solution
Manually keeping watch over a large number of properties is simply not practical, which can lead to dangerous situations developing. But engineer and machine learning enthusiast Eivind Holt came up with an idea that could solve this problem. With the help of Edge Impulse and an inexpensive Arduino development board, Holt created a proof of concept device that can detect the presence of icicles the moment that they begin to form, and send a long-range wireless message to inform the property owner.
As Holt drew it up, the device would be positioned where it has a view of a roofline that needs to be monitored. It would then regularly capture images and run an object detection algorithm to look for the presence of icicles. When they are detected, the device would transmit a message via LoRa to give an alert.
Since communities might need to deploy a large number of these devices, Holt was focused on keeping costs low to make use of the system practical. For this reason, he selected the Arduino Portenta H7 development board. This board has a powerful STM32H747XI dual 32-bit Arm Cortex-M7 and Cortex-M4 processor and a megabyte of RAM that allow it to run edge computing-optimized machine learning models. He paired this with a Portenta Vision Shield - LoRa, which adds the camera needed for visual inspections, and also a LoRa radio for sending alerts.
Digitally designed ice
Before being able to build the object detection pipeline, Holt needed to capture some images of icicles to train the model. This particular use case is more challenging than many others. Due to the translucent nature of ice and the optical tricks sunlight can play with it, detecting icicles reliably via computer vision algorithms can be difficult. A very large dataset should be able to overcome these issues — but that specific dataset is not exactly easy to track down.
For this reason, Holt decided to fake the data. This may sound like a dark and deceptive practice that should only be spoken of in hushed tones, but in reality, as AI models become increasingly more specific — or larger — they quickly run out of relevant data on which they can be trained. For this reason, synthetic data is finding its way into more training sets by the day.
Holt chose NVIDIA Omniverse Replicator for the generation of the synthetic dataset because it is able to produce high-quality synthetic images that can be used in the development of accurate machine learning models. After setting up a scene including a house and a realistic model of icicles, then leveraging the “Sun Study” Omniverse extension to realistically model outdoor lighting (remember how sunlight plays tricks with icicles?), Holt was able to quickly generate a dataset of tens of thousands of images. Holt configured Omniverse Replicator to vary lighting conditions, background colors and textures, and many other factors throughout the images to ensure that a well-generalized model could be produced from the training data.
Building a FOMO model in Edge Impulse
Another nice feature of Omniverse Replicator is that it labels the objects of interest for you, so there is no need to manually annotate the images, drawing boxes around each icicle. Holt uploaded that annotated dataset to a project in Edge Impulse, then created an object detection impulse. This impulse consisted of preprocessing steps that resized the images and converted them to grayscale to reduce the amount of computational resources that would be needed downstream. The optimized images then moved into a FOMO object detection model.
Object detection is normally a very resource-intensive task, but by using FOMO, objects can be tracked in real-time using up to 30 times less processing power and memory than MobileNet SSD or YOLOv5. In this case, the FOMO model would detect the presence of icicles.
The first round of training was conducted with 2,000 images, and the resulting accuracy score of 68.6 percent left much to be desired. Ice is challenging to detect, so Holt created more synthetic data (fortunately, there is no limit to the amount that can be generated!) and ran additional rounds of training. Holt settled on 26,000 images for his best result, getting an accuracy score of just under 87 percent. The accuracy did climb each time more images were added, so he could presumably improve this even further with a larger dataset.
Results
To determine how well this model worked, Holt first tested it out in a simulated environment with the help of NVIDIA Isaac Sim and the Edge Impulse extension. After deploying the model in a format that Isaac Sim can understand, it was put through its paces and found to reliably detect icicles. Finally, Holt tested the model on-device, using OpenMV in a real-world setting. This confirmed the ability of the device to recognize icicles with a high degree of accuracy.
By exporting the impulse as an Arduino library, Holt was able to modify the software to meet the project’s exact needs, which in this case involved sending LoRa messages. This was achieved by using The Things Network to transmit messages over LoRaWAN when the inference results indicated the presence of icicles.
Given how well the device worked, about the only thing left to do is put it inside a weatherproof case and mount it on the wall.
Are you ready to start preparing for next winter? Excellent! Holt has written-up a detailed guide that will help you along the way. Be sure to check out the public Edge Impulse project as well for some additional guidance.