There are millions of elevators in the United States alone, and these elevators are indispensable for moving people around large buildings and giving assistance to the disabled and elderly. Without these elevators, large cities in particular, with their many skyscrapers, would become much less accessible and less efficient. But this does not mean that these lifts are without their risks; in fact, 17,000 people are injured on elevators each year in the United States according to the U.S. Bureau of Labor Statistics. This is a significant problem, but since we cannot do without elevators, the best course of action would be to make them safer.
Each elevator is rated to operate under certain conditions, and exceeding these limits is a recipe for disaster. Overloading is especially problematic, so many elevators have weight sensors that halt operation when loading is outside of the allowed parameters. The problem with weight sensors is that they are costly to install and maintenance is also expensive. Sometimes these costs prevent weight sensors from being installed in elevators in the first place, which can result in failures.
A very interesting idea from engineer Nekhil Ravi may get around this problem by checking elevator loading without requiring expensive sensors or maintenance. He built a computer vision-powered device, with the help of Edge Impulse, that can count the number of people on an elevator using a low-cost Arduino development board. The number of people detected in an elevator can be used as a proxy to determine how much weight the elevator is loaded with, and that can be used to trigger a shutdown switch when limits would be exceeded.
The only hardware required for the build is an Arduino Nicla Vision, which has a powerful STM32H747AII6 dual Cortex-M7/M4 processor that is ideal for running machine learning algorithms. As the name implies, the Nicla Vision also has a camera — a two megapixel color camera to be precise. This inexpensive board comes in a tiny 22.86 x 22.86 mm form factor. The GPIO pins on the board could be used to control an elevator directly, for example to prevent it from operating until loading is reduced, but for this prototype device, Ravi added in a simple buzzer and LED to serve as an alarm. The hardware was installed in a custom, 3D-printed case so that it would be ready for real world use.
Since the goal was to count the number of people in an image, Ravi chose to use an object detection algorithm built with Edge Impulse. This algorithm needed to learn what a person looked like, so a small dataset was collected that contained various images of people. Ravi had first linked the Nicla Vision to his Edge Impulse project, so as he captured images they were automatically uploaded. After the images were collected, he drew bounding boxes around the people in them using the AI-assisted labeling queue tool.
An impulse was created that first resized the images and converted them to grayscale. This reduces the amount of data that downstream steps need to process, which will improve the performance of the object detection pipeline on resource-constrained edge computing devices like the Nicla Vision. This data is then fed into Edge Impulse’s ground-breaking FOMO object detection neural network that was trained to recognize people using the uploaded data. A few hyperparameters were adjusted, then the training process was kicked off.
An excellent F1 score of better than 95% was observed after training. The model was further validated against the test dataset, which was not involved in the training process. This check revealed an 83.33% accuracy rate, which is quite impressive considering that the total data collected was limited to 73 images. Before rolling out a tool like this for real world use, a much larger dataset would be collected, which would be expected to improve the model’s accuracy significantly. But in any case, this is a great result for a proof of concept, so Ravi moved on to deployment of the model on the Arduino.
Edge Impulse offers a range of deployment options to allow the pipelines to be deployed on many hardware platforms. In this case, an Arduino library fit the bill nicely, so Ravi downloaded a library containing everything needed to run the model, then opened it in Arduino IDE and flashed it to the board. All that is left to do is mount the device inside of an elevator.
With the cost savings and reduction in maintenance that a device like this offers, it is probable that all manner of legacy sensors will be replaced in the future with devices similar to this one. If you want to get ahead of the curve and start building tomorrow’s solutions today, Ravi has a great write-up that will help you to get on your way. And don’t stop there — clone the public Edge Impulse project to try it out for yourself. No worries if you do not have an Arduino Nicla Vision just yet, the same algorithm can just as easily be deployed to dozens of other edge computing platforms.
Want to see Edge Impulse in action? Schedule a demo today.