The convenience and time savings that come with online shopping have made it an ever more popular choice with shoppers in recent years. Of course, having those online purchases delivered right to your front door is also very appealing. But unfortunately, where new trends turn up, so do new types of crooks. According to one recent survey, about 210 million packages were stolen from Americans’ front porches in the past year alone. That is a very serious problem for both online retailers and shoppers alike.
It is possible to have packages, for example, delivered to the local post office rather than your home to safeguard against theft, but by doing so the advantages of having purchases delivered directly to your home are largely lost. Machine learning enthusiast Solomon Githu had an idea to protect against package theft without having to give up the convenience of front door deliveries. He made use of a low-cost computer vision development board and the Edge Impulse platform to build a device that can detect when packages are snagged by porch pirates.
Githu chose the Espressif ESP-EYE development board, which is fully supported by Edge Impulse. This tiny, low-power board comes with an ESP32 microcontroller, 8 MB of PSRAM, 4 MB of flash memory, a Wi-Fi radio, and an OV2640 camera. At just 41 by 21 millimeters, the ESP-EYE is just about the perfect platform for tinyML computer vision applications. To make the most of the resources available on this board, Edge Impulse’s brand new approach to run object detection models on constrained devices, called FOMO, was utilized.
To get started building the object detection pipeline, Githu first had to build a dataset containing images of packages. 275 images of boxes were collected — other types of packages, like envelopes and poly-mailers, will need to be added to the dataset in the future to make the device more robust, but this starting point is sufficient to prove the concept. The images were uploaded to Edge Impulse with an 80/20 training/test dataset split. To complete preparation of the data, the labeling tool was used to draw bounding boxes around the packages.
Preprocessing blocks were added to the impulse to resize images and convert them to grayscale. Both transformations reduce the dimensionality of the input data, which reduces the computational resources needed for the final analysis pipeline. Finally, a FOMO object detection learning block was added to complete the impulse. The model was then trained, after which an F1 score of 94.4% was observed. The 20% of the dataset that was initially set aside from the training process was then used to test the model, and a 91% accuracy level was found, which is quite impressive for such a small training dataset.
With the ESP-EYE being fully supported by Edge Impulse, deployment to this platform is very simple. A few clicks can generate native firmware for the board that can be flashed to it with a one-liner on the command line. Even on this resource-constrained platform, by using the FOMO algorithm, it was possible to achieve one frame per second performance for object detection. This may not be blazing fast, but it is more than sufficient for detecting packages at your front door. If faster inference times were needed, the same model could be deployed to a more powerful platform with a few more clicks. A Raspberry Pi 4, for example, can process up to 60 frames per second using FOMO.
With Githu’s device already handling the heavy lifting, it would be a small step to extend it to send alarms when a package has been stolen. His idea for future development is to count the number of packages detected, then if that count unexpectedly becomes smaller an alarm could be sounded, or a text message could be sent. With the ESP-EYE having onboard Wi-Fi, the possible options for actions to take are only limited by your imagination.
Githu noted that building and deploying machine learning models with Edge Impulse was very fast and easy. And if you are interested in building your own package detection device, it will be even easier since he has made his Edge Impulse project public. Make sure to also take a look at the project tutorial for all the details.
Want to see Edge Impulse in action? Schedule a demo today.