We are now over two years into the global chip shortage, which has directly affected more than 169 industries. Many of the integrated circuits needed to produce everything from automobiles to computers, and even the graphics cards that are so important to machine learning, are very difficult to come by. Several factors have contributed to this present state of affairs, and it is not certain when supply and demand will return to normal levels and alleviate the problems we are experiencing. Until that time comes, we will have to make do with the supply of chips that is available. One area worth exploring that may at least help reduce some of the pressures industry is experiencing is reducing waste.
Electrostatic discharge, or ESD, is a brief flow of electric current between two charged objects that occurs on contact. Static electricity that builds up in objects like wrist watches or jewelry can be released into a chip while handling it, which can cause damage and render the chip useless. If safe practices are not consistently adhered to while assembling electronic devices, valuable chips will be destroyed, and shortage-related problems will only be exacerbated. Engineer Justin Lutz came up with a clever idea that can help to ensure that safe handling practices are consistently followed while working with electronics. Using Edge Impulse and off-the-shelf hardware, he showed how it is possible to prevent costly errors before they ever occur.
Lutz designed and built a prototype device that uses computer vision and machine learning to keep vigilant watch over an electronics assembly area. The system verifies that the technician is not wearing any problematic items, like rings or wristwatches, and also checks that they are wearing an anti-static wrist strap. If any safety violations are noted, the device can then generate an alert to prevent the technician from beginning work until the safety concerns have been addressed. As Lutz notes, it would not take much additional work to also send an SMS message to alert a supervisor of the issue in a future iteration of the device.
A Seeed Studio reComputer with a Jetson Nano 4 GB and 16 GB of eMMC storage was selected to power the creation. This computer packs a 128-core Maxwell GPU, a quad-core ARM A57 CPU running at 1.43 GHz, and lots of ports for expansion into a tiny, energy-efficient form factor that is ideal for edge machine learning applications A camera connected to the reComputer’s CSI port was included to give the device vision. The camera was mounted above the workbench of an electronics assembler, where it can get a good view of everything that is going on.
To identify items like jewelry and ESD wrist straps in images, Lutz chose to use Edge Impulse’s powerful FOMO object detection algorithm. FOMO makes it possible to run what is traditionally a very computationally intensive algorithm type on resource-constrained edge devices at upwards of 60 frames per second. As a prerequisite, training samples needed to be collected to teach the model how to recognize objects of interest. So, Lutz started collecting videos from his electronics workbench where he demonstrated both good and bad ESD safety practices.
Images were then extracted from the video files at regular intervals of one second. These images were uploaded to Edge Impulse Studio, where the labeling queue tool was used to draw bounding boxes around each object to identify them. Using the AI-assistance feature, this was made pretty simple — the tool automatically draws boxes around each object, then it is only necessary to review the boxes, and perhaps tweak them just a bit. For this proof of concept, a very small dataset was collected — only about 100 example images for each class.
Having squared away the training data, everything was in place to create the impulse that processes the data. Images were first preprocessed to reduce their resolution and convert them to grayscale, which reduces the computational resources needed by subsequent steps in the impulse. The processed images were then forwarded into a FOMO object detection algorithm that can recognize four object types (ring, watch, wrist strap, and background). Recognizing more object types for future versions of the device would be as simple as collecting and labeling more training data.
After training was finished, the model was validated against a portion of the collected dataset that was held out of the training process. This revealed that the model had achieved an average object detection accuracy rate of 71%. Considering the small amount of training data supplied, this is a nice result. The confusion matrix showed that the model sometimes had difficulty distinguishing between watches and wrist straps as they are very similar visually. With a larger dataset, the model would be expected to distinguish between them much better, and the detection accuracy would increase significantly.
The result was more than good enough for a proof of concept, so Lutz deployed the model to the reComputer. This was done by selecting the Linux binary option from the deployment tool, and downloading it. Just like that, the ESD safety monitor was finished. It was tested out on Lutz’s workbench, and it was working exactly as expected.
Lutz is exploring adding the ability to send text messages in a future version of the device, and is also excited to see how FOMO-AD, which was teased at Imagine 2022, can improve the prototype by also incorporating anomaly detection into the algorithm. That is slated for release in 2023, so we will have to wait for an answer on that, but in the meantime, take a look at the documentation for this project to learn some tips and tricks. When you are ready to build your own object detecting device, you can clone Lutz’s project to get a headstart.
Want to see Edge Impulse in action? Schedule a demo today.