The famous aphorism “time is money” rings just as true today as when it was first written centuries ago. This is especially relevant in the workplace, where anything that causes workers to waste their time reduces productivity and hurts the bottom line. And with macroeconomic conditions worsening and starting to put a pinch on business, staying efficient might make the difference between keeping afloat and closing the doors forever. Solutions that help workplaces to work smarter are needed now more than ever.
The master engineers of Zalmotek have recently turned their attention to a problem that plagues certain types of workplaces. Shift workers that share a common set of tools lose a lot of time when those tools are not properly put away by workers from the previous shift. In a series of informal interviews with facility managers conducted by Zalmotek, this was the most prominent issue impacting efficiency that was brought up. Considering the importance of this problem, the team decided to build a device that could help solve it with Edge Impulse and an NVIDIA Jetson embedded computing board.
After considering a few options, they decided on an approach that involves placing brightly colored tokens underneath tools that are hanging on a pegboard. When tools are not in their places, the tokens are visible to a computer vision system that can count them and determine if any tools are out of place. They believe that by using bright colors and distinctively shaped tokens, this method will be highly accurate and less resource intensive than alternate techniques.
An NVIDIA Jetson Nano 2GB Developer Kit provided the horsepower for the device, and a basic USB webcam was used to capture images of the pegboard. With a 128-core NVIDIA Maxwell GPU and a quad-core Arm A57 CPU running at 1.43 GHz, the Jetson is a beast in edge computing. And as a full Linux computer, it is capable of bringing even the most complex of machine learning-powered creations to life.
With the hardware all squared away, the team got busy 3D printing the tokens to place behind the tools. After some experimentation, they found that the best performing tokens were red and blue triangular shapes. A token of this sort was printed, then placed behind each tool. The next step in the build involved detecting and counting the tokens on the pegboard. To do so, of course they first needed to capture images of the pegboard populated with various patterns of tokens and tools.
A relatively small dataset of 115 images was captured initially, then uploaded to Edge Impulse Studio, where a data analysis pipeline could be designed. The labeling queue tool was then used to draw bounding boxes around the tokens so that the model could learn to recognize them during training. After that, preprocessing steps were added to the impulse to resize the images, and to extract the most relevant features from them. These features serve as the input to a MobileNet V2 SSD FPN-Lite object detection neural network.
This neural network was trained using the previously uploaded pegboard images, then the model was tested to ensure that it was performing well. This validation showed that a full 100% of samples registered a precision score over 80%. That finding certainly confirmed Zalmotek’s belief that using brightly colored tokens would produce a highly accurate model.
Using the Edge Impulse CLI, the trained model was downloaded directly to the Jetson. Then after creating a simple Python script to handle running the classifications, a service was created that would run the script at the end of every shift. By using this device, the workers would be reminded if they needed to clean up their tools before heading out the door so that everything would be ready for the next shift to hit the ground running.
While this application is directly applicable to many situations, Zalmotek notes that it would be possible to make small tweaks to their methods to adapt the solution to a whole new set of problems. Take a peek at the project page to help get your own workplace in order.
Want to see Edge Impulse in action? Schedule a demo today.