.svg)
Multiple industrial companies rely on tracking and managing the inventory of their products accurately. Manual counting methods can be time-consuming, prone to human error, and don’t provide real-time visibility into inventory levels. From parcels to packages to any other items found in an industrial warehouse, the need for automating the counting process of inventory is becoming a necessity.
In this case study, we will look at pallet counting, a key part of global shipping and logistics systems. The organizations that manage pallets have tools for tracking pallets but these require considerable manual interaction, causing various inefficiencies that result in slowdowns or missing inventory, and ultimately, lost value.
To demonstrate an automated, efficient pallet counting solution, Edge Impulse, the leading end-to-end platform for edge AI, created a vision-based setup that can accurately detect and track pallets as they move through a warehouse.
The two keys to building this are NVIDIA Omniverse and NVIDIA TAO. NVIDIA provides customers with the resources and tools to allow them to create custom ML solutions for a wide variety of environments. With NVIDIA Omniverse, customers are now able to easily augment their datasets by creating a digital twin of their environments to get synthesized images that can train real-world algorithms. In this scenario, Edge Impulse used Omniverse to digitally create a simulation warehouse, which includes virtual pallets and forklifts that can be captured as data for the model’s dataset.
Then within Edge Impulse’s platform, that dataset was used to create a pallet-detection ML model using NVIDIA TAO model architecture.
Finally, Edge Impulse deployed the model to various hardware targets for testing and implementation. Edge Impulse offers model compatibility with any type of hardware, from powerful NVIDIA GPUs to resource-constrained MCUs from any vendor — an exclusive capability that Edge Impulse provides for NVIDIA TAO models.
The result is a customer solution with a full custom computer vision solution for detecting pallets, created within a few weeks. Traditionally, the data collection, industrial-grade model training, and deployment process would take months to years to achieve, but with Edge Impulse and NVIDIA, the time to market has been reduced significantly.
When building professional-grade AI models, obtaining real-world data can be challenging, due to it being costly, time-consuming, or incurring privacy concerns. Recent advancements in synthetic data, however, are now offering a viable and valuable way to supplement that data collection. This is a game-changer for industries operating in industrial environments. NVIDIA Omniverse can generate highly realistic synthetic datasets precisely tailored to specific industrial scenarios, of a fidelity high enough to be used to train AI vision algorithms that then function accurately in the real-world environments where they are deployed.
-min.png)
The images depicted here show a warehouse with a forklift carrying wooden pallets created within NVIDIA Omniverse, specifically in the NVIDIA app called Isaac Sim. This is a robotics simulation application and synthetic data generation tool that allows users to create industrial digital twins to map their real-world counterparts. NVIDIA provides a variety of pre-made Omniverse content (such as their Industrial 3D Models Pack) so that users don’t need to create common industrial elements. This pack contains different types of forklifts, pallets, boxes, railings, and more. All of the synthesized images shown here were created using this pack. In this example, we recreated a real-world scenario of a white forklift carrying a pack of wooden pallets and lowering it onto a warehouse floor.
The side-by-side comparison shows just how accurate and realistic the synthesized image can be. A lot of factors were taken into consideration when creating the synthesized image. These include the angle of the camera, the top-down lighting, the physics of the pallets being stacked on each other, the forklift and pallet and floor color being accurate, and the busy warehouse background.
It may look like the job has been done since we were able to create an accurate image from the video, but this is just the start. Now that we have this digital twin, we can create an entire dataset of images that wouldn’t have been feasible before. Going out to the same warehouse and capturing more images would be expensive, time-consuming, and disruptive, but with NVIDIA Omniverse we’re able to generate whole new views like the following:
This enables a customer to synthesize a dataset of hundreds or even thousands of images. This entire synthesized dataset can be easily imported into Edge Impulse and labeled. Then Edge Impulse can be used to take these synthesized images to build an industrial-grade pallet detection ML model based on the NVIDIA TAO model architecture.
NVIDIA TAO (Train, Adapt, Optimize) provides a powerful and flexible toolkit for building machine learning models using the world’s best vision transformers. This toolkit is now integrated into Edge Impulse and this allows Edge Impulse users to replace their traditional ML models with NVIDIA TAO models.
Why choose an NVIDIA TAO model over another model?
TAO leverages transfer learning, allowing developers to start with pre-trained models and fine-tune them for specific tasks or datasets. This significantly reduces the amount of labeled data and training time required to develop high-performance models.
TAO is optimized on GPUs, CPUs, MCUs, and more.
TAO provides pre-built models and workflows tailored for specific use cases, making it easier for developers to get started and iterate quickly. This includes support for popular deep learning frameworks such as TensorFlow and PyTorch.
TAO is fully enabled on Edge Impulse allowing developers to customize models to suit their specific requirements, ensuring that the resulting models are optimized for performance and accuracy in their target applications.
Edge Impulse makes it easy to get started with NVIDIA TAO. Users can select their preferred NVIDIA TAO model within the learning block. For beginners, these models can be used as-is, or for ML experts, Edge Impulse provides an array of neural network training settings to adjust, such as which backbone to use — pre-trained backbones from NVIDIA or others trained by Edge Impulse on ImageNet. Other settings include the learning rate, data augmentation, loss, and much more.
What if you don’t know what settings to change, and iterating through each model seems daunting and time-consuming? Edge Impulse’s EON Tuner solves this. It’s essentially auto ML where you can point to different NVIDIA TAO model architectures, hyper-parameters, input, and processing blocks, as well as choose your target end device. It takes all of these factors and provides a list of all the permutations and their performance metrics to allow the user to choose which model is the most accurate. Essentially, it allows users to test their dataset on all NVIDIA TAO models without going back and forth tweaking parameters manually.
Here’s a look at the EON Tuner view:
At this point, Edge Impulse has created a dataset using NVIDIA Omniverse and brought that dataset into our platform to build an NVIDIA TAO ML model to detect pallets. Now the final step to production is to deploy this model to an edge device. In this case, we want to consider two hardware options — a GPU and an MCU. To stay in the NVIDIA ecosystem, the GPU chosen was the NVIDIA Jetson Orin Nano. The MCU used for testing is based on the Arm Cortex-M85.
Using Edge Impulse, with a click of a button, the model file will be optimized for the target device the user chooses. That means the entire ML pipeline flow used for deploying a model onto an NVIDIA Jetson Orin Nano is the exact same as deploying to an Arm Cortex-M85. Edge Impulse allows you to replicate this for any device under the sun; for more information check out the Deployment documentation page.
We fed both the GPU and MCU model a video of our scene on the edge devices. Here are the side-by-side inference results of the final pallet counting model:
Inferencing can be run not only on a physical edge device but also in Omniverse. This can be done through the open-source Edge Impulse Data Ingestion Omniverse Extension. This Omniverse extension allows users to upload data from a specified location on their computer to their Edge Impulse project. The final trained model can then be exported back to Omniverse for generating inference results.
-min.png)
This allows developers ways to test their models that they haven’t had before. Instead of needing physical hardware setups right away, the models can quickly be tested using synthetic data. This provides users with a quick and simple feedback loop to validate their ML models. It also allows testing to be scaled up since hardware isn’t a limiting factor anymore.
The NVIDIA Omniverse scene was created using an Amazon EC2 instance, leveraging the flexibility of running Windows powered by RTX-enabled GPUs in the cloud. By utilizing a cloud-based solution, the customer avoided the need for direct access to physical Windows hardware, gained the ability to scale resources on demand, and allowed collaboration between team members accessing the same Omniverse setup. This streamlined workflow ensured efficient design and simulation, regardless of local infrastructure constraints.
This collaboration between Edge Impulse, NVIDIA Omniverse, and NVIDIA TAO represents a significant advancement in industrial asset tracking. This automation of pallet counting using AI and computer vision is just the beginning. With Edge Impulse, this can be reproduced by any business for their own product in their own environment.
NVIDIA Omniverse's contribution to generating synthetic datasets seamlessly complements NVIDIA TAO's toolkit for ML model development and optimization. Edge Impulse further streamlines the process by facilitating the entire ML pipeline from data generation down to the deployment of these models to edge devices. This collaboration reduces the time and resources required for productizing AI solutions in industrial environments, empowering businesses with greater efficiency and competitiveness.
Contact us to discuss how Edge Impulse can help bring your solutions to market.
