Surgical instruments, sponges, or other objects that are inadvertently left inside a patient's body following a procedure are referred to as retained surgical bodies. Despite stringent safety protocols and advancements in surgical practices, these incidents can still occur, posing serious risks to patient health and safety.
Several factors contribute to the occurrence of retained surgical bodies. One common factor is the hectic nature of surgical environments, where numerous instruments and items are used during procedures. High-stress situations, time pressure, and distractions can lead to oversight, causing medical teams to unintentionally leave objects inside patients. Additionally, during complex surgeries or emergency situations, maintaining an accurate count of surgical instruments and items may prove challenging, increasing the likelihood of retention.
There are a number of practices that are intended to prevent the occurrence of retained surgical bodies. The most fundamental measure is the implementation of standardized counting procedures before, during, and after surgery. Surgical teams meticulously count and document all instruments and items used, cross-verifying with a designated checklist. The use of radio frequency identification technology and barcoding systems on surgical instruments has also been introduced to enhance tracking and minimize human error.
Unfortunately, errors do still occur. And those errors can be very costly — complications arising from retained surgical bodies can be severe and potentially life-threatening. Patients may experience localized pain, infection, inflammation, and, in some cases, damage to surrounding organs or tissues. The foreign object can lead to the formation of abscesses or contribute to the development of systemic infections. In extreme cases, retained surgical bodies may necessitate additional surgeries to remove the object, which poses further risks and complications.
Building a smart solution
In an effort to remove human error from the equation, Eivind Holt has developed an automated system that can objectively assess if the entire surgical inventory has been accounted for at the end of a procedure. This system never suffers from stress in emergency situations, and never gets distracted from the task at hand, which could make it the ideal solution to the problem. The initial prototype has been implemented as a wearable device that can recognize a variety of surgical instruments as it sees them so that medical professionals can later ensure that they are all accounted for.
Building a device like this can quickly get complicated. To be practical, a wearable device must be small and capable of operating for long periods of time on battery power. Designing an object detection algorithm that can efficiently run on such a tiny hardware platform also presents a number of issues, as computer vision-related tasks are often very resource-intensive. Moreover, for a typical person, collecting enough images of surgical instruments to train the object detection model can be challenging.
Choosing hardware
Holt was able to work through these problems by carefully selecting a set of tools that can address each of the challenges. On the hardware front, an Arduino Nicla Vision was chosen. At just 22.86 mm x 22.86 mm, it can easily clip on a shirt without getting in the way. With its built-in image sensor and an Arm Cortex M7 processor with 1 MB of RAM, it has the power to run some fairly sophisticated algorithms. Object detection algorithms, however, can easily stretch well beyond the capabilities of this device, so Holt utilized Edge Impulse’s FOMO object detection algorithm to not only make the development and deployment process simple, but also to optimize the model for the Nicla Vision hardware.
With the basic game plan finalized, Holt got together a few basic instruments, like scalpels and forceps, then took a set of about 600 images with the Nicla Vision. After these images were uploaded to Edge Impulse and used to train the object detection model, it quickly became clear that something was wrong. The object detection accuracy rate was too low for the model to be of any use. After some investigation, Holt determined that reflective objects were the problem. All of the reflections changed the features of the objects that the model detected. And unfortunately, a large percentage of surgical instruments have a reflective chrome surface.
More data needed
Collecting a larger, more diverse dataset would likely resolve this issue, but that can be very challenging, especially in the context of surgical equipment. So, Holt came up with the idea of utilizing NVIDIA’s Omniverse Replicator to quickly generate a large, varied dataset to further train the model. With this tool, 3D models of a wide variety of surgical instruments could be imported into a scene, then Replicator will move them around, adjust angles, lighting, and other factors to produce synthetic images that can be leveraged in model training. In this way, over 30,000 training images were produced.
After uploading this new set of data to Edge Impulse, Holt kicked off another round of model training. The results were much more favorable. The average object detection accuracy had climbed to 85.5%, which is more than good enough to prove the concept. This is an important reminder just how important good training data is — without the boost from Omniverse Replicator, this project may have been abandoned as too impractical.
To put the finishing touches on the project, the object detection pipeline was deployed from Edge Impulse to the physical device. It was noted that the model can be deployed as an Arduino-compatible library, which allows for total customization of the application running on the hardware. In this way, the object detector could wirelessly communicate with a phone or tablet app, for example, via Bluetooth Low Energy to provide information about the surgical instruments used during a procedure. As a final step, Holt 3D-printed a small housing for the Nicla Vision and a LiPo battery that can be clipped to a shirt.
It is amazing to think that with a bit of refinement, this inexpensive and relatively simple project could become a tool that saves lives in the future. Holt included a great deal of additional information in the project write-up, so be sure to take a look if you have any lingering questions or would like to get started building a wearable object detector for your own application.