.svg)
A print shop faced two distinct challenges: extending the life of an essential, aging copier and safeguarding air quality for workers. These challenges are found in many industries and are ideal for solving with edge AI. In this guide we outline a practical solution utilizing the Sony Spresense MCU, CommonSense sensor board, and Edge Impulse. The result is a step-by-step approach to implementing predictive maintenance for the copier and monitoring volatile organic compounds (VOCs) in the workplace environment.
Sony’s Spresense microcontroller is a powerhouse development board. Equipped with Sony's CXD5602 microcontroller (Arm® Cortex®-M4F × 6 cores), designed for Sony’s smart earbuds, the Spresense packs impressive capabilities into a compact package; ample I/O pins and hookups for camera, microphone, and speakers make it ideal for media applications, and its GNSS chip lets it connect with GPS and other geopositioning services. With the wearable origins of its chipset, Spresense is also highly energy efficient, capable of running on a pair of AA batteries for days without sleep.
Devices that combine this type of power, size, and efficiency are ideal for running embedded AI models, where the processing of sensor inputs and machine learning model predictions occur directly onboard. With its launch, Spresense quickly made a name for itself for AI applications. Machine learning developers have now prototyped a range of solutions with Spresense and Edge Impulse, from drone-based agriculture monitoring to analog meter tracking to occupancy-aware HVAC systems.
The powerful chip at the heart of Spresense has the capacity to incorporate a wide range of inputs beyond the built-in sound and vision modules. This is where the new CommonSense sensor board from SensiEDGE comes in. The recent launch of CommonSense, done in conjunction with Sony and Edge Impulse, compactly stacks 10 additional environmental sensors onto the Spresense. With the two boards coupled together, data from these sensors is directly accessible for users in Edge Impulse, including using multiple sensors concurrently with the Edge Impulse sensor fusion module.
Now in addition to sound and vision, users can track, measure, and create applications based on air quality, device movement, temperature, humidity, air pressure, light conditions, and more.
.png)
For creating and running AI algorithms directly on devices like the Spresense, Edge Impulse is a vital software tool. Its web-based platform and Python SDK offer the ability to gather and label data, train and optimize a model, and load it directly onto the Spresense, or any other hardware. Edge Impulse offers built-in algorithms including classification, object detection, and anomaly detection. With Edge Impulse, enterprises are able to massively reduce the development time and resource needs for creating bespoke edge AI solutions, including predictive maintenance.
Spresense is natively supported in Edge Impulse, as is CommonSense when coupled with the Spresense board. This means that you can access the board and sensors directory in our platform to gather data, build an algorithm, and deploy it onto the board for use. Edge Impulse makes these normally complicated procedures quite simple.
Predictive maintenance can help avoid costly downtime and repairs by detecting when equipment is going to fail, rather than replacing possibly still-good equipment at predetermined maintenance schedules. One print shop chose to monitor one of their copiers, an aging Xerox iGen4 machine, with a Spresense-and-CommonSense-based predictive maintenance device, in order to determine possible breakdowns and keep it in regular use.
This particular printer required regular maintenance and care to remain operational. Some common problems include paper jams, toner issues, and printer errors, all of which result in lost production when they occur. While the unit has basic features to identify the above problems in their interface, once you start using the machine there are some blind spots that you cannot safely detect.
By detecting early signs of equipment failure, such as abnormal vibrations, sounds, or electrical current draw, the print shop can undertake preemptive repairs and keep their Xerox from incurring more time-consuming breakdowns. These devices will be strategically positioned within the print unit at critical points to monitor vibrations and air quality, enabling them to identify potential issues and relay this information in real-time.
Additionally, the air quality in a print shop can be dangerous because of the chemicals used in the printing process, which can be harmful if you are exposed to them for too long. The environmental sensors on the CommonSense are ideal for monitoring conditions and sending out a warning when they approach unhealthy levels.
This project employs two of the sensors on the CommonSense board: the LSM6DS3’s 3D accelerometer and 3D gyroscope, and the SGP-40 air quality sensor.
Assembly is simple: Couple the Spresense and CommonSense boards together, and install them into the enclosure. The necessary sensors are all functional in the case. Then configure the software, and finally mount the hardware inside the Xerox iGen4.
Among its 10 sensor types, the CommonSense offers the SGP40 digital gas sensor, designed for easy integration into air purifiers or demand controlled ventilation systems. Sensirion’s CMOSens® technology offers a complete, easy to use sensor system on a single chip featuring a digital I2C interface and a temperature controlled micro hotplate, providing a humidity- compensated, VOC-based indoor air quality signal. The output signal can be directly processed by Sensirion’s powerful VOC algorithm to translate the raw signal into a VOC index as a robust measure for indoor air quality. The VOC algorithm automatically adapts to the environment to which the sensor is exposed.
GNU Arm Embedded Toolchain
The first step is setting up the build environment for the Sony Spresense board equipped with the CommonSense sensor board. For this, install the GNU Arm Embedded Toolchain with the following steps:
Next, you’ll need to install Python 3.7 to get the firmware onto the board:
Install the prerequisites for adding custom PPAs:
And then, add deadsnakes/ppa to the local APT package source list:
Run
And then:
With Python 3.7 installed, it’s time to create a virtual environment.
To do so, make sure you have pip installed:
And then issue the following command to install virtualenv:
With virtual env installed, run the following command to create an environment that runs Python 3.7:
There are a few modules that must be installed before moving on to building the firmware and flashing the board with it:
Everything is now in place to build and flash firmware on the Sony Spresense board.
To build a machine learning (ML) model that is able to perform predictive maintenance by detecting trends in machine vibration or volatile organic compound levels in the air (characteristic to ink or solvent spillage in the printing industry), use Edge Impulse. Register a free account and then create a new project, give it a fitting name, and press Create new project.
To connect the Spresense to Edge Impulse, first download the data forwarder firmware from github.com/edgeimpulse/edge-impulse-predictive-maintenance-vibration-commonsense-sony-spresense
There are two firmware options: one to measure the vibration of the printer, and one for measuring volatile organic compound levels. Pick either one to get started.
Launch a terminal, navigate to the software folder and activate the build environment:
Afterwards, build the firmware using make:
After the build is successful, flash the board by running:
Next, connect the board to the platform by using the data forwarder tool provided in the Edge Impulse CLI suite:
You will be prompted to fill in the username and password used to log into the Edge Impulse platform, and then assign the board to one of the existing projects. After doing so, you will see the development board appear under the Devices tab with a green dot next to it, signifying the fact that it is online and ready for data acquisition.
In printing industries, there are numerous volatile organic compounds (VOC) that can be found in the air inside and in the near vicinity of printing presses, like isopropanol, benzene, ethyl-toluene isomers, and styrene. To emulate an ink and solvent spillage, we will expose the CommonSense board to varying concentrations of isopropanol, which, being a very volatile compound, easily vaporizes and is picked up by the sensor.
Once the device is connected, go to the Data acquisition page, choose the one axis sensor defined when starting the edge-impulse-data-forwarder tool, set the data acquisition frequency to 25Hz, and begin data recording.
We will define two classes for this application, "Ink_Leakage" and "Normal".
To prevent the detection algorithm from producing a false positive result, it is crucial to add the "Normal" class, which contains readings specific to the normal conditions of the environment in which the system would be implemented. When gathering data, it is recommended to have at least 2.5 minutes worth for each defined class.
When recording data, you will notice that the board records the sensor readings in a buffer that is ultimately uploaded in the Edge Impulse platform and afterwards, you are presented in the Raw Data tab with the time-domain representation of the newly acquired signal.
It is noticeable that in normal working conditions, the sensor oscillates in a narrow channel, between 30000 and 31000. When an ink or solvent leakage takes place, the value abruptly drops to around 23000.
After you have gathered enough data for both classes, remember to click on the red triangle to rebalance the dataset. An optimal ratio would be 80% training data to 20% testing data.
The testing data pool will be used at the end of the process to see how the machine learning model fares on unseen data, before deploying it on the edge, saving a great deal of time and resources.
For vibration data, we must name the three axes exposed by the board as X, Y, and Z. Then when recording data, make sure the 3-axes sensor is selected. Use a Sample length of 10 seconds and leave the frequency on the default value.
When dealing with industrial machinery, running it in a faulty manner to acquire data specific to various malfunctions is out of the question. Instead, the approach here is to collect plentiful data when the machine operates nominally, is idle, or is powered off and creates an anomaly detection algorithm that will detect if something is out of order.
Just like when gathering VOC data, make sure you have at latest 2.5 minutes of data and perform an 80-20 split.
After populating the dataset it’s time to design the impulse. The impulse allows the user to control the defining parameters of the whole process of taking raw data from the dataset, pre-processing it into manageable chunks called "windows," extracting the relevant features from them using digital signal processing algorithms, and then feeding them into a classification neural network that puts them in one of the defined classes.
For the input block, we will be using time series data split into 5000ms Windows, with a Window increase of 1000ms and a frequency of 25Hz.
As a signal processing block we will be using a Raw data block and for the learning block, we will be using a Classification (Keras) block.
The Impulse for the anomaly detection algorithm (vibration) is rather different from the one used in the VOC case.
For the input block we have decided to go with a 1s window and a 1s window increase. For the processing block we have used a Spectral Analysis block and as a learning block, we have picked an Anomaly DetectionNN.
After pressing on Save Impulse, each block can be selected from the Impulse Design submenu and configured.
The Raw data block could be the most straightforward of the processing blocks because it has just one adjustable option, the "Scale axis," which we'll set to 15. The time-domain representation of the chosen sample may be seen on the upper side of the screen.
Afterwards, click on Save parameter and navigate to the feature generation tab. Nothing can be modified here so click on Generate features and wait for the job to end.
The Feature explorer lets you quickly assess if the data neatly separates, because it provides a visual representation of all the data from the Training dataset. When you click on a data item, the raw waveform, the signal window that was utilized, and a direct link to the signal processing page are all presented. This makes it possible for you to locate the dataset's outlier data points quickly and determine what went wrong with them.
The next block in line is the NN Classifier block. Here you can control the number of training epochs, the rate at which the weights of the linkages between neurons are changed, and the percentage of training dataset samples that are used for validation. Additionally, if needed you can even change the structure of the neural network.
After clicking on Start Training, a random value between 0 and 1 is assigned to the weight of each link between the neurons that make up the neural network. Then, the NN is fed the Training data set gathered during the data acquisition phase and the classification output is compared to the correct results. The algorithm then adjusts the weights assigned to the links at a rate defined in the Learning rate field and then compares the results once more. This process is repeated for a number of epochs, defined by the Number of training cycles parameter.
At the end of this process, the Classification Neural Network will be tested on a percent of the samples from the Training dataset held on the side for validation purposes.
Accuracy represents the percentage of predictions in which the result coincides with the correct value and Loss represents the sum of all errors made for all the samples in the validation set.
Underneath those performance indexes, the Confusion matrix presents in a tabulated form the percent of samples that were miscategorised. In our case, 31.3% of Ink_Leakage data points were mislabeled as Normal. This comes with a low rate of false positives as an advantage, but also with a low sensitivity to the phenomena it’s trying to detect.
Finally, the Feature explorer displays all the data from the Training dataset on a 2-axis graph and allows users to quickly determine what data points are outliers and trace back to their source by clicking on them and finding out why a misclassification might occur.
To extract relevant power and frequency characteristics of the accelerometer signal we will be using a Spectral Analysis block. Here you have the option to add low-pass or high-pass filters to remove unwanted frequency from the signal, to scale the axes and to modify the FFT length. It is worth spending some time on this signal processing block until the results are acceptable. A good rule of thumb here is that for similar input signals you must obtain similar processing results.
After clicking Save parameters, you will be redirected to the Generate features tab where you must make sure to check the Calculate feature importance option before pressing on Generate features.
The Anomaly Detector block is a great way of detecting any anomalous behavior of the machinery during its runtime. Click on Select suggested axes to greatly increase the performance of this NN and to reduce its resource usage, as it will only take into account the features identified in the previous step.
For our application, the algorithm identified the X RMS, X Skewness, X Kurtosis, and X Spectral Power 0.41 – 1.22 Hz as the most important features. This algorithm groups similar data points in a predefined number of clusters. Then, a threshold value is detected based on which an area is defined around each of those clusters. When doing an inference, the NN computes the distance from the center of a cluster to the new datapoint and if it falls outside a cluster, aka the distance between the nearest centroid and the datapoint is greater than the threshold value, that datapoint is registered as an anomaly.
The user can see how the neural network performs when presented with data that it has never seen before using the Model Testing page. After clicking Classify all, Edge Impulse will then feed the neural network with all the data in the Testing data pool and display the classification results and model performance, just like during the training process of the NN.
Another way of testing out your model is to use the Live Classification function of the Edge Impulse Platform. This tab enables the users to validate the machine learning model using data captured directly from any connected device, giving them a great overview on how it will perform when deployed on the edge. This is great to check if the device is mounted accordingly on the machine it monitors or if it has all the conditions it needs to run optimally.
The solution presented above allows you to schedule maintenance before the problem occurs, instead of waiting for something to break. Additionally, predictive maintenance can improve safety by identifying potential hazards before they cause an accident. In our case detecting unusual vibration in the printer and solvents or ink leaks can keep the workers safe and the workflow uninterrupted. The combination of the Sony Spresense and the CommonSense board can cover a wide array of use cases, we have only scratched the surface with the accelerometer and the gas sensor.
For more information about implementing predictive maintenance in your professional project, reach out to us: edgeimpulse.com/contact
