Overview / Usage

The can inspection project was multidisciplinary project in which I developed the fully functional PoC solution allowing for inspecting cans corruptions. The project touches many different aspects related to bringing intelligent systems to life. In order to do so, I created high-level software on linux allowing for control of Hikvision MV-CE013-50GC camera and image classification using C++ with opencv and tensorflow. I also introduced integration with Microsoft Azure and Google Drive allowing for data storage. I programmed Arduino as a communication node between system sensors (IR gate, cameras), system computer and system pneumatic actuator. I was responsible for designing the customized lighting for the system and whole system stand itself. I took major role in the mechanical and electronic integration of the system in the factory. This project is still furthered developed as a VisionQb in TechOcean Sp. z o.o.

Methodology / Approach

The first step was to create the proper environment allowing to create good photos of the pretty demanding object(clean can), especially in terms of reflections. This is why at the beginning I did mostly research regarding different lighting systems used in industrial Quality Assurance solution. From that research, I learnt that the best solution would be dome diffuse light. However, after contacting some companies selling this kind of equipment it turned out that the cost of this kind of lamp is from around 400-1000 euros, it was clear to high. This is why I decided to try to produce this kind of lamp but by using really strong LEDs(2912lm/m) and 3D printed elements. I designed the model using Autodesk Fusion 360. After mounting the lamp it turned out that it plays its role perfectly.

Next step was to prepare the solution allowing for making automatically and repeatedly photos of cans moving on the factory line. As a camera we decided to use Hikvision MV-CE013-50GC. By using dedicated API, I was able to configure the camera to create good quality photos of moving cans. By having this solution we were ready to gather the dataset of photos to build the first model. Additionally, as the factory was about 120 km from our headquarters, I configured the Google Drive API that allowed us for uploading data to the cloud and accessing it remotely.

As we were not able to always get the can positioned on the photo at the same place (related to the triggering method and the fact that there was no possibility to have feedback loop from the line speed), I was forced to create the pre-processing solution allowing to cut the can from the image. I was able to do it quiet successfully using the opencv and Canny algorithm for contours detection. After this, at adjusting photos to network_input size 227x227, I was able to produce the model using Microsoft Custom Vision and exporting the model to frozen_graph.pb for tensorflow.

After getting the model, testing it and getting satisfactory results on one side of the can, we start to further develop solution. We added two more cameras for top and another side of the can and we also add the actuator allowing to push the wrong cans from the line. The algorithm used for the top of the can did not use any ML model, it turned out that the simple solution allowing to detect circularity of the can was enough as there were not different defects considered.

All in all the flow of the system was following: can triggers IR gate --> Arduino gets info that can 1 is in --> Arduino triggers camera 1 --> camera 1 grabs can 1 photo and sends it to computer via ethernet --> computer pre-process the photo and loads it to tensorflow model --> if the photo is bad it sends info to Arduino --> ...analogous for cam 2(no model, circularity algorithm) and cam 3... --> if at any of position Arduino got that the photo is bad it triggers actuator The main trick was to handle triggers of up to 5 cans simultaneously.

There were many additional challenges and tricky obstacles related to this project, but it was extremely satisfying as at the end I was really able to make it WORK.

Technologies Used

• Gigabyte Brix with Intel Celeron N2807
• Arduino
• Hikvision MV-CE013-50GC camera
• Hikvision MV C++ API
• Linux Debian Stretch 9
• C++
• python
• bash
• SQL
• TensorFlow C++ API
• Google Suite (Google Drive API)
• Microsoft Azure (IoT Hub, Stream Analytics, SQL Database)
• Microsoft Custom Vision
• Autodesk Fusion 360

Other links

• https://techocean.pl/en/home/