Overview / Usage

In the field of Deep Neural Network and Deep Learning, it is commonly observed that it is very difficult for deep convolution neural network (DCNN) based semantic segmentation to estimate correct object boundary. We think that the reason lies in assigning labels for pixels on the object boundaries because the cascaded feature maps generated by DCNNs blur them. In order to segment foreground objects from background object the DCNN algorithm should classify the boundary pixel precisely.

This problem poses major hindrance to the accuracy in the semantic segmentation and it also leads to very high losses in ventures involving satellite imaging. Due to the involvement of very fine edges in ‘Satellite images’ missing them would lead to unbearable expenses and losses. Hence this becomes a major problem to be dealt with.

We solve this anomaly through the use of clustering methods. Clustering helps in Fine Edge Detection of the images and together with DeepLabV3 proves to be an indispensable tool for the solution.

After clustering the images, the segmentation of the Neural Network (here DeepLabV3) is enhanced by the use of our merging algorithm known as Disparting.

Methodology / Approach

Our main objective of the algorithm is to better out edge detection.

We address solving the segmentation problem by using different clustering methods for better segmentation which generates better segments of the input images especially around the edges which tends to either get left out or gets overshoot when segmented using neural networks. This, in turn, increases IoU value between the predicted result and Ground truth, and hence a more accurate and precise result is obtained.

The need for clustering arises to the fact that the output of the images generated by the DeepLabV3 or any other neural network in many cases tends to be inaccurate hence and clustering and our “disparting” method could solve this problem.Clustering is a technique for grouping the data points (in this case different pixel points) having similar features into different clusters. Then the curve corresponding to the boundary of these different clusters is used to differentiate the clusters. This boundary thus generated is used to define the borders accurately and hence these boundaries prove to be vital in our segmentation process.

The clustering methods like KNN clustering uses a very hard-line to differentiate the clusters but with this, the method does not take into account the little dissimilarities during clustering as there is no smoothing involved but here a more advanced approach is used to cluster the image. Here, kernel density estimation(KDE), a smoothing method is used; it works by placing a kernel-a weighting function that is useful for quantifying density on each data point in the data set and then summing the kernels to generate a kernel density estimate for the overall region. Areas of greater point density will sum out with greater kernel density, while areas of lower point density will sum out with less kernel density.

After this smoothing additional clustering methods which are felzenszwalb, SLIC and quick-shift are used to improve our segmentation

Another clustering method, Compact Watershed method is also used after the image is passed through a Sobel operator.

All these clustering methods prove to be useful in getting the segmented images of the DeepLabV3 neural network more accurate.

As observed the outputs of the DeepLabV3 Neural Network generally do not get the edges accurate. And this leads to inaccurate results in cases where there is a need for precise detection, especially near the boundary. So, at this juncture, the technique proves useful in removing this particular problem.

This clustering along with “disparting” generates well-segmented images, but for different types of images, different clustering methods with different kernel size and smoothing factors gives the best results. In cases where the edges of DeepLabV3 output are just slightly out of the correct shape at those cases a large number of clusters with smaller kernel size gives better result whereas, in the images where there is a large difference between the ground truth and DeepLabV3 output at these cases, larger clusters provide the best results.

Technologies Used

Here we use a class voting method named disparting, which basically merges the DeepLabV3 segmentation output and the output of the Clustering classes.

After the generation of segmented outputs from both the methods, our algorithm named Disparting is used to merge this and produce a final segmentation with better IoU than that done with only Deeplab output. This Disparting algorithm performs another pixel-wise traversal and comparison of the DeepLabV3 and clustered output producing a final merged image with better segmentation.

The algorithm is as follows

we have clustered_output(H, W) and deeplab_output(H, W)

for all a and b

for all a

Here,

a= number of cluster centers

b= number of segmentation classes in DeepLabV3

for j = 0:H-1 :

clusterStat(clustering(i,j), deeplab(i,j)) += 1

clusterSelect(i) = max_index(clusterStat(i,j)) of j for each i

for i = 0:W-1 :

for j = 0:H-1 :

finalSeg(i,j) = clusterSelect(clustering(i,j))

return finalSeg(W, H)

Our Disparting method is very useful in improving the edges and depicting the input image to more like the ground truth. There are different clustering algorithms that have different impacts on different images, thereby we have to tune for every image. In most cases, quick-shift stood out to have better results. Therefore, implementing this method of clustering on semantic segmentation would surely generate a segmented mask with better edges.

The post-processing of the segmented output of DeepLabV3 model, involving the clustering method to make the edges more distinct makes the approach a bit more CPU intensive but it’s worth that complexity because of increased similarity of the final segmented image thus produced with the expected output.

Even though this proposed method works in most of the cases but may fail in distinguishing the boundaries of the object from the surrounding background in a dark environment.

Repository

https://github.com/CYBORG-NIT-ROURKELA/Improving_Semantic_segmentation

Other links

• Medium File