Overview / Usage

Traditional techniques to obtain the 3D transformation to align two models are based on registration methods [1][2][3] of point clouds or meshes. Commonly, a registration process can be divided into three main steps: feature extraction; feature matching; and transformation compute to align both data sets. However, the accuracy of these methods depend on the quality of the input data, the extracted features, and the type of transformation (rigid/non-rigid, large/small, isometric/free).

In this scope, learning techniques can improve the traditional registration algorithms by generalization using large datasets for the training. The main idea is to improve or complement traditional solutions using higher level knowledge extracted from the data, and not using exclusively the extracted features. For instance, if a deflated ball is going to be registered with its inflated state, the method should have the knowledge that the result must be a spherical object. When a person perceives a deflated ball, the brain does not have an exact model of how the ball will look once inflated. Instead, the brain has a conceptual model with properties and knowledge about the behavior of the object and the material of which it is made. In this way the brain can have a rough idea of how the ball should look like.

Some authors have been published works in using convolutional neural networks to learn deformation flows, like Yumer et al. [4] paper, in which the CNN transforms the 3D model to satisfy a given semantic deformation, e.g. shorter, more sporty.

[1]. Myronenko, A., & Song, X. (2010). Point set registration: Coherent point drift. IEEE transactions on pattern analysis and machine intelligence, 32(12), 2262-2275.

[2]. Saval-Calvo, M., Azorin-Lopez, J., Fuster-Guillo, A., Villena-Martinez, V., & Fisher, R. B. (2018). 3D non-rigid registration using color: Color Coherent Point Drift. Computer Vision and Image Understanding, 169, 119-135.

[3]. Sotiras, A., Davatzikos, C., & Paragios, N. (2013). Deformable medical image registration: A survey. IEEE transactions on medical imaging, 32(7), 1153-1190.

[4]. Yumer, M. E., & Mitra, N. J. (2016, October). Learning semantic deformation flows with 3d convolutional networks. In European Conference on Computer Vision (pp. 294-311). Springer, Cham.

Methodology / Approach

The main objective of the project is to create a system that, given two sets of views from two objects (different state of a deformed shape), first the views are aligned to generate the two models in different states, and after to find the trasformation to non-rigidly align the two models. This is shown in the schematic of the figure.

Collaborators