Overview / Usage

Image in-painting is the process of completing masked region with a possible content. In augmented reality, our aim is to include virtual objects into reality using markers (or marker less). However, in diminished reality (DR) we want to remove objects from the image and replace it with another content. One of the applications of DR is Google Street in which private citizens faces, house numbers, military facilities, etc. must not appear because it violates privacy. In the literature, there are many image completion (in-painting) techniques, most of them are patch-based methods; e.g.: PatchMatch [Barnes et al, 2009], Image Melding [Darbi et al, 2012], Fast Marching [Telea et al, 2004], etc. These methods search the image for possible patches around the missing region (small textures of size S*S) and propagate it to fill in it, but it is not effective to fill in holes (missing regions) on complicated structures, also they are unable to generate novel objects not found in the image.

To fill in large complicated structured regions, Generative Models are one of the reasonable choices because it showed a huge success on many applications like Image-To-Image Translation [Isola et al, 2016], Cycle GAN [Eran et al, 2017], etc. in which two Convolution Neural Networks (CNNs) are completing against each other in a mini-max game, the first one is called a Generator (modeled by an Auto-encoder) which takes an image as an input and produces an image with a specific style as an output, e.g. given a zebra image to the generator it generates a horse image, or a black and white image to a colored image, etc. The discriminator tries to distinguish real images (in our case: images including the missing region) from generated images (plausible missing region). After too many epochs, the discriminator will be unable to distinguish real from generated images. In this way, we can generate complicated structures because the generator is trained enough to generate realistic looking images which cannot be distinguished from real ones. In the next section, we discuss the network architecture of the GAN used to solve the image in-painting problem and the training procedure.