Ulyanov et al, CVPR 2018, [link]
tags: image analysis - generative models - cvpr - 2018
Given a random noise vector and conditioned on an image , the goal of conditional image generation is to generate image (where the random nature of provides a sampling strategy for ); for instance, the task of generating a high quality image from its lower resolution counterpart .
In particular, this encompasses inverse tasks such as denoising, super-resolution and inpainting that acts at the local pixel level. Such tasks can often be phrased with an objective of the following form:
where is a cost function and is a prior on the output space acting as a regularizer. is often a hand-crafted prior, for instance a smoothness constraint like Total Variation , or, for more recent techniques, it can be implemented with adversarial training (e.g.,
In this paper, the goal is to replace by an implicit prior captured by the neural network, relatively to input noise . In other words
Which results in the following workflow:
One could wonder if this is a good choice for a prior at all. In fact, , being instantiated as a neural network, should be powerful enough that any image can be generated from for a certain choice of parameters , which means the prior should not be constraining.
However, the structure of the network* itself effectively affects how optimization algorithms such as gradient descent will browse the output space:
To quantify this effect, the authors perform a reconstruction experiment (i.e., ) for different choices of the input image ((i)** natural image, (ii) same image with small perturbations, (iii) with large perturbations, and (iv) white noise) using a
U-Net  inspired architecture. Experimental results show that the network descends faster to natural-looking images (case (i) and (ii)), than to random noise (case (iii) and (iv)).
Figure: Learning curves for the reconstruction task using: a natural image, the same plus i.i.d. noise, the same but randomly scrambled, and white noise.
The experiments focus on three image analysis tasks:
The method seems to outperform most non-trained methods, when available, (e.g. Bicubic upsampling for Super-Resolution) but is still often outperformed y learning-based ones. The inpainting results are particularly interesting, and I do not know of any other non-trained baselines for this task. Obviously performs poorly when the obscured region requires highly semantic knowledge, but it seems to perform well on more reasonable benchmarks.
Additionally, the authors test the proposed prior for diagnosing neural networks by generating natural pre-images for neural activations of deep layers. Qualitative images look better than other handcrafted priors (total variation) and are not biased to specific datasets as are trained methods.
Figure: Example comparison between the proposed Deep Image Prior and various baselines for the task of Super-Resolution.
Heckel and Hand, [link]
This paper builds on Deep Image Prior but proposes a much simpler architecture which is under-parametrized and non-convolutional. In particular, there are fewer weight parameters than the dimensionality of the output image (in comparison, DIP was using a
U-Netbased architecture). In particular, this property implies that the weights of the network can additionally be used as a compressed representation of the image. In order to test for compression, the authors use their architecture to reconstruct image for different compression ratios (i.e., number of network parameters , is -times smaller as the output dimension of the images).
The deep decoder architecture combines standard blocks include linear combination of channels (convolutions ), ReLU, batch-normalization and upscaling. Note that since here we have a special case of batch size 1, the Batch Norm operator essentially normalizes the activation channel-wise. In particular, the paper contains a nice theoretical justification for the denoising case, in which they show that the model can only fit a certain amount of noise, which explains why it would converge to more natural-looking images, although it only applies to small networks (1 layer ? possibly generalizable to multi-layer and no batch-norm)