独家优惠奖金 100% 高达 1 BTC + 180 免费旋转

Understanding Neural Style Transfer

Image to Image translation is a well-known problem that has been very widely researched in Deep Learning.

For those who don’t know image-to-image translation, it is an approach to translate an image from one domain into another domain. Eg. Day ➝ Night, Black-and-White Image ➝ Color Image, Sketch ➝ Image, etc.

One particular problem in the image-to-image translation is Style Transfer, wherein style from one image (style-image) is transferred to another (content-image).

So, how do we decide the loss function for the Style Transfer problem?

The main idea of this paper is that the style and content of an image can be represented separately in Convolutional Neural Networks. This allows us to combine the style representation of one image (style-image) and content representation of another image (content-image) to generate a new style-transferred image.

What does content and style representations really mean?

We know that in a CNN trained for object recognition, each layer of the network learns a representation of the image and these representations become more specific as we go deep into the layers. For example, the initial layers learn to detect edges and contours while the higher layers learn to detect some objects. This means that image content is better represented in higher layers of CNN, while the lower layers only provide the same pixel values. We use this content representation for calculating Content Loss.

In style transfer, we need the content representation of the content-image and generated-image to be the same. Let’s suppose that the output of a layer of CNN is given by ϕ(x). The Content Loss is simply the Euclidean distance between the content representation of the content-image and generated-image from a particular layer and is calculated as

Note: Cⱼ, Hⱼ, Wⱼ represent channels, height, and width respectively of the output of the jth layer.

Every layer of the Convolutional Neural Network provides a feature map as an output. For a CNN trained on object recognition, each channel in the feature map represents some aspect of the image eg. edges, circles, spirals, etc. There exist a correlation between different channels of these feature map. Taking these correlations into account from multiple layers, we obtain a multi-scale representation of input image that captures texture.

Using this technique we can obtain style representation of any image. Now to perform a proper style transfer, we need the style representation of the input image and style representation of the reference style-image to be the same. So the distance between these two style representations can be used as a loss which we need to minimize.

But how do we calculate the correlations and the distance between correlations?

Gram Matrices can be used for calculating the correlations between different channels of a feature map. Let’s suppose that the output of a layer of CNN is given by ϕ(x). Then the gram matrix for the feature map can be calculated as

Once we know how to calculate the correlation between features, now we can move forward to calculate the Style Loss. This loss is nothing but Euclidean distance between the gram matrices of style-image (yₛ) and generated-image (ŷ). Note that the paper mentions the Frobenius norm between the difference of gram matrices which is nothing but euclidean distance. Since we are using multiple layers, we sum the distance for each layer.

Content Loss and Style Loss together are termed as Perceptual Loss. To be exact Perceptual Loss is the weighted sum of Content Loss and Style Loss.

Using the losses we discussed earlier, now we need to set up the algorithm to generate the style transfer image. We use perceptual optimization for this task.

Perceptual Optimization takes in as input a white noise, style-image, and content-image. Our aim is to update white noise in such a way that it matches the style representation of the style-image and content representation of the -content-image. To do this, we calculate the Perceptual Loss i.e Style Loss between white noise and style-image (for matching style representation) and Content Loss between white noise and content-image (for matching the content representation). Once we get the loss we then backpropagate to calculate gradients and update the white noise. We repeat this until the perceptual loss converges to minima.
Note: Instead of white noise we can also use content-image.

The drawback of this algorithm is that we need to perform perceptual optimization from scratch every time for a new image. This is not efficient.

So, how can we make the process faster?

With this approach, we need to train the network only once. Once we have trained the transformer network we can now use it for style transfer. This takes considerable less time as compared to the previous approach.

However, we can only train the network only for one style-image. We need to train a new network from scratch for a different style-image. Training a completely new network is a time-consuming task.

Is it possible for a single network to learn all styles?

Yes, the answer is Conditional Instance Normalization.

In order to train a single model on multiple styles, we need to have a conditional network.

But where should we put our condition?

The normalization layer can be used to integrate our condition. Before we go into integrating condition lets review what the normalization layer does.

The normalization layer takes in the output features of the previous convolutional layer calculates the mean (μ)and standard deviation (σ) and standardized those features. The standardized features are then scaled and translated using learnable weights γ and β.

Now that we know how normalization works, we can move forward. It was found that we can use different learnable weights for different styles. Having different learnable weights for each style makes it possible for us to condition the network. By using different γ and β for different styles we are able to learn each style individually.

Since normalization only scales and translates the features, training an N-style transfer model requires fewer parameters than training N separate networks from scratch.

The perceptual results of the model are similar to single style-transfer networks.

Apart from being able to perform multiple style-transfer. The network also performs well on video input and provides results in real-time.

Learning Neural Style transfer helped me in understanding the working of each layer in Convolutional Neural Networks. Moreover, the techniques used for calculating the loss gave me a good knowledge of how texture can be viewed in Neural Networks. I hope that this post gives you a slight intuition for how the style transfer network works and explains to you the reason behind using the Perceptual Loss function.

Please feel free to share your thoughts regarding the content as this is my first ever post and help me in improving it. Thank You.

Add a comment

What is your view?

Send

Related posts:

Implementing Java Interfaces in Clojure

As the micro-service was written in Clojure, we had the challenge to implement quartz scheduler interfaces which are primarily written in Java. We also came across one of the implementations of…

Try Again

There is one thing we don’t give up on, and that is breathing. It might be labored or wheezy, but we keep breathing until we can’t anymore. I think that has to be, hands down, the only thing none of…

Islamabad Hot Girls

Are you looking for an unforgettable experience in Islamabad? Then you have come to the right place! We offer the best escorts and call girls in Islamabad to make your stay in the city as pleasurable…