Hi,

 

Today, I got progress on the software part of the project. I started searching for stylization / style transfer implementations and I found one that may be suitable for the Raspberry Pi.

 

Image stylization / style transfer is a process the can be described as "extracting the style" from an image and "applying it" to a second image.
An example would be to take a painting (ex take Vincent van Gogh's The Starry Night) and apply to a photo (Stanford campus):

(example from the neural-style GitHub Repo)

 

Two early techniques of image style transfer are described in the following articles:

 

 

 

I tried to run examples from the two implementation, but I quickly realized that these will not work for the project. Both techniques needs to train convolutional neural networks "on-the-fly" to apply a style to an input image. The process takes significant time (minutes on a powerful GPU, hours on CPU). This means there is no way to get this running on the Raspberry Pi.

 

Fortunately, the research was ongoing on the topic and some techniques for real-time stylization / style transfer were developed.

 

I found the the fast_neural_style example in the PyTorch repository which implements style transfer using pre-trained models. The example is based on the following two articles:

 

 

The example can be run as follows:

$ git clone https://github.com/pytorch/examples.git pytorch-examples

 

$ cd pytorch-examples/


$ pip3 install torch torchvision

$ python3 download_saved_models.py

$ python3 neural_style/neural_style.py eval --content-image images/content-images/amber.jpg --model saved_models/<model>.pth --output-image amber-candy.jpg --cuda 0


Here is what I got by running the examples:

  • input image
  • output images
    • "candy" model
    • "mosaic" model


    • "rain princess" model
    • "Udnie" model

 

The example run in ~ 6.5 seconds each on CPU, so they are relatively fast. I think, it can be ported to the Raspberry Pi and hopefully it will run in a reasonable amount of time.

 

Cheers,

Attila

 

Refs :