> For the complete documentation index, see [llms.txt](https://lauradang.gitbook.io/notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://lauradang.gitbook.io/notes/machine-learning/research-papers/convolutional-neural-networks-for-automatic-image_colorization.md).

# Convolutional Neural Networks For Automatic Image Colorization

The paper can be found [here](https://pdfs.semanticscholar.org/5a71/e3ada938c6c45a87988670118291b3028df6.pdf).

Other resources: [Automatic Colorization](https://tinyclouds.org/colorize/)

## Abstract

* Fully automatic image colorization
* Feed-forward, two-stage architecture based on CNN that predicts U and V colour channels
* SUN database for images
* Quaternion Structural Similarity (QSSIM) for quantitative evaluation

We formulate image colorization as a **regression** problem and CNNs are used to solve the problem.

## Model Architecture

![](https://868646840-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LztfBhQUrZzyA7O_ZkJ%2Fuploads%2Fgit-blob-2887add9b65e522856dae76cee9368044d68b251%2Farchitecture.png?alt=media)

Similarly to Larsson et al. paper, semantic information and localization are important to this problem. Hence, this paper also uses VGG-16 since it is a pretrained model incorporates huge amounts of semantic information as it was trained on a dataset with >1 million images.

**Input:** 224x224 RGB. Our image is grayscale though, so we need to concatenate the image one after the other 3 times.

**Layers in VGG-16 that were used**:

* All layers before the 3rd max pooling layer.
* These layers were merged and upscales to the same size as the input: 224x224x451 matrix denoted with $$T$$. Shown in the figure above, $$T$$ is then converted into UV channels.
  * Two more convolutional layers are output from the merged VGG-16 layers (no max pooling layers since we want the output image to have the same size as the input image)
  * These two layers are concatenated, now we have a new 224x224x144 matrix called $$Q$$.

**The CNN that we're actually training**:

**Input**: Matrix $$T$$

**Target**: $$UV\_i$$ (predicted UV values)

**Optimizer**: Stochastic Gradient

* learning rate of 8×10−2
* weight decay of 8×10−7
* step decay of 0.5 for every 300 epochs.

**Loss Function**: Since Euclidean loss was too saturated they used **cross-entropy**

$$L(UV\_p, UV) = \frac{-1}{WH}\Sigma\space UV\space log(UV\_p)$$

* $$W$$ is width of input image
* $$H$$ is height of input image

**Training Dataset**: ILSVRC 2012 classification dataset

* 500.000 images selected randomly from database then converted the training color images into greyscale images
* For each image, the U and V channels were calculated

**Calculating UV**:

```bash
Y = R *  .299000 + G *  .587000 + B *  .114000
U = R * -.168736 + G * -.331264 + B *  .500000 + 128
V = R *  .500000 + G * -.418688 + B * -.081312 + 128
```

##