Face sketch colorization using conditional normalizing flow

• Author / Creator

  Pi, Ruiqin

• To colorized a face sketch involves adding appropriate color to the original sketch while maintaining the sketch's structure. This task combines the concept of gray-scale image colorization, which focuses on introducing color elements without changing the image's geometry, and image translation, which adds missing luminance and chrominance information to transform a sketch to a color image. However, existing works for gray-scale image colorization tend to produce satisfactory colorization results only when they are provided with a rich and detailed gray-scale image as input. In contrast, sketches have much less information because of its minimalistic nature and the absence of shading or texture details, which makes the colorization process more challenging, as it requires the model to not only add the necessary color elements but also infer the missing details based on the sketch's structure. Some researchers tend to address this problem using image translation techniques. State-of-the-art approaches using Generative Adversarial Networks (GANs) based models are preferred due to their impressive performance and realistic results. Nevertheless, the semantic difference between synthesized images and their corresponding input sketches remains a key challenge that needs further investigation. Recently, normalizing flow, which is a new generative model, with ability to losslessly encode and decode data, has shown potential to tackle the face sketch colorization problem.

  In this thesis, we first focus on converting sketches and their corresponding color images to high-dimensional latent features using a conditional normalizing flow within an Encoder-Decoder architecture. This framework adopts an inverse learning mechanism to simultaneously learn the color encoding and decoding processes. To ensure preserving geometry of the sketch while maintaining high-quality colorization, we integrate a Feature Aggregation module into the coupling block of our normalizing flow. This module can adaptively fuse sketch and color information within a high-dimensional space. Moreover, our normalizing flow is designed to be memory-efficient, which requires much less computational resources compared to that of similar models. We also demonstrate the effectiveness of our method to preserve the input geometry and to achieve perceptually satisfying colorization quality using both qualitative and quantitative evaluations comparisons with existing methods. This research contributes to advancing the field of face sketch colorization, and providing an innovative solution that addresses some of the limitations of existing approaches.

• Subjects / Keywords

  • Normalizing flow

• Graduation date

  Fall 2023

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-8j15-3w52

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.