A Deep-Learning Based Semi-Interactive Method for Re-colorization Tengfei Zheng PB20000296 July12,2023
A Deep-Learning Based Semi-Interactive Method for Re-colorization Tengfei Zheng PB20000296 July 12, 2023
Contents Contents Abstract 1 Introduction 1 1.1 Colorizing Gravscale Images 1.1.1 Background. 1 11) Image Representation 1 1.2 Two Views of Colorization 2 12 1 From certain color style 2 122 From Given Points 2 Style Transferring Methods 4 2.1 Pixel-wise LUT 4 2.1.1 Description of Content and Style 21 2 Look-Un Table 6 2.1.3 RGB Matching Results 2.1.4 YUV Matching 2.2 Wavelet Methods.. 9 2.2.1 Frequency. 2.2.2 Wavelet Transform 9 2.2.3 Soften the Result 2.3 Edge Detection 12 2.3.1 Convolution Methods 13 2.3.2 Transforming with edge orientation information 14 3 Colorization by Optimizing 15 3.1 Continuity Preserving Methods 15 3.11 Mask 。。。 15 3.12 RGB Optimizing 16 3.1.3 Poisson Results 3.2 Optimizing on YUV Space 19 3.2.1 Loss Function on YUV 3.2.2 YUV Optimization Results 4 Introduce of CNN 22 4.1 Colorization Nets 2 4.1.1 Basic Classification..... 。。。。。。。。。。。。。。。。。。 4.1.2 Plain Network Constructions........................ 4.1.3 Colorize by GAN..·.·.···. 4.2VGG-l9 and Gram Matrix.·········· 2 4.2.1 VGG-19.... 4.2.2 Representation of Metrics ........................ 4.2.3 Result generation.............................. I
Contents Contents I Abstract III 1 Introduction 1 1.1 Colorizing Grayscale Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.2 Image Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Two Views of Colorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.1 From Certain Color Style . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.2 From Given Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2 Style Transferring Methods 4 2.1 Pixel-wise LUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.1 Description of Content and Style . . . . . . . . . . . . . . . . . . . . . 4 2.1.2 Look-Up Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.3 RGB Matching Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.4 YUV Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Wavelet Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.1 Frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.2 Wavelet Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.3 Soften the Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Edge Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3.1 Convolution Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3.2 Transforming with Edge Orientation Information . . . . . . . . . . . . 14 3 Colorization by Optimizing 15 3.1 Continuity Preserving Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.1.1 Mask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.1.2 RGB Optimizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.1.3 Poisson Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2 Optimizing on YUV Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2.1 Loss Function on YUV . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2.2 YUV Optimization Results . . . . . . . . . . . . . . . . . . . . . . . . . 20 4 Introduce of CNN 22 4.1 Colorization Nets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.1.1 Basic Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.1.2 Plain Network Constructions . . . . . . . . . . . . . . . . . . . . . . . . 24 4.1.3 Colorize by GAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2 VGG-19 and Gram Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2.1 VGG-19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2.2 Representation of Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.3 Result Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 I
4.3 Implementation of Transferring 4.3.1 Inserting Loss Layers 4.3.2 Some Improvements 29 4.4 Results and Semi-Interactive Methods. 4.4.1 Content Weights...... 4.4.2 Style Weights 4.4.3 Semi-interactive Colorization. 34 5 Conclusion and Discussion 盼 5.1 More Details. 35 5.1.1 Large Datasets and Large Models..................... 5.1.2 Judging Results........................,..,.. 5.2.1 52.2 Future Enhancements.,..,..,·,..,,·,·,········· References 38 Appendix
4.3 Implementation of Transferring . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.3.1 Inserting Loss Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.3.2 Some Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.4 Results and Semi-Interactive Methods . . . . . . . . . . . . . . . . . . . . . . . 31 4.4.1 Content Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.4.2 Style Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.4.3 Semi-interactive Colorization . . . . . . . . . . . . . . . . . . . . . . . . 34 5 Conclusion and Discussion 35 5.1 More Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 5.1.1 Large Datasets and Large Models . . . . . . . . . . . . . . . . . . . . . 35 5.1.2 Judging Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.2.1 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.2.2 Future Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 References 38 Appendix 39 II
Abstract Aiming at the problem of colorizing grayscale images.this paper concludes commonly used algorithms about style-transferring and traditional colorization methods,indicating different views of re-colorization:by a similar image or by given colored points,both of which gives rise to an interactive method Ways to solve stvle-transferring problem include pixel-wise color transform,freqt image (source image)we some eparts is already kn vn, optimiza on to fill othe parts of the re by the orinciple that points having shor and bris ld also be oth views nee ed prior info ation abo th of the ecomes a n By intr ained c thi s the limitation while views to a teractiv proac T the method to c situatio fferent col orization nets leading to more appealing res As a conclusion,this paper mpares the features of different views and methods,analyzing how to apply them automatically by modifying network construction
Abstract Aiming at the problem of colorizing grayscale images, this paper concludes commonly used algorithms about style-transferring and traditional colorization methods, indicating different views of re-colorization: by a similar image or by given colored points, both of which gives rise to an interactive method. Ways to solve style-transferring problem include pixel-wise color transform, frequency methods and edge-detecting methods. Given a grayscale image (target image) and an RGB image (source image), we can use these methods to transfer the color onto target image. Otherwise, if the color of the target image in some parts is already known, we can introduce optimization to fill other parts of the image by the simple principle that points having short distance in space and brightness should also be close in color. However, both views need prior information about the image, especially about the semantic information of the image. Convolution Neural Network (CNN) is expertise in identify such information, so using neural networks becomes a natural choice. By introducing pretrained deep learning models, this paper discusses the limitation of conventional methods, while synthesizing both views to a full semi-interactive approach. To extend the method to automatic situations, this paper studies the structure of different colorization nets, and applies some integration and improvements on them, leading to more appealing results. As a conclusion, this paper compares the features of different views and methods, analyzing how to apply them automatically by modifying network construction. III
Course Paper L.-K.Hua Seminar Jmly12,2023 1 Introduction 1.1 Colorizing Grayscale Images 1.1.1 Background aphs were in black and white Co of old hem fo re- colorization restoring As figure 1 she omare original mg,grayscale mage and bya certain agorithm 协办边 Figure 1:Sample re-colorize results Nevertheless,colorization is a challenging problem,for we have to find a fixed work p dure that can cove o and cuds are hite to reonie th hih onditi for instanc Before taking a look at traditional or deep learning methods,we must first describe the in language of math and programming 1.1.2 Image Representation The grayscale image is often represented as a matrix,each pixel of which transformed into an integer value (called intensity)between 0 and 255,meaning the brightness at that point,see figure 2. 107104 9875 intensity Figure 2:Grayscale representation 1
Course Paper L.-K. Hua Seminar July 12, 2023 1 Introduction 1.1 Colorizing Grayscale Images 1.1.1 Background Before color photographic technology became widespread, all photographs were in black and white. Coming to the problem of restoring old photos, how to fill color on them forms an important part. What is more, re-colorization technology can deal with distortion caused by different light conditions, restoring the real color of the image. As figure 1 shows, the three columns are original images, grayscale images and re-colorized images by a certain algorithm. Figure 1: Sample re-colorize results [1] Nevertheless, colorization is a challenging problem, for we have to find a fixed work procedure that can cover varying image conditions. Although scene semantics are usually helpful, for instance, grass is green and clouds are white, to recognize these high level semantics is always a difficult task. Before taking a look at traditional or deep learning methods, we must first describe the problem in language of math and programming: 1.1.2 Image Representation The grayscale image is often represented as a matrix, each pixel of which transformed into an integer value (called intensity) between 0 and 255, meaning the brightness at that point, see figure 2. Figure 2: Grayscale representation [2] 1