Using MCycleGAN to Realize High-Resolution Reconstruction of Solar Speckle Image

Wenhao Cui¹, Murong Jiang¹, Pengming Fu¹, Lingxiao Zhu¹, Lei Yang²

¹School of Information, Yunnan University, Kunming, China.
²Yunnan Observatory, Chinese Academy of Sciences, Kunming, China.

DOI: 10.4236/jcc.2020.811009                

Abstract

High-resolution reconstruction of solar speckle image is one of the important research contents in astronomical image processing. High-resolution image reconstruction based on deep learning can obtain the end-to-end mapping function from low-resolution image to high-resolution image through neural network model learning, which can recover the high-frequency information of the image. However, when used to reconstruct the sun speckle image with single feature, more noise and fuzzy local details, there are some shortcomings such as too smooth edge and easy loss of high-frequency information. In this paper, the structure features of input image and reconstructed image are added to CycleGAN network to get MCycleGAN. High frequency information is obtained from structural features by generator network, and the feature difference is calculated to enhance the ability of network to reconstruct high-frequency information. The edge of the reconstructed image is clearer. Compared with the speckle mask method level 1+ used by Yunnan Observatory, the results show that the proposed algorithm has the advantages of small error, fast reconstruction speed and high image clarity.

Keywords

Solar Speckle Image, High-Resolution Reconstruction, MCycleGAN

1. Introduction

Due to the influence of atmospheric turbulence and atmospheric disturbance, the solar activity observation image acquired by ground-based optical telescope will be seriously blurred or degraded, which needs image restoration method for reconstruction. The speckle imaging method based on the statistics of each order of the speckle map and the multi-frame blind deconvolution and phase difference method based on the deconvolution instantaneous point spread function are the two mainstream technologies for the reconstruction of high-resolution images of the sun [1]. In the reconstruction process, more prior knowledge is needed, such as atmospheric visibility, speckle interference function [2]. In addition, a large number of image frames are required, resulting in a large amount of calculation in the reconstruction process, which cannot meet the real-time business processing requirements of astronomical observations. As deep learning is widely used in the fields of computer vision and image processing, combining machine learning methods to reconstruct solar speckle images and improve solar observation data analysis and processing capabilities have become one of the hot research issues in solar observation image processing.

High resolution image reconstruction based on deep learning is mainly the reconstruction of a single low resolution image. The end-to-end mapping function from low-resolution image to high-resolution image is obtained by neural network model learning, and the image to be built is reconstructed by image conversion. The representative network models are SRCNN, DRCN, ESPCN, VESPCN and SRGAN [3]. Their implementation methods are: first, the low resolution image is sampled and interpolated to get the same size as the high-resolution image, and then convolution calculation is performed as the network input to reconstruct the high-frequency in-formation from the low-frequency components. These networks are good for the reconstruction of low resolution images with obvious features and a large amount of information, but they are not ideal for the reconstruction of solar speckle images with single characteristics, more noise and fuzzy local details. The Generative Adversarial Network (GAN) is a type of neural network composed of a generative model and a discriminative model. It learns by allowing the two models to play against each other to achieve Nash equilibrium. Compared with other networks, GAN can generate clearer and more realistic samples, but it has problems such as unstable training and easy gradient disappearance [4]. The cycle generation confrontation network CycleGAN is composed of two mirror-symmetric GANs, which can convert information from one form of expression to another form of expression, and learn the corresponding transformation function by minimizing the loss [5]. It has good stability, but it will also be affected by the color or contrast to make the reconstructed image distorted.

Aiming at the characteristics of low contrast of sun spots, similar rice grain shapes, and small differences between frames, this paper uses the VGG network to extract the structural features of the input image and the reconstructed image [6], these structural features are added to the CycleGAN network as MCycleGAN. The feature difference is used to obtain the high frequency information of the input image, and the calculation of feature loss is combined to improve the network’s ability to reconstruct the high frequency information of a single frame image. The results show that this method has the advantages of small error, fast reconstruction speed and high image clarity.

The content of this paper is arranged as follows: In the second part, detailed description of MCycleGAN network structure; In the third part, introduce the construction of the data set in this article and the training process of MCycleGAN, and compare the reconstruction results and fusion results with other algorithms to verify the feasibility of the method in this article; In the fourth part, the methods in this paper are summarized.

2. Reconstruction Method

In this paper, the structural features of the input image and the reconstructed image are added to the Cycle-GAN network and denoted as MCycleGAN.

In CycleGAN, because Lcyc only calculates the pixel by pixel difference between the input image and the reconstructed image, this simple pixel matching method makes the ability of cap-turing high-frequency details in CycleGAN very limited. In order to make better use of the advantages of CycleGAN and at the same time improve the edge information ability of the network to reconstruct images. The loss function was proposed in the CycleGAN network. Proceed as follows:

2.1. First Step: Extract Structural Features

The deep-seated features of the image are extracted by using VGG19 network [5], as shown in Figure 1. Among them, x, y represent the input image, (y), G(x) represents the reconstructed image.

2.2. Second Step: Using Feature Structure

The extracted deep feature map is added to CycleGAN network, and the generator network is used to obtain high frequency information from the extracted feature map. The calculation of feature loss is combined to improve the network’s ability to reconstruct the high frequency information of a single frame image.

Loss function L_per is obtained by weighting characteristic loss L_con and resistance loss L_GAN, which is defined as formula (1)-(3) [6] [7].

Figure 1. Feature map extracted from VGG19.

$L_{per}=\alpha L_{con}+\beta L_{GAN}$ (1)

$\begin{array}{c}{L}_{con}\left(X,Y\right)=E_{x~X}\left({\Vert G_{{\theta }_G}\left(F\left(y\right)\right)-{\phi }_{i,j}\left(X\right)\Vert }_2\right)\\ +E_{y~Y}\left({\Vert G_{{\theta }_G}\left(G\left(x\right)\right)-{\phi }_{i,j}\left(Y\right)\Vert }_2\right)\end{array}$ (2)

$L_{GAN}\left(G,D_Y,X,Y\right)=E_{y\text{~}Y}\left(\log D_Y\left(y\right)\right)+E_{x~X}\left(\log \left(1-D_Y\left(G\left(x\right)\right)\right)\right)$ (3)

Among them, means extracting and reconstructing image structural features, ${\phi }_{i,j}$ means extracting structural features of input image. L_GAN is still composed of two least squares-based adversarial losses $L_{LSGAN}\left(G,D_Y,X,Y\right)$, $L_{LSGAN}\left(G,D_X,Y,X\right)$ composition.

The whole loss function is composed of characteristic loss, resistance loss and cyclic consistent loss, which is defined as formula (4)

$\begin{array}{c}L\left(G,F,D_X,D_Y\right)=L_{per}\left(G,D_Y,X,Y\right)+L_{per}\left(F,D_X,Y,X\right)\\ +\lambda L_{cyc}\left(G,F,X,Y\right)\end{array}$ (4)

CycleGAN includes generator network G and discriminator network F. In the generator network G, G(x) represents the generated reconstructed image, and y represents the reference image; in the generator network F, F(y) represents the reconstructed image, and x represents the reference image; the loss function L_cyc is defined as shown in formula (5) [5]:

$L_{cyc}\left(G,F\right)=E_{x~X}\left({\Vert F\left(G\left(x\right)\right)=x\Vert }_1\right)+E_{y~Y}\left({\Vert G\left(F\left(y\right)\right)=y\Vert }_1\right)$ (5)

2.3. MCycleGAN Network Structure

The network framework of MCycleGAN is shown in Figure 2, in which the generator and discriminator are the same as CycleGAN, and the feature loss Lcon is defined as the Euclidean distance between the features of the original high-resolution image and the reconstructed image.

Figure 2. MCycleGAN framework.

In the framework of MCycleGAN, the mapping functions G(x) and F(y) are periodic consistent: for each image x from domain X, the image conversion period should be able to return x to the original image, that is, $x\to G\left(x\right)\to F\left(G\left(x\right)\right)\approx x$, which is called forward cyclic consistency. Similarly, for each image y from the domain Y, G and F satisfy reverse cyclic consistency: $y\to F\left(y\right)\to G\left(F\left(y\right)\right)\approx y$.

The MCycleGAN and CycleGAN were used to reconstruct the solar spot map, and the results are shown in Figure 3.

As can be seen from Figure 3, the image reconstructed by MCycleGAN has better reconstruction effect around the edge of detail texture restoration, and there is no too smooth and fuzzy situation. Much high-frequency information is completely recovered, and the subjective visual effect is better, which greatly improves the shortcomings of CycleGAN.

3. Experimental Analysis

In this paper, the observation images provided by Yunnan Observatory are used as data sets. Firstly, the reconstruction network MCycleGAN is trained respectively, and then the sun speckle pattern is reconstructed by using the trained network. The results are compared with the level 1+ algorithm.

3.1. Training Data Set

The solar speckle map has the characteristics of low contrast, similar shape of rice grains and small difference between frames. At present, there is no similar public data set to use. Therefore, this paper selects the high-resolution solar speckle image in March 2019 provided by Yunnan Observatory to form two data

Figure 3. Comparison of reconstruction results. (a) Original image; (b) CycleGAN; (c) MCycleGAN.

sets. Data set a includes: each 1280 × 960 solar speckle low resolution image is divided into 80 grid blocks, a total of 440,000 pieces, each sub block pixel size is 256 × 256; data set B includes: each 2360 × 1910 sun spot high resolution image is divided into 80 grid blocks, a total of 44,000 pieces, each sub block pixel size is 256 × 256.

3.2. Training Network

MCycleGAN network training:

In the training process, the optimization algorithm uses Adam optimizer to update the batch size of 1, the exponential decay rate β1 of control weight allocation is set to 0.5 by default, the initial learning rate of generator is set to 0.00002, and the initial learning rate of discriminator is also set to 0.00002. The super parameters of Lcon and LGAN are α = 1, β = 10⁻³. Data set A and data set B are input into MCycleGAN network during training. The learning rate of the network remains unchanged in the initial training. After repeated training, the learning rate tends to zero in a linear way. When the learning rate reaches the preset value, the training can be stopped, and finally the mapping relationship between the two data sets is obtained, that is to say, the sun spot image in data set B should be consistent with the original solar speckle image after the generator, and the model is generated by this mapping relationship for reconstruction.

3.3. Analysis of Reconstruction

Due to the large size of the sun spot observation image, using all the images to reconstruct will lead to a large amount of calculation, which cannot meet the actual requirements. Therefore, this paper uses the power spectrum frame selection method to select 20 initial images from the sunspot image and the sun spot image to reconstruct.

In order to verify the effectiveness of this method, this paper selects several latest super resolution image reconstruction methods (DRRN [8], LapSRN [9], SRGAN [10] ) for comparison, and uses the selected 20 frames of sun spot images for reconstruction, and compares them with level 1+. The results are shown in Figure 4.

As can be seen from Figure 4, the reconstruction result of this method is

Figure 4. Comparison of reconstruction results.

better than other net-works and closer to level 1+.

In order to evaluate the network more accurately, SSIM and PSNR are used to evaluate the network. The results are shown in Table 1.

It can be seen that the PSNR and SSIM values of the reconstructed network in this paper are better than those of other reconstructed networks.

In order to verify the effectiveness of network reconstruction in this paper, this paper has carried out experiments on three widely used benchmark datasets set 5 [11], set 14 [12] and bsd100 (test set of bsd300 [13] ) and the results are shown in Figure 5.

It can be seen from Figure 5 that the reconstructed network in this paper is better in details and clearer than other networks.

Evaluate the network by observing CycleGAN, SRGAN and LapSRN, the network in this article under the same sample and the convergence of the loss function during the training process. (Among them, the DRRN network is not compared with it due to the poor reconstruction effect). The result is shown in Figure 6.

As shown in Figure 6 and Table 2, the MCycleGAN network in this article has a smoother convergence of the loss function during training and the convergence speed is better than other networks.

After all the sub-blocks are reconstructed, these sub-blocks are spliced together. This article has a pixel-level splicing algorithm that uses the pixel value of each sub-block to splice each sub-block according to the pixel position. Since the overlap between the sub-blocks is used during splicing, the problem of partial splicing lines is reduced. The result is shown in Figure 7.

4. Conclusions

In this paper, MCycleGAN network is used to reconstruct a single frame

Table 1. Objective evaluation of reconstruction network.

Figure 5. Reconstruction results and corresponding reference original images.

Figure 6. Network training convergence process.

Table 2. Total loss data.

Figure 7. Solar spots after stitching.

high-resolution solar speckle image. Combined with converged network, multi-frame fusion is carried out to make up for the missing high-frequency information in single frame reconstruction, so that the reconstructed image has more detailed information, closer to the target image, and the reconstruction result is better than level 1+.

This method has the following characteristics: MCycleGAN network is constructed by introducing feature loss into CycleGAN network, which realizes the generation countermeasure network based on cyclic consistency loss and perception loss, improves the perception ability of image high frequency information and the recovery ability of detail features, and further enhances the effect of image reconstruction.

Due to the limitation of the resolution of the sun spot image, the source image cannot be used in the reconstruction, so the source image must be divided into blocks, and then stitched after the reconstruction is completed. Sometimes, the resulting image will have the problem of splicing, which needs further improvement.

Acknowledgements

This work is supported by the Natural Science Foundation of China (No. 11773073) and the Program for Innovative Research Team (in Science and Technology) in the University of Yunnan Province (IRTSTYN).

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Xiang, Y., Liu, Z. and Jin, Z. (2016) High Resolution Solar Image Reconstruction Method. Progress in Astronomy, No. 1, 94-110
[2]	Huo, Z. and Zhou, J. (2010) Method of Astronomical Image Reconstruction from Speckle Pattern. Progress in Astronomy, No. 1, 74-94.
[3]	Wu, R. (2020) Image Super Resolution Reconstruction Based on Deep Learning. University of Chinese Academy of Sciences (Institute of Optoelectronic Technology, Chinese Academy of Sciences).
[4]	Goodfellow, I.J., Pouget-Anadie, J., et al. (2014) Generative Adversarial Networks. Advances in Neural Information Processing Systems, 3, 2672-2680.
[5]	Simonyan, K. and Zisserman, A. (2014) Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 [cs.CV]
[6]	Tang, C., Li, J., et al. (2019) Unpaired Low-Dose CT Denoising Network Based on Cycle-Consistent Generative Adversarial Network with Prior Image Information. Computational and Mathematical Methods in Medicine, 12, 1-11. https://doi.org/10.1155/2019/8639825
[7]	Ledig, C., Theis, L., et al. (2017) Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 574-591. https://doi.org/10.1109/CVPR.2017.19
[8]	Tai, Y., Yang, J. and Liu, X. (2017) Image Super-Resolution via Deep Recursive Residual Network. IEEE Conference on Computer Vision & Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 112. https://doi.org/10.1109/CVPR.2017.298
[9]	Wei, S., Lai, J., et al. (2017) Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. IEEE Conference on Computer Vision & Pattern Recognition (CVPR), Hawaii Convention Center, 618.
[10]	Ledig, C., Theis, L., Huszar, F., et al. (2017) Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. IEEE Conference on Computer Vision & Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 137. https://doi.org/10.1109/CVPR.2017.19
[11]	Bevilacqua, M., Roumy, A., Guillemot, C. and Alberi-Morel, M.L. (2012) Low-Complexity Single-Image Super-Resolution Based on Nonnegative Neighbor Embedding. Proceedings British Machine Vision Conference 2012, 135.1-135.10. https://doi.org/10.5244/C.26.135
[12]	Zeyde, R., Elad, M. and Protter, M. (2012) On Single Image Scale-Up Using Sparse-Representations. Curves and Surfaces, Volume 6920 of Lecture Notes in Computer Science, 711-730. https://doi.org/10.1007/978-3-642-27413-8_47
[13]	Martin, D., Fowlkes, C., Tal, D. and Malik, J. (2001) A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. IEEE International Conference on Computer Vision (ICCV), Vancouver, 7-14 July 2001.