Development of a Deep Learning Model for Inversion of Rotational Coronagraphic Images Into 3D Electron Density

We present, for the first time, a deep learning model that returns the three-dimensional (3D) coronal electron density from coronagraphic images. The intensity of coronagraphic observations arises from the Thomson scattering of photospheric light by the coronal electrons. We use MHD numerical simulations to obtain realistic 3D electron density and construct error-free training sets consisting of input (observation) and target (electron density) images. In the training sets, the input images are directly synthesized from the target 3D electron density by applying the Thomson scattering theory. The input and target images are in the form of latitude–longitude maps given at a radius, often referred to as synoptic maps. Using synoptic maps reduces a tomographic method to an image translation problem. We use pix2pixHD, one of the well-established supervised image translation methods and develop models for six selected heights: 2.0, 2.2, 2.5, 4.0, 6.0, and 12.0 solar radii. All six models have similar performance and the mean absolute percent error of the generated density images is less than 7% with respect to the ground-truth simulated data sets.


Introduction
Deep learning is a state-of-the-art technique that has been established to overcome the limitations of algorithms based on human perception. This method has been dramatically improved, and many attempts have been made in various domains, such as linguistics (Wu et al. 2016), genomics (Zou et al. 2018), and medical image analysis (Ker et al. 2017). However, its inference is not understandable by humans, and thus it has been often considered as a black box. Such a black box could be a severe drawback when using them for scientific purposes. Nevertheless, we would like to point out that deep learning results can be considered as empirical solutions even to mathematical problems. In this regard, they could be of great strength in solving physical problems that are not strictly defined mathematically.
In the present Letter, we present an attempt to apply a deep learning method to solving a physical problem. In particular, we present a model that extracts the three-dimensional (3D) information from a set of two-dimensional (2D) solar coronagraphic images. Such methods are often referred to as tomography (e.g., Herman 2009). In ideal situations, e.g., radiology in the medical field, this method uses analytical solutions, and thus the calculation is fast and stable. Similarly, solar X-ray or EUV imaging observations obtained for a solar rotation can be used for such a robust technique (e.g., Hurlburt et al. 1994;Cho et al. 2020, and references therein). However, the coronagraphic observation is an artificial total eclipse in which the Sun's disk is occulted, and the observed values are not the simple line-ofsight (LOS) integration. In this regard, an iterative reconstruction must be used (e.g., Frazin 2000; Butala et al. 2010;Kramar et al. 2014). However, an iterative reconstruction requires huge computing power and long computing times so that it has not been widely used for scientific purposes (cf. Kwon et al. 2013).
We describe results of a deep learning image translation technique that replaces the iterative process in the tomography methods. The Generative Adversarial Network (GAN; Goodfellow et al. 2014) is used to synthesize realistic images (e.g., Isola et al. 2016). GAN proposes a supervised method that simultaneously trains the generator and discriminator that lead to the adversarial process. The generator generates artificial images from input images, and the discriminator attempts to determine whether given images are from the generator or from the training set. The training process continues until the discriminator becomes unable to judge. Isola et al. (2016) presented a conditional GAN (cGAN) for an image translation. Unlike some of previous techniques doing per-pixel translation (references can be found in Isola et al. 2016), cGANs learn a structured loss, enabling each pixel of an output image to be dependent on the values on a large area of the input image. This set of source code is widely known as pix2pix, and later pix2pixHD, after modification by Wang et al. (2017) for high-definition (HD) images. The results of their image translations are shown in Isola et al. (2016) and Wang et al. (2017), including realistic images synthesized from outline drawings, labels, and maps. Inspired by these promising results, Kim et al. (2019) and Shin et al. (2020) used these methods to generate pseudo-solar photospheric magnetograms from Sun disk images observed in an extreme ultraviolet passband and Ca II K passband, respectively. We use pix2pixHD and develop a model that returns electron density from coronagraphic observations. The Letter is organized as follows. In Section 2, we show how the inversion (tomography) method can be reduced to an image translation problem that can be done by the pix2pixHD technique. The concerns and their mitigation are also given in this section. Section 3 shows the method to construct the training sets in practice and the model evaluation with the test sets. Section 4 gives concluding remarks, including future works.

Method
We develop a deep learning model that extracts the 3D information from a 2D observational image, using one of the well-established supervised methods known as pix2pixHD . Pix2pixHD is the same as the parent method pix2pix (Isola et al. 2016) in that the methods perform image-to-image translations as trained by input-target image pairs. Pix2pix uses a cGAN that determines each pixel value of the output image as a part of structures on the input image. We have adapted the codes in Shin et al. (2020) that generate pseudo-magnetograms from Ca II K images. 5 As when using a supervised method, the training set consisting of input and target images defines the problem that the model solves. The model aims to derive the 3D electron density from polarized brightness values in coronagraphic images. The observed intensity in coronagraphic images results from Thomson scattering of photospheric light by coronal electrons. It has long been established that the emission from the K-corona, i.e., free electrons, is polarized while from others, including the F-corona and noise, are not. Coronagraph observations use three or four filters to detect linearly polarized light at different angles, and these three or four sequence observations convert into a single polarized brightness (pB) image. Traditionally, the intensity in coronagraphic images is calibrated to the customary units of mean solar brightness. As a result, a pB value at each pixel results from the distribution of the number of electrons along the LOS. Our model uses the pB images as input, and the target images are the electron density corresponding to the input image domain.
To generate error-free input-target image pairs for the training set, 2D input images are synthesized from 3D electron densities given by numerical simulations with the Thomson scattering theory (Billings 1966). An observed pB value is written as the integration of the volume electron density N e [cm −3 ] over a given LOS, The function Λ(ρ, s) describes the Thomson scattering geometry, characterized by ρ and s, including the constants such as the Thomson scattering cross section and the limbdarkening coefficient. As shown in Figure 1(a), ρ is the distance from the electron to the Sun center on the plane of the sky (POS), and s is the distance from the electron to the POS. r is the distance of the electron from the Sun center (r = r + s 2 2 ). The full expression of Λ(ρ, s) can be found in previous literature, e.g., Billings (1966), Howard & Tappin (2009), and Wang & Davila (2014).
As indicated by this equation, the Thomson scattering geometry determines the strength of scattering for an electron and serves as a weighting function. Figure 1(b) shows the shape of function Λ as a function of s for the cases where ρ = 2.0, 2.5, 3.0, and 6.0 R e (solar radii). It shows that the maximum occurs on the POS (s = 0) for a given ρ, and the maxima vary with ρ. There is a tendency that Λ decreases with ρ, and the profile gets broader. It indicates that an observed pB value is not simply proportional to the total number of electrons over the LOS. Once a 3D electron density and the viewing geometry are given, the pB image can be synthesized using Equation (1).
In fact, there are an infinite number of solutions of the 3D structure for a given 2D observational image. It indicates that original observational images cannot be directly used for our task. As the typical tomography method does, one can solve this problem by adding a number of different LOS observations that can constrain one another and exclude inapposite solutions. For the input and target images, we use latitude-longitude maps at a given height (ρ), which are often referred to as synoptic maps. These synoptic maps contain pB or N e on a spherical surface of radius ρ, so that the dimension of the image corresponds to latitude θ (−90° θ 90°) and longitude f (0° f 360°). This map could be given in Stonyhurst coordinates or Carrington coordinates depending on the selection of the reference longitude. By using the synoptic map, an input image contains observations from different viewing directions and becomes physically related to the output image. In addition, the weighting function Λ becomes only a function of s since ρ is fixed in a synoptic map. An input-target image pair in the form of the synoptic map has a one-to-one correspondence with a cause-and-effect relationship and allows us to have a physically reasonable solution (see below for a more detailed description). Figure 1(c) illustrates how an input pB synoptic map can be constructed from observed (synthesized) pB images. When B0 angle (heliographic latitude of the observed solar disk center) is neglected and the solar north (rotational axis) is up on the image plane, the pB values are taken along the semicircle on the image plane as shown by the yellow curve (ρ = ρ′). These values compose a column (0° θ 180°) of the synoptic map, at f = f 0 − 90°, where f 0 is the observer's longitude. A pB synoptic map is completed once these pB values are taken for a solar rotation, i.e., 0° f 360°(see Figure 2 for examples of pB and the associated N e synoptic maps).
Figure 1(d) shows a plane parallel to the x-y plane (seen from north). The concentric circles refer to the cross sections of the spherical surfaces they make with a number of LOSs obtained during a solar rotation. More specifically, a set of the LOSs at ρ = ρ′ and θ = θ′ for a solar rotation forms a circle with the radius p (=r q¢ sin ) as shown with a thick line in this panel. A pB value at the given viewing geometry is the integration of Λ(ρ, s) · N e (s) over the LOS as shown by the dashed line. The corresponding target N e (s = 0) is taken at the point of contact between the LOSs and the concentric circle. These pB and N e values compose a row (0° f 360°) of their synoptic maps at θ = θ′. Consequently, a pB synoptic map includes a number of different LOS observations that constrain our model to have a physically reasonable 3D solution.
We point out that our inversion method can be approximated to a deconvolution problem. Let us make a bold assumption that the radius p of a circle in Figure 1(d) is so large (ρ is large) that N e (s) along the LOS is similar to N e (f(s)) along the circle, when |s| is close to 0. In this limit, the pB values obtained in succession for a solar rotation are approximately the result of the convolution, Λ (f) * N e (f). Thus, the inversion can be treated as a deconvolution problem. Our assumption may not be too far from reality because the function Λ peaks at s = 0 and thus an observed pB value largely originates from N e near the POS. For the cases where ρ is small (p is small), the assumption still remains valid because Λ drops sharply away from the POS. This indicates the existence of a clear cause-and-effect relationship between input and target images. Moreover, this relationship is evident from the similarity between pB and N e synoptic maps in the upper panel of Figure 2. The lower panel shows the maps when the density structure is relatively complicated, and the relationship may not be found by intuition. We expect that the image translation technique, pix2pixHD provides us with empirical solutions to such a quasi-deconvolution problem, by training the model with a number of input-target image sets.
In addition to the physical solution problem described above, there are some concerns about the accuracy of the inversion when using real observations. First of all, the dynamic range of the coronal electron density and that of pB values in the low corona are very large, and it may affect the model performance and introduce large uncertainties in N e . Second, the absolute values of the electron density and their range in numerical simulations may differ from those in the true corona. In this respect, it is not guaranteed that our model would have the same performance when using true observations.
Note that the use of the synoptic maps mitigates these concerns and removes the consequent ambiguities. While the gradient in N e occurs with the radial distance in general, the synoptic map is only at a selected height, ρ = ρ′. In addition, we use logarithmic values for both pB and N e synoptic maps. Next, we use the normalized values in order to make our model independent of their absolute values and also the ranges. In practice, we convert these images into 8-bit byte-type images in a JPEG format, in which their minimum and maximum values are 0 and 255, respectively.

Results and Discussion
Our model aims at translating the pB images into N e values by training the model with a number of pairs of pB (input) and N e (target) synoptic maps. For the sake of error-free training sets, we have obtained the input-target image pairs from MHD simulations that is the thermodynamic version of the "Magnetohydrodynamics    (Mikić et al. 2018). Note that it is not necessary that density structures and values are strictly realistic for this purpose, since we are only seeking empirical solutions to a quasi-deconvolution problem, as discussed in Section 2. We expect that such a realistic model enables efficient model training.
For data augmentation, we used the synoptic maps reversed in longitude and latitude and shifted by 45°on the longitudinal direction in succession. As a result, the number of training sets increases by 32 times. The original image size of the synoptic maps is 360 × 181, but the image size in the model is 256 × 256. As done by Wang et al. (2017), we have trained the model by iterating 200 epochs. Figure 2 shows two examples (ρ = 4.0 R e ) among the 60 test sets in which the density distribution is relatively simple (upper panel) or complicated (lower panel). The left panels are the pB synoptic maps that are synthesized by the Thomson scattering theory (Section 2) from the 3D electron density. The middle panels are the N e synoptic maps directly taken from the 3D electron density, i.e., the ground truth. The generated N e synoptic maps are given in the right panels. It seems that the model reproduces well the density structures in the synoptic maps for both simple and complicated cases. Figure 3 provides detailed comparisons of the generated N e synoptic maps with the ground truths. In panels (a) and (b), the normalized error ( ¢ N e − N e )/N e , where ¢ N e is the electron density generated by the model, is indicated by the color bars. It is evident from these maps that N e is determined with errors less than 20% overall. Panels (c) and (d) show the pB (black), generated ¢ N e (green), ground-truth N e (red) along the equator as shown by the black line in panels (a) and (b), respectively. As discussed in Section 2, the pB and N e values are logarithmic and normalized to 0-255. It is clearly seen that the generated ¢ N e profiles tend to be smoother and simpler than the groundtruth ones. The larger error occurs in the region where the density changes rapidly and where the density profile has multiple peaks. It is noteworthy that such error distributions seem to be the same as expected from a deconvolution method, while using a deep learning method. Meanwhile, larger errors are seen in the south pole. Note that the polar region is spatially distorted in the synoptic map, and the true area is much smaller than that seen in these maps. Panels (e) and (f) show that the generated ¢ N e is well correlated with the ground-truth N e , indicating that the model reproduces well the density structure and value even when the density structure is complicated. Figure 3 shows two examples of the test sets at ρ = 4R e . Since our model uses the synoptic map at a given ρ, the full 3D density can be constructed by combining the N e synoptic maps at different ρ. Since the weighting function in Equation (1) is dependent on ρ, the model at different ρ should be trained independently. Supposing that our model will be applied to Solar Terrestrial RElations Observatory (Kaiser et al. 2008) Sun Earth Connection Coronal and Heliospheric Investigation (Howard et al. 2008) COR1 (FOV of 1.5-4R e ) and COR2 (2.3-15R e ), we have selected six ρ, and the first column in Table 1 shows ρ selected in this work.
To find the best models and evaluate them, we determine five parameters and their averages over the 60 test sets at each ρ. The five parameters are (1) correlation coefficient (CC), (2) mean square error (MSE), (3) structural similarity index (SSIM; Wang et al. 2004), (4) slope of linear regression, and (5) mean absolute percent error (MAPE). SSIM quantifies the similarity between two images, and the perfect case is 1. MAPE is the ratio of the difference in percent between two images , and zero value means perfect. It is clearly seen in this table that CC, SSIM, and slope are close to 1 for all ρ. MAPE demonstrates that our models reproduce the N e synoptic maps with errors less than 7%. Table 1 demonstrates that the models at different ρ have similar performance. It is noteworthy that the best models for all ρ are found earlier among 200 epochs (second column in Table 1). We interpret that it is because of the clear cause-and-effect relation between the input and target images.

Concluding Remarks
We have presented, for the first time, the use of a deep learning method to determine the 3D electron density from coronagraphic observations. By using synoptic maps, we reduced such a complex iterative tomographic process to an image translation problem. Besides the technical issues overcome by using synoptic  Note. See Section 3 for details. maps described in Section 3, variance in the coronal electron density during a solar rotation is an intrinsic source of the error in solar rotational tomographic methods (Hurlburt et al. 1994). In the synoptic map, the longitude difference (distance in the longitude direction) corresponds to the difference in time. Thus, our model is also subject to the uncertainty caused by the temporal coronal evolution. However, it is noteworthy that the weighting function Λ of Thomson scattering could reduce this error while it was an obstacle to applying coronagraphic images to a tomography method. The shape of Λ in Figure 1(b) indicates that the electron density away from the image plane (or at a large distance in the longitude direction in the synoptic map) has less of an effect on the observed brightness. Thus, the error should be less than that of general solar rotational tomographic methods. Furthermore, our fast deep-learning-based inversion method offers a way to study the temporal evolution of the coronal electron density. Since our method is fast, multiple N e synoptic maps can be obtained in Stonyhurst coordinates by shifting the reference date. Such timeseries N e synoptic maps will reflect the temporal evolution of the coronal electron density structures. Before the use of our models for scientific purposes, the full 3D electron density should be reconstructed from the synoptic maps composed of the normalized values. Nevertheless, our results even in this current form can provide information related to the global coronal magnetic field topology that fully relies on models and numerical simulations. While coronal magnetic fields themselves are invisible, the density structures are the manifestation of the magnetic structures, e.g., heliospheric current sheet (Szabo et al. 2020, and references therein). Our model results can assist numerical simulations (e.g., Riley et al. 2019) in interpreting data obtained by spacecraft traveling in the inner heliosphere, i.e., Parker Solar Probe (Fox et al. 2016) and Solar Orbiter (Müller et al. 2020), by comparing our output density synoptic map with their in situ measurements of densities. Also our density distributions can be used to constrain or validate global magnetohydrodynamic models reproducing large-scale coronal and heliospheric structures, such as the Wang-Sheeley-Arge (Arge et al. 2004) and MAS (Lionello et al. 2009) models. In future work, we will show the full 3D reconstruction retrieving actual electron density values from the normalized and layered electron densities generated by our models.