Deep learning for improving the spatial resolution of magnetic particle imaging

Yaxin Shang; Jie Liu; Liwen Zhang; Xiangjun Wu; Peng Zhang; Lin Yin; Hui Hui; Jie Tian

doi:10.1088/1361-6560/ac6e24

1. Introduction

Magnetic particle imaging (MPI) is a functional tomography technique based on tracers that can directly detect the concentrated distribution of superparamagnetic iron oxide nanoparticles (SPIONs) (Gleich and Weizenecker 2005, Weizenecker et al 2007). MPI has high contrast, high sensitivity, and excellent depth penetration with no background tissue signal, enabling it to be a potential molecular imaging technique for biomedical applications (Weizenecker et al 2009, Ferguson et al 2012, Haegele et al 2012, Murase et al 2015, Khandhar et al 2017, Knopp et al 2017, Yu et al 2017, Talebloo et al 2020, Tong et al 2021). Current applications of MPI involve vascular imaging (Vaalma et al 2017, Kratz et al 2018), cancer detection (Yu et al 2017), cell tracking (Zheng et al 2015), and perfusion mapping (Sedlacik et al 2016).

Spatial resolution is a key metric for evaluating the performance of MPI, which is primarily restricted through the performance of SPIONs and the hardware system. Previous studies have explored improving MPI resolution through different methods. Goodwill and Conolly (2010) showed that larger SPIO core sizes can improve MPI spatial resolution; however, the size expansions can lead to relaxation effects (Dhavalikar and Rinaldi 2014, York et al 2014, Zhao et al 2020, Tay et al 2021) developed superparamagnetic iron oxide nanoparticle chains to achieve an ideal stepped magnetization response and improve image resolution in MPI. Several studies have been focused on adjusting drive parameters to mitigate relaxation losses and improve spatial resolution (Kuhlmann et al 2015, Weber et al 2015, Croft et al 2016). However, there is a concomitant decrease in the MPI tracer sensitivity. Tay et al (2019) proposed a pulsed MPI to overcome relaxation effects; however, the trade-off between the scan time/signal-to-noise ratio (SNR) and resolution efficiency is untenable. Bagheri and Hayden (2020) presented a novel MPI signal acquisition and conditioning protocol that achieved significant breakthroughs in spatial resolution at the expense of image contrast. Overall, current techniques for refining the spatial resolution limit either tracer sensitivity, SNR, or image contrast. Moreover, improving the spatial resolution via increasing the gradient strength may not be feasible owing to the high-power consumption.

Recently, deep learning (DL) has yielded remarkable results in imaging applications (Noh et al 2015, Schmidhuber 2015, He et al 2016, Li et al 2018, Zhang et al 2018, El Naqa and Das 2020, Khan et al 2020) and has been proven efficient in improving image quality (Ronneberger et al 2015, Chen et al 2021, Fang et al 2021, Sheng et al 2021, Yang et al 2021). Chen et al (2021) presented a convolutional neural network (CNN) to optimize low-dose computed tomography (CT) images. Tripathi and Bag (2020) designed a new CNN (CNN-DMRI) to denoise MRI scans. Van Sloun et al (2021) designed deep ultrasound localization microscopy to obtain super-resolution vascular images. You et al (2020) proposed a semi-supervised method to precisely obtain high-resolution (HR) CT images. In the context of MPI, Hayat et al (2021) used k-means++ algorithm for segmentation of MPI images and perform iron quantization analysis. Sun et al (2021) used machine learning algorithms to monitor islet organ transplants labeled with superparamagnetic iron oxide nanoparticles and provide quantitative information about their presence in vivo. Notably, several deep learning algorithms have been also reported to improve the performance of MPI. A CNN-based super-resolution technique was proposed in Aşkın et al (2021) to obtain the HR system matrix from the LR system matrix of MPI. Baltruschat et al (2020) used deep learning to restore the small-size system matrix to large-size system matrix, which saved the time of calibration of the system matrix. Dittmer et al (2020) used deep image prior for iterative image reconstruction, according to the quantitative metrics of peak signal-to-noise ratio (PSNR) and structural similarity metrics (SSIM), the method achieves better performance than variational and iterative regularization techniques.

In this study, we developed a new fusion dual-sampling convolutional neural network (FDS-MPI) to overcome the low spatial resolution occasioned by hardware limitations and accomplish high-resolution reconstruction of MPI. To assess the performance of FDS-MPI, we first used simulated LR and HR MPI datasets to train the FDS-MPI model. Then, we conducted experiments on the effect of noise on the resolution, which further demonstrated the robustness and superiority of the proposed model for MPI images. Furthermore, we tested the DL-based method on real MPI scans, resulting in favorable outcomes.

2. Methodology

2.1. Problem setting

It was previously reported (York et al 2014, Khandhar et al 2017) that the MPI imaging equations are similar to the k-space theory of MRI, and the X-space theory was described mathematically in Goodwill and Conolly (2011), Goodwill et al (2012). Spatial resolution is a key metric for evaluating the MPI imaging performance, which can be characterized by the full width at half maximum (FWHM) of the point spread function. The FWHM equation is

$\begin{eqnarray}&&{\rm{FWHM}}=\displaystyle \frac{24T{k}_{{\rm{B}}}}{{\mu }_{0}\pi {M}_{sat}}{G}^{-1}{d}^{-3},\end{eqnarray} \tag{ 1 }$

where $T$ is Kelvin temperature; ${\mu }_{0}$ is the vacuum permeability; ${k}_{{\rm{B}}}$ is Boltzmann's constant; ${M}_{sat}\,\,$ is the saturation magnetization; $G$ is the gradient strength, and $d$ is the core size of the magnetic nanoparticles. This equation indicates that spatial resolution can be enhanced by improving $G.$

Low-resolution and high-resolution MPI images are obtained by setting low gradient intensity and high gradient intensity respectively. Then, a high-resolution image ( ${I}^{HR}$ ) is derived from the low-resolution one ( ${I}^{LR}$ ). The problem can thus be expressed as

$\begin{eqnarray}&&\mathop{{\rm{\arg }}\,{\rm{\min }}}\limits_{f}{\parallel f\left({I}^{LR}\right)-{I}^{HR}\parallel }_{2}^{2},\end{eqnarray} \tag{ 2 }$

where $f$ is regarded as a function of improving spatial resolution of the MPI image at a lower gradient strength and can be estimated using DL techniques.

2.2. Architecture of the proposed network

Based on two subnetworks, we propose a fast and simple network. As described in figure 1, the network with fusion dual-sampling comprises two different networks: branches A and B, both of which incorporate a popular encoder-decoder structure. A feature encoder network (FEN), a feature decoder network (FDN), residual learning (RL) (Drozdzal et al 2016, He et al 2016), rectified linear units (ReLUs), and a pooling layer (PL) were integrated into the designed network for model learning. Next, concatenate and convolutional layers were fused into dual-channel networks to improve performance. As shown in figure 1, branch A contains an auto-encoder architecture with a residual connection. Two PLs are added to extract representative information and simultaneously remove useless information; the MPI image has a large number of artefacts, and this system can remove unnecessary artefacts from the MPI image. In contrast to branch A, branch B has no PL. Consequently, useful information from the MPI image is not lost in branch B and can reconstruct high-resolution MPI images. The features extracted by the two branches are complementary in improving image resolution. The proposed network is detailed as follows.

2.2.1. Feature encoder network

A chain was developed of fully connected convolutional layers as convolutional encoders. Adding a PL (Ronneberger et al 2015) can reduce the computational cost of the network. Nonetheless, useful information is typically lost. Therefore, considering the balance between efficiency and accuracy, branch A was chosen as a PL to remove artefacts from MPI images. In branch B, we avoided using a pooling layer because this may discard important structural details.

2.2.2. Feature decoder network

Although the artefacts of MPI images are reduced after the convolutional layer, the convolution process may further diminish the structural details. Inspired by recent breakthroughs in biomedical imaging and semantic segmentation (Long et al 2015, Noh et al 2015, Ronneberger et al 2015, Drozdzal et al 2016), we included deconvolutional layers in the proposed network to recover structural details, which may be considered image reconstruction based on extracted features. We adopted fully deconvolutional layers to compose deconvolution decoders for image reconstruction. The convolutional and deconvolutional layers must have the same kernel size to ensure a match between the output and input of the proposed network. As shown in figure 1, this network framework has symmetry of paired convolutional and deconvolutional layers.

2.2.3. Residual learning

RL can make the network train quickly and effectively. In He et al (2016), the training and testing accuracy of the residual model was significantly improved. Accordingly, we introduced an RL method into the proposed network. RL is used in both branches (A and B). A skip-connection is incorporated in the proposed model to utilize RL.

The contribution of the aforedescribed FDS-MPI is three-fold: (1) It increases width rather than depth via two subnetworks to improve the resolution of the MPI images; (2) It applies RL to prevent the gradient from disappearing or exploding; and (3) It utilizes a PL to balance performance and computational cost.

2.3. Loss function

The mean square error (MSE) was taken as the loss function for the FDS-MPI model. Given X is a high-gradient-strength MPI image, and Y is its lower equivalent, MSE can be obtained:

$\begin{eqnarray}&&{\rm{MSE}}=\displaystyle \frac{1}{M\times N}\displaystyle \sum _{i=1}^{M}\displaystyle \sum _{j=1}^{N}X\left(i,j\right)-{\rm{\Phi }}{\left(Y\left(i,j\right)\right)}_{2}^{2},\end{eqnarray} \tag{ 3 }$

where ${\rm{\Phi }}$ represents a trained model or parameters, through which an estimated high-resolution MPI image $\hat{X}$ can be derived. The sizes of X and Y were ${M}\times {N}.$

2.4. Evaluation metrics

For a quantitative assessment, we used three metrics: root mean square error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM). RMSE is usually used for evaluating the quality of predicted images. PSNR and SSIM are commonly used for image enhancement or improving the fidelity of the image. RMSE is estimated as

$\begin{eqnarray}&&{\rm{RMSE}}=\sqrt{\displaystyle \frac{1}{m}\sum _{i=1}^{m}{\left({X}_{i}-\widehat{{X}_{i}}\right)}^{2}},\end{eqnarray} \tag{ 4 }$

where ${X}_{i}$ and $\widehat{{X}_{i}}$ are the means of the real and estimated high-resolution MPI images, respectively, and $m$ represents the number of images. PSNR is estimated as

$\begin{eqnarray}&&{\rm{PSNR}}=10{\mathrm{log}}_{10}\left(\displaystyle \frac{MA{X}^{2}}{MSE}\right),\end{eqnarray} \tag{ 5 }$

where MSE denotes the mean square error between the real and estimated high-resolution MPI images. SSIM is estimated as

$\begin{eqnarray}&&{\rm{SSIM}}=\displaystyle \frac{\left(2{\mu }_{X}{\mu }_{\hat{X}}+{t}_{1}\right)\left(2{\sigma }_{X\hat{X}}+{t}_{2}\right)}{\left({\mu }_{X}^{2}+{\mu }_{\hat{X}}^{2}+{t}_{1}\right)\left({\sigma }_{X}^{2}+{\sigma }_{\hat{X}}^{2}+{t}_{2}\right)},\end{eqnarray} \tag{ 6 }$

where ${\mu }_{X}$ and ${\mu }_{\hat{X}}$ are the means of the real and estimated high-resolution MPI images, respectively; ${t}_{1}$ and ${t}_{2}$ are constants; ${\sigma }_{X}^{2}$ and ${\sigma }_{\hat{X}}^{2}$ represent the variances, and ${\sigma }_{X\hat{X}}$ is the covariance between $X$ and $\hat{X}.$

3. Experiments and results

The performance of the proposed DL-based method is evaluated using simulated and real MPI scans, and the dataset, network training, and experimental results are discussed.

3.1. Dataset

3.1.1. Simulated data

In this study 10 000 images from the Mixed National Institute of Standards and Technology database (MNIST) dataset were selected and resized to $100\times 100$ pixels as simulated phantoms for MPI. To simulate the magnetization of magnetic nanoparticles in response to an excited magnetic field in MPI, we adopted the Langevin equation, as follows:

$\begin{eqnarray}&&M(r,t)={M}_{0}(r)\times \left[\coth \left(\frac{m| B(r,t)| }{{k}_{B}T}\right)-\frac{{k}_{B}T}{m| B(r,t)| }\right],\end{eqnarray} \tag{ 7 }$

where $B\left(r,t\right)$ is the total magnetic field at position $r$ and time $t;$ $m$ is the magnetic moment; and ${M}_{0}\left(r\right)=\tfrac{N(r)m}{{\rm{\Delta }}V}$ is the saturation magnetization of the particles contained in the volume ${\rm{\Delta }}V;$ $N(r)$ is the number of particles each contributing a magnetic moment $m=\tfrac{1}{6}\pi {D}^{3}{M}_{sat}.$ We used MATLAB toolbox to program the X-space reconstruction method (Shen et al 2022) as previously described in Goodwill and Conolly (2010, 2011) to generate simulated MPI images. The scan parameters in simulating MPI images were consistent with the configurations of a preclinical MPI scanner (MOMENTUM, Magnetic Insight, Alameda, CA, USA). All the parameters for the MPI simulation are listed in table 1.

Table 1. Parameters for the MPI simulation.

Symbol	Parameters	Value	Unit
${\mu }_{0}$	Permeability of vacuum	$4{\rm{\pi }}\times {10}^{-7}$	${\rm{N}}\,{{\rm{A}}}^{-2}$
${\rm{D}}$	Nanoparticle diameter	$20$	${\rm{nm}}$
${M}_{s}$	Saturation magnetization	$4.77\times {10}^{-5}$	${\rm{A}}\,{{\rm{m}}}^{-1}$
${m}$	Magnetic moment	$6.75\times {10}^{-18}$	${\rm{A}}\,{{\rm{m}}}^{2}$
${T}$	Kelvin temperature	$293.15$	${\rm{K}}$
${K}_{{\rm{B}}}$	Boltzmann constant	$1.38\times {10}^{-23}$	${\rm{J}}\,{{\rm{K}}}^{-1}$
${G}$	Gradient	$3\,(\mathrm{low}\,\mathrm{resolution}\,\mathrm{mode})\,\mathrm{or}$ $6\,(\mathrm{high}\,\mathrm{resolution}\,\mathrm{mode})$	${\rm{T}}\,{{\rm{m}}}^{-1}$
${\rm{FOV}}$	Field of view	$20\times 20$	${\rm{mm}}$

To prove the robustness of the proposed method, different levels of white Gaussian noise were applied to the simulated MPI images. Of these simulated MPI images, 8000 were used for training and 2000 for testing. In addition, a two-tube phantom with different distances was used to evaluate the spatial resolution of the MPI image. Each tube had a diameter of 1 mm, and the gap between the two tubes was set to 0.6, 1, and 2 mm.

3.1.2. Phantom data

To validate the effectiveness of FDS-MPI, MPI images of three phantoms, including two tube phantoms with different distances and one resolution phantom as described in OpenMPIData (Knopp et al 2020), were chosen as the testing data. All the phantoms were filled with Perimag SPIONs (Micromod Partikeltechnologie GmbH, Rostock, Germany). The tube phantoms had a diameter of 2 mm, and the distances between the tubes were set to 1 and 2 mm. Imaging of the phantoms was performed using a commercial MPI scanner (MOMENTUM, Magnetic Insight Inc., Alameda, CA, USA). The low and high resolution MPI images were acquired by setting the scan modes as high sensitivity (G = 3.0 T m⁻¹) and high resolution (G = 5.7 T m⁻¹), respectively.

3.2. Training of the network

We specially trained our network on simulated MPI images. During training, the patch size was $64\times 64$ with random cutting on datasets. In the experiments, the base learning rate started at 10⁻⁵ and was reduced by a factor of 0.5. The mini-batch size was set at 512. The number of epochs used to train the FDS-MPI models was 10 000. The convolution and deconvolution layers were initialized using a Gaussian kernel. The number of filters of the last layer was 1, whereas the number of filters of the other layers was set to 64. All kernels were 5 × 5 in size. The strides of convolution and deconvolution were 1 with 1 padding. In our work, the models were implemented in the open framework Pytorch with CUDA and CUDNN support. The model was run on a single NVIDIA Titan GPU. The network could process images of arbitrary sizes despite the training being performed in patches.

3.3. Experimental results

3.3.1. Simulated experimental results

We compared the proposed FDS-MPI with several other networks to analyze its performance. CNN consists of only convolutional layers for medical image processing (Chen et al 2017b). In residual encoder-decoder CNN (REDCNN), there are convolutional layers, deconvolutional layers, and a skip connection (Chen et al 2017a). UCNN and REDUCNN become neural networks by adding pooling layers in CNN and REDCNN. The reconstructed MNIST images obtained using the different methods are displayed in figure 2(a). The MPI spatial resolution improved with increasing gradient strength. We also evaluated the impact of RL on MPI image resolution. As shown in figure 2(a), the spatial resolution obtained via RL (REDUCNN, REDCNN) is better than that without RL (UCNN and CNN). Furthermore, we evaluated the effectiveness of the FDS model against that of the single-network model. With regard to resolution improvement, we observed that FDS-MPI surpasses single networks (UCNN, CNN, REDUCNN, and REDCNN). A comparison of the FWHM results is listed in table 2.

**Figure 2.** Simulated experimental results using different methods. (a) Different deep learning methods are used for enhancing the simulated low resolution MPI images (the first column) to compare with the simulated high resolution MPI images (the last column). (b) Normalized signal intensity profiles are plotted from the red lines in (a) for the simulated phantom 1, (c) 2 and (d) 3.
Download figure:
Standard image High-resolution image

Table 2. FWHM of the simulated experimental results via different methods.

	FWHM (mm)
Methods	Simulated phantom 1	Simulated phantom 2	Simulated phantom 3
UCNN	4.0	5.0	4.4
CNN	3.8	5.0	4.2
REDUCNN	2.6	4.4	4.0
REDCNN	2.8	4.0	3.2
FDS-MPI	2.6	3.2	2.6

Additionally, we visualized the loss and PSNR values in the training processes of different networks. The curves of the training loss and PSNR of 100 epochs are shown in figure 3. The first row displays PSNR values, whereas the second displays loss values. The first column shows the results of UCNN and REDUCNN, and the second shows the results of CNN and REDCNN. The model with RL products has better PSNR and loss values and converges more rapidly. The results of the third column show that the FDS model produced better PSNR and loss values compared to the single-network models. These findings show that our method can sharpen the spatial resolution of MPI images.

**Figure 3.** PSNR and loss curves of different training models.
Download figure:
Standard image High-resolution image

For a quantitative evaluation, the PSNR, SSIM, and RMSE of FDS-MPI were 28.11, 0.94, and 0.04, respectively, as listed in table 3. The results of all the evaluation metrics show that FDS-MPI can achieve the best performance compared with other single networks in improving the resolution of MPI images.

Table 3. Evaluation results for the simulated experimental data via different methods.

Methods	PSNR	SSIM	RMSE
UCNN	23.9762 ± 2.0114	0.8531 ± 0.0308	0.0650 ± 0.0159
CNN	25.4232 ± 1.9907	0.8819 ± 0.0194	0.0550 ± 0.0133
REDUCNN	23.6605 ± 1.6443	0.8608 ± 0.0237	0.0668 ± 0.0127
REDCNN	26.4208 ± 2.1710	0.9226 ± 0.0144	0.0493 ± 0.0189
FDS-MPI	28.1090 ± 2.3986	0.9430 ± 0.0127	0.0409 ± 0.0129

In MPI, system noise is inevitable during the acquisition of signals. This can consequently reduce the SNR as well as the resolution of the MPI image. At this point, we conducted experiments on the effect of noise on the resolution to analyze the robustness of the proposed method. We used arbitrary noise to train the proposed network. The sample results at three different noise levels between 20 and 40 dB are shown in figure 4(a). Different rows represent different noise levels. The first, second, and third rows illustrate MPI images corrupted with 40, 30, and 20 dB noise levels, respectively. As shown in figure 4(a), artefacts can still be found in the results reconstructed by UCNN and REDUCNN. Consequently, CNN and REDCNN cannot retain the structural features of the MPI image. In contrast, better robustness is shown in the FDS-MPI.

**Figure 4.** Simulated experimental results via different methods. (a) MPI scans are corrupted with 40, 30 and 20 dB noise levels. Different deep learning methods are used for enhancing the simulated low resolution MPI images (the first column) to compare with the simulated high resolution MPI images (the last column). (b) Normalized signal intensity profiles are plotted from the red lines in (a) for the corrupted MPI images with the noise level of 40 dB, (c) 30 dB and (d) 20 dB.
Download figure:
Standard image High-resolution image

For quantitative evaluation, the results obtained using the different methods are listed in table 4. At a noise level of 40 dB, the PSNR of FDS-MPI reached 26.81, while the best PSNR of the other networks was 25.20. Additionally, the RMSE and SSIM were significantly improved in FDS-MPI compared to other networks. At the 30 dB noise level, the PSNR of FDS-MPI reached 26.34, whereas the highest PSNR of the other networks was 25.12. The RMSE and SSIM were still significantly improved in FDS-MPI compared to other networks. At the 20 dB noise level, FDS-MPI obtained the best metrics. Thus, our method can effectively handle noise affected MPI images. Table 5 lists the quantitative results of resolution evaluation metrics.

Table 4. Evaluation results of simulated experimental results corrupted with different noise levels via different methods.

		Metrics
Noise levels	Methods	PSNR	SSIM	RMSE
40 dB	UCNN	23.7118 $\pm \,\,$ 1.8503	0.8228 $\pm \,\,$ 0.0281	0.0667 $\pm \,\,$ 0.0144
	CNN	24.3736 $\pm \,\,$ 1.7801	0.8246 $\pm \,\,$ 0.0324	0.0656 $\pm \,\,$ 0.0188
	REDUCNN	23.9864 $\pm \,\,$ 2.3178	0.8393 $\pm \,\,$ 0.0310	0.0617 $\pm \,\,$ 0.0127
	REDCNN	25.1996 $\pm \,\,$ 1.9737	0.8891 $\pm \,\,$ 0.0240	0.0564 $\pm \,\,$ 0.0128
	FDS-MPI	26.8062 $\pm \,\,$ 1.7467	0.8899 $\pm \,\,$ 0.0127	0.0466 $\pm \,\,$ 0.0100
30 dB	UCNN	23.6726 $\pm \,\,$ 1.8432	0.8135 $\pm \,\,$ 0.0380	0.0670 $\pm \,\,$ 0.0144
	CNN	24.4362 $\pm \,\,$ 1.7847	0.8183 $\pm \,\,$ 0.0180	0.0613 $\pm \,\,$ 0.0127
	REDUCNN	25.1181 $\pm \,\,$ 1.9567	0.8248 $\pm \,\,$ 0.0322	0.0569 $\pm \,\,$ 0.0128
	REDCNN	24.0027 $\pm \,\,$ 2.3075	0.8404 $\pm \,\,$ 0.0305	0.0654 $\pm \,\,$ 0.0186
	FDS-MPI	26.3443 $\pm \,\,$ 1.5834	0.8805 $\pm \,\,$ 0.0246	0.0490 $\pm \,\,$ 0.0095
20 dB	UCNN	23.5139 $\pm \,\,$ 1.8132	0.6512 $\pm \,\,$ 0.0690	0.0682 $\pm \,\,$ 0.0144
	CNN	24.4203 $\pm \,\,$ 1.7855	0.6558 $\pm \,\,$ 0.0344	0.0614 $\pm \,\,$ 0.0128
	REDUCNN	24.8097 $\pm \,\,$ 1.8955	0.8388 $\pm \,\,$ 0.0300	0.0588 $\pm \,\,$ 0.0128
	REDCNN	23.9863 $\pm \,\,$ 2.3178	0.8246 $\pm \,\,$ 0.0324	0.0656 $\pm \,\,$ 0.0188
	FDS-MPI	24.8704 $\pm \,\,$ 1.1851	0.8499 $\pm \,\,$ 0.0269	0.0576 $\pm \,\,$ 0.0084

Table 5. FWHM of simulated experimental results corrupted with different noise levels via different methods.

	FWHM (mm)
Methods	40 dB	30 dB	20 dB
UCNN	4.2	4.2	4.6
CNN	3.4	3.8	3.8
REDUCNN	4.0	4.0	3.2
REDCNN	4.2	4.4	3.8
FDS-MPI	2.6	2.6	2.8

To further evaluate the performance of the FDS-MPI approach with respect to spatial resolution improvement, we investigated the network via two tube phantoms. The reconstructed images obtained using the different methods are shown in figure 5(a). The signal-intensity curves of the different methods are shown in figures 5(b) and (c), facilitating a visual comparison of spatial resolution. The signal-intensity curves are the line scans (blue midpoint lines), as shown in figure 5(a).

For tubes with a gap of 0.6 mm, the two tubes reconstructed by UCNN and CNN in figure 5(a) are obscure, and the corresponding intensity curves in figure 5(b) have no peaks. This may indicate that UCNN and CNN would yield vague results at two-tube gaps of 1 and 2 mm. Although REDUCNN and REDCNN present two tubes in figure 5(a) and equivalent peaks in figures 5(b)–(d), artefacts are still present in the reconstructed scans, and the red and light-blue intensity curves are not congruent with the black intensity curve. However, FDS-MPI can clearly distinguish the two-tube phantoms at different distances. Furthermore, our method's intensity curve (pink curve) is more similar to the high-resolution intensity curve (black curve), indicating that the FDS-MPI method outperforms the other methods.

3.3.2. Phantom experimental results

We applied FDS-MPI to a real MPI phantom to further evaluate its performance. Accordingly, we scanned two-tube phantoms under high-sensitivity and isotropic modes. As shown in figure 6(a), the two rows represent the two tubes with distances of 1 and 2 mm, respectively. The first column is the color-image phantom taken during the scan of MPI data. The second column is the low-resolution image scanned by a high-sensitivity mode. The third to seventh columns represent the experimental results of UCNN, CNN, REDUCNN, REDCNN, and FDS-MPI, respectively. The eighth column shows the high-resolution image scanned in isotropic mode.

**Figure 6.** Dual tube phantom experimental results. (a) Two phantoms scanned by the MPI scanner are compared using different methods. (b) Normalized signal intensity profiles are plotted from the blue lines in (a) for a dual-tube phantom with gaps of 1 mm and (c) 2 mm.
Download figure:
Standard image High-resolution image

In the high-sensitivity mode (the second column), the two tubes are blurred to a lump. In addition, figures 6(b) and (c) show the signal-intensity curves of the different methods. The signal-intensity curves correspond to the line scans (orange midpoint lines) in figure 6(a). The two tubes reconstructed at a 1 mm gap via other methods are indistinguishable. However, for the tubes with a gap of 2 mm, the reconstructed images (FDS-MPI) in figure 6(a) show two distinct phantom tubes and their respective peaks in the pink intensity curves in figures 6(b) and (c). Conversely, the reconstructed images of the other methods show only vague phantoms of the two tubes. The image reconstructed by our network resembles the real high resolution MPI image (6 T m⁻¹ gradient). Likewise, the peak intensity curve of FDS-MPI is similar to the black intensity curve of the high-resolution image.

To analyze a more complex phantom, we supplemented the experiments with a 3D phantom. The experimental phantom was designed based on the resolution phantom of OpenMPIData (Knopp et al 2020). Because MPI is a tomographic technique, we processed each 2D tomography image, and then, the processed 2D images were composed into a 3D MPI image. This simplifies the computational cost and makes the network easy to train compared to processing 3D images directly. As shown in figure 7(a), we only show a 2D tomographic image. The first column is the color-image phantom. The second column is the low-resolution image scanned by a high sensitivity mode. The third column represents the experimental results of FDS-MPI. The fourth column shows the high-resolution image scanned in isotropic mode. The image after FDS-MPI processing is closer to the high-resolution image, and the artefacts are effectively removed. Figure 7(b) shows the signal intensity curves of the image profile (red lines) of figure 7(a). In figure 7(b), the pink intensity curve of FDS-MPI is similar to the black intensity curve of the high-resolution image. The 3D dynamic pictures are shown in video. This implies that the proposed method can improve the resolution of the 3D phantom image.

**Figure 7.** Experimental results of 2D tomography phantom of 3D image. (a) 2D tomographic image and the result processing by FDS-MPI; (b) signal intensity curves—extracted from the profile (red line) in figure 7(a).
Download figure:
Standard image High-resolution image

In addition, we also designed the in vivo experiments. Mice were scanned in high sensitivity and isotropic modes. Figure 8(a) shows the colour-image. Figure 8(b) shows the low-resolution image. Figure 8(c) shows the experimental results of FDS-MPI. Figure 8(d) shows the high-resolution image. In the low-resolution mode in mice, the spleen is nearly obscured by the liver. The results showed that the shape of the mice spleen was clearly displayed after processing by FDS-MPI (see the red arrows in figure 8). Therefore, our method works effectively in vivo data as well.

4. Discussion

The MPI imaging results are influenced by system performance, especially the spatial resolution. Hence, improving spatial resolution is crucial to precisely locating the distribution of tracer materials. To date, traditional methods of improving resolution stifle other imaging factors, such as sensitivity, image contrast, and SNR. Moreover, high-resolution MPI requires expensive high-end hardware, such as a high-gradient field, hence the high-power dissipation (Graeser et al 2019). This study introduced a novel FDS-MPI method to resolve the spatial resolution problem caused by hardware limitations.

The most crucial contribution of this research involves the deep-learning mechanism proposed to sharpen the spatial resolution. In this work, we performed simulations and phantom experiments to verify the performance of our proposed method. Based on the simulation, our method could effectively improve the spatial resolution of MPI images by doubling the theoretical resolution. Through a comparison against single-network techniques, FDS-MPI exhibited superior results; we combined RL with FDS when designing the network. Background noise is a problem with MPI. Many studies (Them et al 2015, Straub and Schulz 2018, Knopp et al 2021) focus on removing the background noise, and these methods have achieved good results. In our experiments, artificial noise was added to the simulated MPI images. We observed that our method improved the resolution and SNR of MPI images despite the addition of noise. In addition to the simulation, we applied our method to a real MPI system, where MPI phantoms were created using an MPI scanner (MOMENTUM, Magnetic Insightic Inc., Alameda, CA, USA), with Perimag magnetic nanoparticles used as tracer material. Using our method, high-quality images were reconstructed at a low gradient. To our knowledge, this research is the first application of DL to reconstruct high-spatial-resolution MPI images from a low gradient. We believe that a deep-learning-based approach is promising. Because this method includes image post-processing, and the MPI reconstruction algorithm used in the simulation is based on X-space theory, this method is applicable to X-space MPI scanners. Theoretically, our method can also be applied to MPI scanners from Bruker by constructing the training data based on a system matrix reconstruction algorithm. However, specific experimental research was not conducted, and we will carry out research in this field in the future. In addition, if the training data and training strategy is changed, we can attempt to use this method in other medical imaging modalities. However, the design of our network is aimed at MPI images. The combination of RL and FDS can effectively extract effective features of MPI and remove useless features, thus improving MPI resolution. The image features of other modalities are different from MPI data; thus, the network details may need to be modified to obtain dependable results.

Despite its many advantages, our study had several limitations. First, we could not obtain large quantities of real MPI data. Consequently, real MPI data were not added to the training dataset. Further, there were some differences between MNIST data and real MPI data. In future work, to solve the problem, we will use the powerful features of transfer learning (Pan and Yang 2009, Weiss et al 2016). First, we trained the network on the simulation data. Then, the real measured data was scanned, and the network was trained based on the measured data; therefore, we did not need to collect a large amount of real measured data, and we could learn the real measured data features and provide more accurate results. Second, a SPION concentration simulation was not incorporated into the study, and as such, sensitivity could not be quantified; real MPI data were scanned using a Perimag tracer at high concentrations. Third, to achieve high efficacy and reduce the complexity of the network, in this study, FDS-MPI was currently developed only for enhancing 2D images, which is a limitation for 3D images that are acquired using a tomographic technique, such as MPI. Nevertheless, we can alternatively process each 2D tomographic image using FDS-MPI to form a 3D MPI image. In future work, an advanced 3D deep learning network will be developed to address these limitations. In addition, the MPI-MRI hybrid system is promising (Vogel et al 2014, Franke et al 2016). In future work, we can extend the method to hybrid MRI-MPI methods. The advantages are that the SPIONs can be precisely localized, and a more accurate quantitative analysis can be achieved.

5. Conclusion

We developed a DL-based simple and quick method, namely FDS-MPI, to improve the spatial resolution in MPI. We used simple networks to solve complex problems. The proposed network combines two different subnetworks to improve image resolution; high-resolution images can then be reconstructed even in an MPI system with a low gradient field. The experimental results demonstrated that deep learning has great potential for image resolution improvement and structural preservation in the field of MPI. To evaluate the indicators of the proposed approach, we conducted experiments using simulated data and phantoms. The FDS-MPI model can perform at a high spatial resolution and improve the imaging quality, enabling MPI to be a more powerful tool in the field of molecular imaging. This study represents preliminary research of MPI in deep learning. This research can provide opportunities for the development of MPI to obtain higher spatial resolution images, rendering a broader prospect for MPI applications.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700401 the National Natural Science Foundation of China under Grant: 62027901, 81827808, 81527805, 81571836, 81671851, KKA309004533, 81227901; CAS Youth Innovation Promotion Association under Grant 2018167 and CAS Key Technology Talent Program; The Project of High-Level Talents Team Introduction in Zhuhai City (Zhuhai HLHPTP201703). The authors would like to acknowledge the instrumental and technical support of Multimodal Biomedical Imaging Experimental Platform, Institute of Automation, Chinese Academy of Sciences.

Ethical statement

All animal studies and procedures were performed according to a protocol approved by the Chinese PLA General Hospital Animal Care and Use Committee in accordance with the National Institutes of Health Guideline on the Care and Use of Laboratory Animals.

Deep learning for improving the spatial resolution of magnetic particle imaging

Article metrics

Author e-mails

Author affiliations

Author notes

ORCID iDs

Dates

Abstract

1. Introduction

2. Methodology

2.1. Problem setting

2.2. Architecture of the proposed network

2.2.1. Feature encoder network

2.2.2. Feature decoder network

2.2.3. Residual learning

2.3. Loss function

2.4. Evaluation metrics

3. Experiments and results

3.1. Dataset

3.1.1. Simulated data

3.1.2. Phantom data

3.2. Training of the network

3.3. Experimental results

3.3.1. Simulated experimental results

3.3.2. Phantom experimental results

4. Discussion

5. Conclusion

Acknowledgments

Ethical statement

Deep learning for improving the spatial resolution of magnetic particle imaging

Article metrics

Share this article

Author e-mails

Author affiliations

Author notes

ORCID iDs

Dates

Abstract

1. Introduction

2. Methodology

2.1. Problem setting

2.2. Architecture of the proposed network

2.2.1. Feature encoder network

2.2.2. Feature decoder network

2.2.3. Residual learning

2.3. Loss function

2.4. Evaluation metrics

3. Experiments and results

3.1. Dataset

3.1.1. Simulated data

3.1.2. Phantom data

3.2. Training of the network

3.3. Experimental results

3.3.1. Simulated experimental results

3.3.2. Phantom experimental results

4. Discussion

5. Conclusion

Acknowledgments

Ethical statement