Deep neural network-based approach to improving radiomics analysis reproducibility in liver cancer: effect on image resampling

The imaging dataset from the LiTS (Bilic et al 2019) was used for analysis in this study. The CT images of 201 patients were included in this dataset. This dataset was collected from seven clinical centers worldwide and contained CT images of liver tumors with diverse subtypes (primary, secondary, metastasis). The CT images were also acquired with varying scanning protocols (scanners, enhancement contrasts, x-ray energy, tube current, etc), which could represent the commonly used CT images of liver tumors. The pixel spacings of these images ranged from 0.56 mm × 0.56 mm to 1.00 mm × 1.00 mm, and the STes ranged from 0.45 to 6 mm. Only the CT images with a ST smaller than or equal to 1 mm were selected for further analysis. All the liver and tumor regions in the CT images had been segmented by trained radiologists at each clinical center and verified by three experienced radiologists.

A previous study revealed that both the liver tumor and the neighborhood liver region around the tumor (tumor ring) could provide prognostic value (Sun et al 2018). As such, we performed training on both the tumor region and surrounding liver regions. We cropped the selected CT images with the minimum box, which contained the whole liver region. All the CT images in the box regions were used to train the DNN. The workflow for dataset preparation was summarized in figure 1. Firstly, as we focus on the up-sampling in the vertical direction, the original CT images in the transverse plane were resliced to images in the coronal and sagittal plane. Then, all the resliced images were resampled into the pixel spacing of 1.00 mm × 1.00 mm, as the high-resolution ground truth. Secondly, as the ST of 3 and 5 mm were commonly used in liver CT scan (Cozzi et al 2017, Nie et al 2020), two datasets were generated by down-sampling high-resolution images to the pixel spacing of 1.00 mm × 3.00 mm and 1.00 mm × 5.00 mm. Thirdly, the two datasets were up-sampled into the pixel spacing of 1.00 mm ×1.00 mm using the 1-D bicubic interpolation approach slice by slice, resulting in the low-resolution samples (input). Then the residual images between the high- and low-resolution images were acquired as the labels (output). The training of DNN requires samples and labels of fixed size. Finally, all the samples and labels of varying sizes were randomly cropped into patches of 50 × 50 (Zhang et al 2017). Then the two datasets for different training tasks were acquired, which are Dataset 3 mm for conversion from 3 to 1 mm and Dataset 5 mm for the job of 5–1 mm. For each image at each training epoch, we empirically extracted 128 patches. Then all the training patches were randomly divided into the training, validation, and testing group by the original patient number in the dataset with the proportion of 60%, 10%, 30%, respectively.

Zoom In Zoom Out Reset image size

The residual learning strategy was applied in the training of DNN. The model was trained to learn the residual between the high- and low-resolution images from the low-resolution image. The architecture of DnCNN exhibited high effectiveness in denoising, single image super-resolution, and deblocking in natural images, showing the good extension ability (Zhang et al 2017). As such, we constructed a DNN using the modified DnCNN architecture for dealing with super-resolution of liver CT images. The network consisted of twenty 2D Conv with 64 filters of kernel size 3 × 3 × 64. The rectified linear units (ReLU) were connected after the Conv layers, except for the last layer. The batch normalization layers were added between the Conv and ReLU layers, except for the first and last Conv layers. The RMSE between the actual residual image and that generated from the DNN was applied as the training criteria. Two DNNs were trained for the up-sampling tasks of 3 mm–1 mm (DNN-3) and 5 mm–1 mm (DNN-5), respectively. The network was trained with the following parameters: epoch count: 40; mini-batch size: 128; initial learning rate: 0.01; learning rate decay schedule: 10. The training process was performed on a computer with one NVIDIA™ 1080Ti GPU and Intel^® Core^TM i7-6900K CPU @ 3.20 GHz and RAM of 128 GB using the Deep learning Toolbox in Matlab.

Two volumes of interest (VOIs) were used for radiomics feature extraction, i.e. the tumor region and tumor ring region. First, the tumor ring was acquired by performing dilation and erosion on the original tumor region defined by experts for 2 mm width respectively, resulting in a tumor ring with 4 mm thickness, as suggested by the study of (Sun et al 2018). Then, Radiomics features were extracted from the original tumor and corresponding ring region. A total of 540 commonly used radiomics features were extracted, including raw features and wavelet-based features. The raw features quantify the intensity and texture of VOIs from original CT images. Wavelet-based features are extracted from images decomposed and reconstructed by wavelet transform. The prefixes of 'HHH_,' 'HHL_,' 'HLH_,' 'LHH_,' 'LLH_,' 'LHL,' 'HLL_,' and 'LLL_' represents the eight subtypes of wavelet features, denoting the different combinations of signal components, namely high-frequency (H) and low-frequency (L) components, in X, Y, and Z directions, respectively. The description of these features can be found in our previous studies (Yang et al 2020). In addition, the shape features reproducibility to variation of up-sampling methods was assessed as described in supplementary I. (available online at stacks.iop.org/PMB/66/165009/mmedia) Only the CT images containing the tumor region were used for feature extraction and reproducibility analysis.

Two widely used image quality metrics which are the SSIM and PSNR, were used to assess the image quality of the liver region on CT images (Zhou et al 2004). The SSIM between given images $x$ and $y$ was calculated as following:

$$\begin{eqnarray*}&&SSIM=\displaystyle \frac{\left(2{\mu }_{x}{\mu }_{y}+{h}_{1}\right)\left(2{\delta }_{xy}+{h}_{2}\right)}{\left({{\mu }_{x}}^{2}+{{\mu }_{y}}^{2}+{h}_{1}\right)\left({{\delta }_{x}}^{2}+{{\delta }_{y}}^{2}+{h}_{2}\right)},\end{eqnarray*}$$

$$\begin{eqnarray*}&&{h}_{1}={\left(0.01L\right)}^{2},\,{h}_{2}={\left(0.03L\right)}^{2},\end{eqnarray*}$$

where ${\mu }_{x}$ and ${\mu }_{y}$ are the mean values of images x and y. ${\delta }_{x}$ and ${\delta }_{y}$ are the variances of x and y. ${\delta }_{xy}$ is the covariance of x and y. ${h}_{1}$ and ${h}_{2}$ are two constants. $L$ is the range of pixel values in x and y. The PSNR of a given image $x$ with the reference image $x$ is defined as:

$$\begin{eqnarray*}&&PSNR=10\,lo{g}_{10}\left(\displaystyle \frac{MA{{X}_{x}}^{2}}{MSE}\right),\end{eqnarray*}$$

where $MA{X}_{\unicode{x01E8B}}$ is the maximum pixel value in $x$ and MSE is the mean square error of $x.$ The CCC between the features extracted from high-resolution images and upsampled images were used to assess the reproducibility of each feature (Tsai 2017). The cutoff of 0.85 was set as the threshold of CCC for selecting reproducible features (Tanaka et al 2019). For comparison, the traditional interpolation-based up-sampling approaches, i.e. the nearest neighbor (Nearest), bilinear (Linear), and bicubic (Cubic) interpolation methods, were also evaluated for both the image quality and corresponding radiomics features reproducibility. Finally, the paired t-test and Mann–Whitney U test were used to compare image quality and feature reproducibility, where appropriate.

Among the 201 patients in LiTS dataset, 108 patients met the inclusion criteria that the ST of CT images should be equal to or smaller than 1 mm, were used in this study. In these patients, 72 patients were with both the tumor and liver region label on the CT images, and 36 patients had no tumor regions. The maximum number of tumors per patient was 75. The tumor size in this dataset ranged from 38 mm³ to 349 cm³. Finally, 63, 11, and 34 patients were included in the training, validation, and testing groups, respectively. The corresponding numbers of resliced images in the three groups were 28902 (61.91%), 4438 (9.51%), and 13344 (28.58%).

The ground truth and converted images using different methods were shown in figure 2. The images converted by DNN restored more details than those using the other three methods, especially in the box region marked by red dotted lines. The RMSE in the liver, tumor, and tumor ring regions using different methods in the test dataset were summarized in table 1. The mean ± standard deviations (mean ± std) of RMSE using DNN were 13.03 ± 3.90, 14.43 ± 3.94 and 14.64 ± 3 .91 in the liver, tumor, and tumor ring regions respectively for the task of 3 mm to 1 mm and 17.63 ± 4.02, 19.8 ± 4.81 and 20.58 ± 5.76 for the job of 5 mm to 1 mm. The RMSE value using DNN was smaller than that using the other three methods. The improvement of RMSE using DNN was also observed in different sub-groups, as shown in supplementary table S2 (3–1 mm) and S3 (5–1 mm).

Zoom In Zoom Out Reset image size

Both the DNN-3 and DNN-5 showed significant improvement in the image quality compared with traditional interpolation methods. The boxplot of SSIM and PSNR values for images up-sampled from 3 mm thickness and 5 mm thickness images in the test group were shown in figure 3. The mean and standard deviations of SSIM and PSNR of images converted using different methods in the test dataset were shown in table 2. The boxplots of SSIM and PSNR values using different methods in different sub-groups were shown in supplementary figure S1 (a) and S3 (a). The statistical comparison results between different methods was labeled above the boxplots. The DNN method showed higher SSIM and PSNR than all the interpolation-based methods with significance (p < 0.05). For DNN-3 in the test group, the mean ± std of SSIM using the Nearest, Linear, Cubic, and DNN scheme were 0.91 ± 0.08, 0.94 ± 0.06, 0.94 ± 0.05, and 0.97 ± 0.02, respectively. The corresponding PSNR of the four methods were 20.61 ± 8.30 dB, 26.62 ± 8.55 dB, 27.23 ± 8.56 dB, and 34.60 ± 7.14 dB. The DNN method also showed significantly higher PSNR and SSIM value (p < 0.05) in the training and validation groups than traditional methods, showing the excellent reproducibility of the DNN methods. The Cubic approach showed the best performance among the three interpolation-based methods, as shown in supplementary figure S1(b) and S3(b).

Zoom In Zoom Out Reset image size

The distribution of robust radiomics features from images converted using different methods was shown in table 3 (3–1 mm) and table 4 (5–1 mm). The features derived from the 3 mm ST images showed better reproducibility than those from 5 mm ST images. For tumor region features from 3 mm ST images using the Cubic interpolation method in the test group, 416 out of 540 (77%) features were found reproducible, while only 322 features (60%) from 5 mm ST images were stable. The comparison result of radiomics feature reproducibility between the tumor and tumor ring regions is summarized in table 5. The comparison was performed based on converted CT images in the test dataset. The tumor and tumor ring regions showed equal average CCC values in the task of 3–1 mm. In the job of 5–1 mm, the tumor ring region showed a higher mean CCC value. All the statistical analyses showed no significant difference between the tumor and tumor ring region (p > 0.05). The CCCs of the radiomics features showed a positive correlation with SSIM and PSNR of the corresponding volumes, as shown in tables 2 and 5.

The DNN method showed significantly higher CCC values than interpolation-based methods, demonstrating improved reproducibility of radiomics features. The boxplot of feature CCC values in the test datasets was shown in figure 4. In the tumor region, compared with the Cubic approach, the number of reproducible features increased 54 (10%) and 57 (11%) in the tasks of 3 mm–1 mm and 5 mm–1 mm, respectively. The improvements in the tumor ring region were 35 (6%) features in 3–1 mm conversion and 60 (11%) in the job of 5–1 mm. The boxplots of CCC values in different sub-groups were shown in supplementary figure S2 (3–1 mm) and S4 (5–1 mm).

Zoom In Zoom Out Reset image size

In the tumor region, the wavelet-based features showed higher robustness to the conversion of ST compared with raw features. For example, in the task of 5–1 mm, among the features extracted from images converted using DNN, 43% (26) raw features were found reproducible, while 74% (353) wavelet-based features were stable. After conversion by DNN, the GLCM and NGTDM based features showed the lowest reproducibility among the texture-based radiomics features in both tasks. Among the wavelet-based tumor region features, the 'HLH−' and 'HHL−' sub-groups showed the most inferior reproducibility in the missions of 5–1 mm and 3–1 mm, respectively. And the sub-groups with more high-pass filters ('HHH−,' 'HHL−,' 'HLH−,' 'LHH−') showed less reproducible features in both the two tasks. In the shape feature robustness analysis, the mask images converted by the Cubic interpolation method can be used for robust shape feature extraction in both the 3–1 and 5–1 mm tasks, as shown in supplementary I.