Semi-supervised segmentation of abdominal organs and liver tumor: uncertainty rectified curriculum labeling meets X-fuse

The MetaFormer architecture [24], known for its versatility and efficacy, has found application across a spectrum of current models [5, 59]. It encompasses two distinct residual sub-blocks. Specifically, the initial sub-block primarily incorporates a token mixer module, exemplified by self-attention [60], multi-layer perception (MLP) [61], and convolutional [62] layers and the second sub-block features a channel MLP with two linear layers, activated by GELU functions, together facilitating both inter and intra-token communication. The configuration of token mixers varies across phases. To obtain coarse body location in the initial phase, we opt for FasterNet employing partial convolution [58] which enhances computational efficiency by simply selectively applying filters to the first quarter of input channels, leaving the remaining ones unaltered. During Phases_2 and 3, the entirety of the X-Fuse structure is employed with two parallel branches concurrently engaging in feature extraction within distinct domains. The spatial encoder branch comprises scale-aware and self-attentive modulation for multiscale feature aggregation. This hybrid design is derived from our previous research, showcasing exceptional performance in the MICCAI FLARE23 challenge [41], while the spectral branch modulates the spectral feature in the frequency domain through the utilization of a learnable GF. Each token mixer is expounded upon in the following section illustrated with input $X\in \mathbb{R} ^{C^{^{\prime}}\times H^{^{\prime}}\times W^{^{\prime}}\times D^{^{\prime}}}$ where C' represent channel number, and $H^{^{\prime}}\times W^{^{\prime}}\times D^{^{\prime}}$ denote feature resolution.

The stem block, namely the patch embedding layer, partitions the input image into a series of discrete non-overlapping patches and projects linearly into high-dimensional token vectors by applying a strided convolution operation employing a kernel size of 3, stride of 2, and padding of 1. It extracts elementary visual features encapsulated within each patch region while simultaneously reducing the spatial dimensions to mitigate computational requirements when fed into MetaFormer-based models.

We leverage scale-aware [63] and self-attentive [38] modulation for spatial token mixing. Self-attentive modulation, commonly known as MSA, assumes a significant role in capturing contextual interdependencies among the discrete embedding tokens within a sequence (flattened input X') as expressed by equation (1). An MSA sublayer consists of n self-attention head modules in parallel. Each head independently entails mapping each patch token into query Q, key K, and value V vectors through linear projection W_QKV , and calculating attention scores $W_\mathrm{attn}$ through a scaled dot-product similarity between the queries and keys with scale factor D_h , signifying the importance of each patch token in relation to others. These attention weights are then applied to the corresponding V vectors, producing a weighted sum that effectively modulates the influence of each token. This dynamic modulation allows the model to adaptively prioritize relevant spatial information. Outputs from each head are combined through linear transformations $W_\mathrm{msa}$ providing a comprehensive representation that integrates various perspectives:

$$\begin{align} & X^{^{\prime}} = \mathrm{flatten}\left( X\xrightarrow{\mathrm{permute}}X^{\tau}\right),\qquad\qquad\qquad\qquad\qquad\ \ X^{\,{\tau}}\in \mathbb{R}^{H^{^{\prime}}\times W^{^{\prime}}\times D^{^{\prime}}\times C^{^{\prime}}}\nonumber\\ & \left[Q,K,V\right] = X^{^{\prime}}W_{QKV},\qquad\qquad\qquad\qquad\qquad\qquad\quad\ \ \ X^{^{\prime}}\in \mathbb{R} ^{N\times C^{^{\prime}}},\,\, N = H^{^{\prime}}\times W^{^{\prime}}\times D^{^{\prime}}\nonumber\\ & W_\mathrm{attn} = \mathrm{softmax}\left(QK^T/\sqrt{D_h}\right),\qquad\qquad\qquad\qquad\quad\,\,\, W_{QKV}\in \mathbb{R} ^{C^{^{\prime}}\times 3D_h}, \ D_h = D/n\nonumber\\ &\mathrm{SA}\left(X^{^{\prime}}\right) = W_\mathrm{attn}V, \qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad\!\! W_\mathrm{attn}\in \mathbb{R} ^{N\times N}\nonumber\\ & \mathrm{MSA}\left(X^{^{\prime}}\right) = \left[\mathrm{SA}_1\left( X^{^{\prime}}\right); \mathrm{SA}_2\left(X^{^{\prime}}\right);\ldots ;\mathrm{SA}_n\left( X^{^{\prime}}\right)\right] W_\mathrm{msa}. \;\quad\! W_\mathrm{msa}\in \mathbb{R}^{n\cdot D_h\times C^{^{\prime}}}\nonumber\end{align} \tag{ 1 }$$

The SAM instead modulates V with the succession of the multi-head mixed convolution (MHMC) and the scale-aware aggregation (SAA) module as defined in equation (2), designed to facilitate the incorporation of multi-scale contexts and adaptive modulation of tokens. The MHMC divides uniformly input channels into multiple heads $j\in \{1,2,\ldots ,n\}$ (in our implementation n = 3) with $i\in \{1,2,\ldots,C^{^{\prime}}/n\}$ channels in each head. For a single-channel feature maps $X_{j}^{i}$, it undergoes depth-wise separable convolutions $\mathrm{DWConv}_{k_j\times k_j\times k_j}$with distinct kernel sizes $k_j\in \{3,5,7\}$ enabling the discernment of a diverse spectrum of granularity features in an adaptive manner, which are aggregates in following SAA module. Specifically, the SAA forms $C^{^{\prime}}/n$ discrete shuffled groups, with each group selectively sampling a single channel from each of the partitioned heads produced by the MHMC. This shuffling integrates the features across granularities. Subsequently, the aggregated groups are processed by point-wise convolution $\mathrm{Conv}_{1\times 1\times 1}$ in inverted bottleneck form (channel expansion = 2) before (plus InstanceNorm (IN) and Relu activation) and after channel-wise concatenation, eventually serving as weight modulator of the value V by Hadamard product $\odot$ in contrast to matrix product in self-attention. In order to mitigate the computational load and model the progression from local to global dependency integration, we implement SAM modules in the initial two stages and MSA in the final two stages:

$$\begin{align} {\begin{array}{*{20}{l}} \kern5pt H_{j}^{i} = \mathrm{DWConv}_{k_j\times k_j\times k_j}\left(X_{j}^{i}\right),\qquad\qquad\qquad\qquad\,\,\,\,\,\,\,\,\,\,\,\, X_{j}^{i},H_{j}^{i}\in \mathbb{R}^{1\times H^{^{\prime}}\times W^{^{\prime}}\times D^{^{\prime}}} \rightarrow \mathrm{MHMC}\\ \begin{array}{l} G_i = \mathrm{Relu}\left(\mathrm{IN}\left( \mathrm{Conv}_{1\times 1\times 1}\left(\left[H_{1}^{i}, H_{2}^{i},\ldots, H_{n}^{i}\right]\right)\right)\right),\quad G_i\in \mathbb{R}^{2n\times H^{^{\prime}}\times W^{^{\prime}}\times D^{^{\prime}}}\\ W_\mathrm{sam} = \mathrm{Conv}_{1\times 1\times 1}\left(\left[G_1, G_2,\ldots ,G_{C^{^{\prime}}/n}\right]\right),\qquad\quad\,\,\,\,\,\,\, W_\mathrm{sam}\in \mathbb{R}^{C^{^{\prime}}\times H^{^{\prime}}\times W^{^{\prime}}\times D^{^{\prime}}} \end{array}\bigg\} \rightarrow \mathrm{SSA}\\ \kern5pt \mathrm{SAM}\left(X\right) = W_\mathrm{sam} \odot V. \qquad\qquad\qquad\qquad\quad\quad\quad\,\,\,\,\,\,\,\, V\in \mathbb{R}^{C^{^{\prime}}\times H^{^{\prime}}\times W^{^{\prime}}\times D^{^{\prime}}}\end{array}}.\end{align} \tag{ 2 }$$

We apply deep frequency filtering [8] to achieve explicit spectral feature modulation. Our approach is underpinned by the application of the discrete Fourier transform (DFT) and its inverse counterpart IDFT, serving as a conduit facilitating the transition between the spatial domain and the frequency domain representation of digital images. For 3D volume feature, its 3D-DFT $X_F\left( x,y,z \right)$ and 3D-IDFT $X\left( h,w,d \right)$ can be defined as:

$$\begin{equation} \begin{aligned} & X_F\left( x,y,z \right) = \sum_{h = 0}^{H-1}{\sum_{w = 0}^{W-1}{\sum_{d = 0}^{D-1}{X\left(h,w,d\right)e^{-j2\pi \left( x\dfrac{h}{H}+y\dfrac{w}{W}+z\dfrac{d}{D} \right)}}}},\\ & X\left( h,w,d \right) = \dfrac{1}{HWD}\sum_{h = 0}^{H-1}\sum_{w = 0}^{W-1}\sum_{d = 0}^{D-1}X_F\left(x,y,z\right)e^{j2\pi \left(x\dfrac{h}{H}+y\dfrac{w}{W}+z\dfrac{d}{D}\right)}. \end{aligned}\end{equation} \tag{ 3 }$$

By virtue of the conjugate symmetric property inherent in FFT, $X_F\in \mathbb{C} ^{C^{^{\prime}}\times H^{^{\prime}}\times W^{^{\prime}}\times ( \dfrac{D^{^{\prime}}}{2}+1 )}$ only needs retain the half of spatial dimensions while preserve the entirety of information. In practice, we apply fast Fourier transform (FFT) [64] algorithm for efficient DFT computation. Specifically, the input features X are first transformed via the FFT from the spatial domain into the frequency domain in which each component in the resulting Fourier spectrum has the intrinsic global vision, thus providing inherent advantages for modeling global context and long-range interactions. The frequency representation allows direct manipulation and filtering of the spectral characteristics of the features. Using a learnable gating mechanism, the GFs W_gf modulate the frequencies by element-wise multiplication $\odot$ to adaptively adjust the feature responses across both local and global scales. Low frequency emphasis enhances large-scale patterns while attenuating high frequencies sharpen local details. The resultant filtered frequency representations are subsequently transmuted back into the spatial domain through the process of inverse fast Fourier transform (IFFT), which de-mixes the updated global frequency representations to recover the local token features. The GF layer can be formulated as Equalton 4. According to convolution theorem [64], GF is equivalent to a depthwise global circular convolution while GF enables explicit tuning of spectral properties within the feature map, concurrently circumventing the computational inefficiencies of conventionally large convolutional kernels.

$$\begin{align} \mathrm{GF}\left(X\right) = \mathrm{IFFT}\left(W_{gf}\odot \mathrm{FFT}\left( X \right)\right). \qquad\quad\ W_{gf}\in\mathbb{C}^{C^{^{\prime}}\times H^{^{\prime}}\times W^{^{\prime}}\times\left(\dfrac{D^{^{\prime}}}{2}+1\right)}.\end{align} \tag{ 4 }$$

The encoded feature representations extracted from the two-branch encoders, fused at the bottleneck by simple channel-wise concatenation, are input into a residual block on skip connections comprising two sequential 3×3×3 convolutional layers with instance normalization and ReLU activation before concatenation with the progressively upsampled decoder features. Such integrated representations are then passed through another residual combo block of the same structure. For explicit architectural perturbation, both trilinear upsampling and transpose convolutions are employed for decoding in two branches which yield final ensembled inference output, and also ensure consistency regularization during SSL.

We employ an adaptable threshold on the confidence map $f_j(x_i)$ for reliable pseudo-label selection such that only predictions surpassing the threshold contribute to the pseudo-supervision loss:

$$\begin{align} L_\mathrm{cpl}^{j} = \unicode{x1D7D9} \left(\mathrm{max}\left(f\left(x_i\right)\right) > \gamma\right) L_\mathrm{cps}^{j},\end{align} \tag{ 8 }$$

where $\gamma \in (0, 1)$, $\unicode{x1D7D9}( \cdot \gt \gamma)$ is the indicator function for confidence-based thresholding, where γ represents the threshold.

Following the principle [25] that lower initial thresholds gather more pseudo labels to accelerate early learning. As model uncertainty on unlabeled data decreases during training, thresholds become more stringent to filter out noisy assignments:

$$\begin{equation} \gamma_t = \begin{cases} \dfrac{1}{C}, &t = 0, \\ \left(1-\lambda\right)\gamma_{t-1} + \lambda \dfrac{1}{B}\sum_{b = 1}^{B}\max\left(f\left(x_i\right)\right), &\mathrm{else}.\\ \end{cases}\end{equation} \tag{ 9 }$$

We set the initial value of γ as $\frac{1}{C}$ where C denotes the number of classes, B is the batch size. Rather than re-evaluating all unlabeled data per iteration to determine precise confidence. The exponential moving average enables efficient aggregation of confidence scores across iterations to estimate global trends by assigning exponentially decaying weights $1-\lambda$ to prior observations with $\lambda = \dfrac{t}{t_\mathrm{max}}$, $t_\mathrm{max}$ is the total iterations.

Motivated by prior works [26], utilizing Kullback–Leibler variance as a measure of prediction uncertainty between teacher-student models, this work incorporates JS divergences into the CPS architecture to quantify and penalize inconsistent predictions from the X-Fuse perturbed decoders. The JS divergence provides a mathematically principled information-theoretic approach to quantifying the dissimilarity between two probability distributions.

$$\begin{align} D_{JS}\left(f_A\left(x_i\right);f_B\left(x_i\right)\right) = \sum_{\left(x_i \in D\right)}f_A\left(x_i\right)\log\dfrac{f_A\left(x_i\right)}{M\left(x_i\right)} + f_B\left(x_i\right)\log\dfrac{f_B\left(x_i\right)}{M\left(x_i\right)},\end{align} \tag{ 10 }$$

where $M(x_i) = \dfrac{1}{2}(f_A(x_i) + f_B(x_i))$, denotes the average distribution of both two predictions.

Specifically, the consistency loss is activated to promote prediction agreement in cases where the JS variance is low, indicating a high degree of confidence in the consistency between the perturbed decoders. Conversely, when the JS variance is high, reflecting significant divergence between the decoder predictions, the consistency loss is largely omitted. Henceforth, we employ the JS divergence as a guiding metric for pseudo supervision, further facilitating the rectification of the consistency loss, which is expressed as follows:

$$\begin{align} L_\mathrm{Rcps}^j\left(f_A\left(x_i\right); f_B\left(x_i\right)\right) = \sum_{\left(x_i\right) \in D}\left[\left(1-D_{JS}\left(f_A\left(x_i\right); f_B\left(x_i\right)\right)\right) \times L_\mathrm{cpl}^j + \frac{1}{-\log \left( D_{JS}\left( f_A\left( x_i \right) ;f_B\left( x_i \right) \right) \right) +\varepsilon}\right].\end{align} \tag{ 11 }$$

Additionally, that last item in equation (11) is incorporated into the loss to prevent scenarios where the variance remains persistently high where ε is a small value to prevent the denominator from being zero.

Overall, as depicted in figure 3, our framework encompasses the integration of a dynamic confidence threshold for pseudo-label selection, subsequently facilitating the generation of model outputs characterized by heightened consistency through additional uncertainty rectification. Both elements collectively contribute to the attenuation on the impact of noise labels, thereby enhancing the learning proficiency of our SSL method. The comprehensive details of the training procedure are expounded upon in algorithm 1.

Zoom In Zoom Out Reset image size

Algorithm 1. Training procedure of proposed method, e.g. liver tumor segmentation.
Input   : $\{(x_i^{l}, y_i^{l})\}_{i = 1}^N \in \mathcal{D}_l$, $\{(x_i^{u})\}_{i = 1}^M \in \mathcal{D}_u$
Output: model's parameters θ
1 for $\boldsymbol{iter\, in [1,\, total\_iters]}$ do
2 Sampling batch Bl and Bu , where $B_l \in D^l$, $B_u \in D^u$, $B = \,\,B^l\cup B^u$  ;
3 Computing prediction outputs of duel decoders:
4 ${p}_A = f_A (x_i; \theta)$ and ${p}_B = f_B (x_i, \theta)$  ;
5 Computing supervised loss according to equation (6):
6 $L_\mathrm{seg} = \dfrac{1}{4|B_l|} \sum\limits_{(x_i^{l}, y_i^{l}) \in B_l}[L_\mathrm{seg}^A(f_A(x_i^{l}; \theta), y_i^{l}) + L_\mathrm{seg}^B(f_B(x_{i}^l; \theta), y_{i}^l)]$  ;
7 Updating γt in B according to equation (9):
8 $\gamma_t = \min((1-\lambda)\gamma_{t-1}+\lambda\dfrac{1}{|B|}\sum\limits_{b = 1}^{B}\max({p}_A),\ (1-\lambda)\gamma_{t-1}+\lambda\dfrac{1}{|B|}\sum\limits_{b = 1}^{B}\max({p}_B))$  ;
9 Generating PLA and PLB according to dynamic threshold γt :
10 $PL_A = \unicode{x1D7D9}({p}_A \gt \gamma_t)$, $PL_B = \unicode{x1D7D9}({p}_B \gt \gamma_t)$  ;
11 Computing JS-divergence $D_{JS}({p}_A\ ||\ {p}_B)$ according to equation (10)   ;
12 Using JS-divergence to rectify the consistency loss according to equation (11):
13 $L_\mathrm{Rcps}^A = \dfrac{1}{2|B|} \sum\limits_{x_i \in B}[(1-D_{JS}) \times L_\mathrm{cps}^{A}(PL_A, {p}_B) + \frac{1}{-\log \left( D_{JS} \right) +\varepsilon}]$
14 $L_\mathrm{Rcps}^B = \dfrac{1}{2|B|} \sum\limits_{x_i \in B}[(1-D_{JS}) \times L_\mathrm{cps}^{B}(PL_B, {p}_A) + \frac{1}{-\log \left( D_{JS} \right) +\varepsilon}]$  ;
15 Computing total loss: $L_\mathrm{total} = L_\mathrm{seg} +\alpha(L_\mathrm{Rcps}^A + L_\mathrm{Rcps}^B)$, $\alpha = 1-e^{-iter}$  ;
16 Updating model's parameters θ   ;
17 End,return θ

To validate the efficacy of our proposed X-Fuse model for the segmentation of anatomically pertinent organs (i.e. bones, liver) as well as pathological regions (i.e. liver tumors), we evaluate our approach under SL conditions with 100% labeled data against prevailing CNN based models such as V-Net [34] and H-DenseUNet$^{\diamond}$ [4] besides the highly configurable and modular architecture nnUNet$^{\diamond}$ [69]; ViT based models such as UNetFormer$^{\diamond}$ [59] and SwinUNETR [5]; and contemporary hybrid models like CoTr [70] and APAUNet$^{\diamond}$ [71] that synergistically couple the advantages of both paradigms, as summarized in table 1. $^{\diamond}$ denotes corresponding methods have been previously applied for liver tumor delineation in their original implementation. Our proposed X-Fuse framework markedly advances previous leading models by substantial margins across all key quantitative segmentation metrics. Specifically, for liver segmentation, X-Fuse exhibits substantial improvements, manifesting a 2.41% enhancement in DSC, a notable 2.04 voxel reduction in HD95, and 0.98 voxel decrement in ASD when compared to the H-DenseUNet. Moreover, X-Fuse demonstrates favorable outcomes in bone segmentation, showcasing a marginal advantage over alternative methodologies where CNNs majorly outperform their transformer counterparts. X-Fuse displays superior performance in the domain of tumor segmentation, registering an increase of 4.17% and 2.91% on DSC over nnUNet and CoTr, respectively. These results underscore the efficacy and competitive edge of X-Fuse in various medical segmentation tasks.

Table 1. Evaluation of X-Fuse against state-of-the-art fully-supervised models in terms of DSC(%)$\uparrow$, HD95(voxel)$\downarrow$ and ASD(voxel)$\downarrow$. Best results are marked in bold.

Method	Bones			Liver			Liver tumors
	DSC	HD95	ASD	DSC	HD95	ASD	DSC	HD95	ASD
V-Net [34]	90.52$^{\ast}$	20.80$^{\dagger}$	7.44$^{\dagger}$	90.13$^{\dagger}$	5.51$^{\ast}$	2.41$^{\ast}$	73.12$^{\dagger}$	24.78$^{\dagger}$	10.04$^{\dagger}$
H-DenseUNet [4]	91.50$^{\ast}$	18.78$^{\dagger}$	5.39$^{\ast}$	90.34$^{\dagger}$	5.34$^{\dagger}$	2.11$^{\ast}$	75.46$^{\dagger}$	21.95$^{\dagger}$	23.89$^{\dagger}$
nnUNet [72]	92.27	12.56	4.45	90.01$^{\dagger}$	7.87$^{\dagger}$	1.78	74.59$^{\dagger}$	22.85$^{\dagger}$	15.38$^{\dagger}$
UNetFormer [59]	90.44$^{\dagger}$	22.32$^{\dagger}$	7.18$^{\dagger}$	91.17$^{\ast}$	10.65$^{\dagger}$	2.68$^{\ast}$	71.46$^{\dagger}$	30.17$^{\dagger}$	12.09$^{\dagger}$
SwinUNETR [5]	90.56$^{\dagger}$	18.82$^{\dagger}$	6.95$^{\dagger}$	91.14$^{\ast}$	9.45$^{\dagger}$	2.96$^{\ast}$	70.93$^{\dagger}$	23.56$^{\dagger}$	10.27$^{\dagger}$
CoTr [70]	91.36$^{\ast}$	25.95$^{\dagger}$	7.89$^{\dagger}$	91.13$^{\ast}$	5.48$^{\dagger}$	2.89	75.85$^{\dagger}$	15.45$^{\dagger}$	6.76$^{\dagger}$
APAUNet [71]	90.53$^{\dagger}$	24.53$^{\dagger}$	8.76$^{\dagger}$	90.46$^{\dagger}$	9.76$^{\dagger}$	3.45$^{\dagger}$	72.45$^{\dagger}$	35.67$^{\dagger}$	18.34$^{\dagger}$
X-Fuse	93.17	12.38	4.01	92.75	3.30	1.13	78.76	10.23	3.23

$\ast$ and $\dagger$ denote statistical significance at $p \unicode{x2A7D} 0.05$ and $p \unicode{x2A7D} 0.01$, respectively, based on a pairwise t-test compared with our method.

To assess the contribution of individual components of X-Fuse, comprehensive ablation studies are conducted in table 2, focusing on the significance of encoder branches and bottleneck fusion. The findings reveal that utilizing spatial encoder alone surpasses their CNN-transformer hybrid counterparts such as APAUNet by 1.14% DSC and 3.9% for bones and tumors respectively, substantiating efficacious encoding of multi-scale context aggregation. Meanwhile, the spectral encoder itself rivals, for all metrics on bone and liver, the reputed global models like SwinUNETR in aptly capturing long-range inter-dependencies while seemingly falls behind its spatial counterpart. Additionally, the bottleneck integration layer fusing the latent embeddings from both pathways enables X-Fuse to confer considerable improvements, e.g. 2.12% on tumor over individual modules as well as outperforming the ensemble of two separate networks containing either type of encoders. This underscores the merits of a harmonious dual-encoder conceptualization, harnessing both spatial and spectral cues in a unified architecture.

Table 2. Ablation studies quantifying the contribution of individual components including spatial encoder, spectral encoder, and bottleneck fusion on the test set.

Setting	Spatial encoder	Spectral encoder	Bottleneck fusion	DSC(%)$\uparrow$
				Bones	Liver	Tumors
SL with 100% labeled images	✓			91.67$^{\ast}$	91.38$^{\ast}$	76.35$^{\dagger}$
		✓		90.67$^{\dagger}$	91.18$^{\ast}$	73.29$^{\dagger}$
	✓	✓		91.79$^{\ast}$	91.64$^{\ast}$	76.64$^{\dagger}$
	✓	✓	✓	93.17	92.75	78.76

$\ast$ and $\dagger$ denote statistical significance at $p \unicode{x2A7D} 0.05$ and $p \unicode{x2A7D} 0.01$, respectively, based on a pairwise t-test compared with our method.

The contrastive semi-supervised approaches (all models are built upon V-Net for a fair comparison) in table 3 encompass the seminal mean teacher self-ensembling paradigm (MT) [13] alongside its uncertainty-guided derivative UA-MT [73], uncertainty-attenuated pyramid consistency regularization via URPC [74], shape-aware adversarial learning through SASSNet [75], multi-task consistency enforcement by DTC [56], and cross-pseudo supervision based mutual consistency learning using a dual-decoder MC-Net [54]. In comparison to the supervised V-Net exclusively conditioned solely upon annotated samples in table 1, all semi-supervised methodologies harnessing on unlabeled exemplars confer notable performance enhancements, thereby substantiating benefit in the exploitation of unlabeled data. The UA-MT method eclipses the vanilla MT approach by orchestrating the guidance of student model optimization, underscoring the instrumental role of uncertainty regularization. Conversely, the DTC model fails to exhibit advantages over the MT model, thereby unveiling the limited positive impact of transformation consistency on our stipulated task. MC-Net, distinguished by its superiority over URPC and UAMT across metrics, decisively validates the robustness of mutual consistency. By assimilating implicit geometric clues, the proposed SASSNet attains top performance on the HD95 and NSD metrics on bones and liver. Our framework transcends prevailing semi-supervised techniques in terms of DSC, demonstrating a 1.27% improvement margin on tumor over the next top-performing method SASSNet. By virtue of uncertainty modulation, our approach further refines the rudimentary CPS in MC-Net to 94.35%, 94.94% and 84.89% for bones, liver and liver tumor on DSC, synergistically complemented by a dynamic thresholding strategy.

Table 3. Quantitative comparisons of semi-supervised segmentation models on the test set in terms of DSC(%)$\uparrow$, HD95(voxel)$\downarrow$ and ASD(voxel)$\downarrow$. Best results are marked in bold.

Method	Bones			Liver			Liver tumors
	DSC	HD95	ASD	DSC	HD95	ASD	DSC	HD95	ASD
MT [13]	92.58$^{\dagger}$	13.64$^{\dagger}$	4.23	92.76$^{\dagger}$	4.67	2.29	80.76$^{\dagger}$	13.73$^{\dagger}$	7.35$^{\dagger}$
UA-MT [73]	93.02$^{\ast}$	17.94$^{\dagger}$	4.56	93.35$^{\ast}$	8.04$^{\dagger}$	4.35$^{\dagger}$	81.15$^{\dagger}$	9.58$^{\dagger}$	6.94$^{\dagger}$
DTC [56]	92.23$^{\dagger}$	15.66$^{\dagger}$	5.82$^{\ast}$	92.67$^{\dagger}$	4.90$^{\ast}$	3.76$^{\ast}$	80.73$^{\dagger}$	15.73$^{\dagger}$	6.45$^{\dagger}$
URPC [76]	92.45$^{\dagger}$	13.96$^{\dagger}$	4.29	93.43$^{\ast}$	7.19$^{\dagger}$	3.98$^{\ast}$	81.12$^{\dagger}$	13.32$^{\dagger}$	5.87$^{\dagger}$
MC-Net [15]	93.24$^{\ast}$	12.18$^{\ast}$	4.81	94.13	6.73$^{\dagger}$	3.68$^{\ast}$	81.82$^{\dagger}$	9.28$^{\dagger}$	3.34
SASSNet [75]	93.32$^{\ast}$	8.45 $^{\dagger}$	3.15 $^{\ast}$	94.10	3.15	2.28	82.24$^{\ast}$	9.24$^{\dagger}$	3.34
Ours	94.35	11.33	4.32	94.94	3.82	2.36	83.89	5.65	3.25

$\ast$ and $\dagger$ denote statistical significance at $p \unicode{x2A7D} 0.05$ and $p \unicode{x2A7D} 0.01$ respectively, based on a pairwise t-test compared with our method.

Figure 5 investigates the impact of our proposed semi-supervised approach under varying labeling budgets, ranging from a modest 10% to a more extensive 80% of the labeled data. Prominently evident is the consistent superior performance of our proposed methodology across varying proportions of labeled data, thereby attesting to its inherent superiority over prior arts. Concerning DSC on tumor, our methodology achieves comparable performance with a minimal 10% allocation of labeled data, demonstrating a marginal deviation of a mere 0.29% compared to 60% usage in the SL baseline (V-Net). This outcome underscores the notable capacity of our approach to substantially mitigate the exigencies associated with labeled data requirements. Additionally, our method, endowed with a sparse 20% allocation of labeled CT scans, approaches performance equipoise over three segmentation objectives, diverging solely around 0.8%, relative to the plain MT architecture with tripled labeled data, demonstrating its significant advantages under limited labeled data regimes.

Zoom In Zoom Out Reset image size

Our SSL framework incorporates three integral constituents, namely the foundational co-training mechanism (baseline) grounded in CPS, coupled with the concomitant incorporation of uncertainty rectification through JS variance and the implementation of dynamic thresholding for pseudo labeling. To discern the efficacy of each individual module on X-Fuse, we conducted a systematic ablation study, incrementally introducing the aforesaid components onto the baseline. The experimental milieu was established with an allotment of 50% and 100% of labeled images, and the resultant performance metrics across various configurations are cataloged in table 4. Evidently, Modules act in clear synergy, unlocking progressive gains to deliver state-of-the-art semi-supervised segmentation. Concretely, While with half labeled data, it reveals that the dynamic thresholding imparted by the CPL strategy serves to amplify the baseline performance by an additional 2.03% and 1.89% on tumors and bones. Uncertainty Rectification bestows a further increment of 0.67% upon the baseline performance of liver delineation. The pinnacle of advancement is attained with the amalgamation of all with more than 1% increments over each individual, propelling our framework to an elevated echelon with 93.12% on Bone, 92.56% on liver Matching the performance of baseline reliant solely on labeled samples in table 1. leveraging all labeled data, in the absence of any regulatory mechanism for pseudo-labeling, the unadorned mutual supervision procedure attains a commendable 83.57% on the DSC of tumor segmentation with a 4.71% increase relative to the supervised-only baseline, demonstrating substantive efficacy of CPS. The implementation of dual regularization techniques engenders the finest results on bones (95.42%), liver (96.26%), and tumors (85.20%).

Table 4. Ablation studies quantifying the individual impact of key loss regulatory components, including CPS, uncertainty rectification, and dynamic thresholding.

Setting	CPS	Uncertainty rectification	Dynamic threshold	DSC(%) $\uparrow$
				Bones	Liver	Tumors
SSL with 50% labeled images	✓			89.31$^{\dagger}$	90.76$^{\dagger}$	76.56$^{\dagger}$
	✓	✓		91.29$^{\ast}$	91.45$^{\ast}$	77.79$^{\dagger}$
	✓		✓	91.34$^{\ast}$	90.83$^{\ast}$	78.45$^{\ast}$
	✓	✓	✓	93.12	92.56	80.19
SSL with 100% labeled images	✓			93.82$^{\ast}$	94.76$^{\ast}$	83.57$^{\ast}$
	✓	✓		94.79	95.45$^{\ast}$	84.59
	✓		✓	94.34	95.83	84.35
	✓	✓	✓	95.42	96.26	85.20

$\ast$ and $\dagger$ denote statistical significance at $p \unicode{x2A7D} 0.05$ and $p \unicode{x2A7D} 0.01$, respectively, based on a pairwise t-test compared with our method.

We examine the components of our TumorAug technique, i.e. TumorSyn and TumorCP, within both scenarios of SL and SSL frameworks in figure 6. In the SL case, TumorSyn imparts a perceptible augmentation of 3.56% to DSC while the TumorCP precipitates an impressive 6.13% surge, indicating the potential of our tumor augmentation approach to further refine outcomes through the nuanced capture of diverse tumor features. While model capacity appears to approach saturation as observed in table 4, our SSL method exhibits a heightened capacity approach by better exploiting TumorSyn and TumorCP for segmentation improvements via providing more diverse, and confident pseudo labeling, enabling TumorSyn and TumorCP to accrue additional gains atop the augmentation-free baseline. Specifically, TumorSyn and TumorCP exhibit a notable escalation of 1.5% and 3.1% in DSC respectively. Notably, the synergistic utilization of TumorSyn and TumorCP under both scenarios catapults raw purely supervised variant by an appreciable 8.35% and 10.49%. Overall, TumorAug potentiates multifarious representation and enhances the generalization capacity of the model across lesional heterogeneous manifestations.

Zoom In Zoom Out Reset image size

Figure 7 shows some qualitative results of different models under SL and SSL segmentation contexts. Particularly, with respect to the segmentation of bone and liver, it is noteworthy to observe that all models under consideration demonstrate a comparable level of performance except that SwinUNETR and V-Net demonstrate over-segmentation and under-segmentation respectively in liver segmentation tasks while X-Fuse distinguishes itself by achieving heightened sensitivity in both learning contexts. Regarding liver tumor segmentation, our model exhibits a commendable capability for delineating fine-grained morphological details and otherwise minute pathological structures, which can be attributed to the innate multi-scale representational ability. Furthermore, the integration of tumor augmentation techniques serves to amplify model competencies for fine-grained characterization of manifold morphological variabilities of liver tumors.

Zoom In Zoom Out Reset image size