Abstract
Solar flares are intense eruptive events caused by the rapid release of magnetic energy, often impacting Earth’s space environment through electromagnetic radiation and high-energy particles. Accurate flare prediction is critical for space weather forecasting. However, many existing deep learning approaches often rely on single-modal inputs or shallow feature fusion, limiting their ability to capture complementary information. In this study, we propose a dual-branch multimodal fusion deep learning model for predicting 24 hr solar flares. The model integrates magnetograms and magnetic parameters through cross-attention mechanisms, followed by cross-scale interactions at the feature level to enhance multiscale representation. It is designed to perform both binary prediction of ≥C-class flares and multiclass classification of C-, M-, and X-class flares. To ensure rigorous evaluation, we employ a stratified group five-fold cross-validation scheme to preserve class representativeness and adopt a splitting-before-sampling strategy based on NOAA active region numbers to prevent data leakage. Experimental results show that the model achieves a true skill statistic (TSS) of 0.661 and a Heidke skill score (HSS) of 0.658 for binary ≥C-class prediction, while notably attaining a TSS of 0.780 and an HSS of 0.775 for X-class flares in the multiclass task. Compared with existing approaches, the model demonstrates superior performance in predicting intense X-class flares, effectively suppresses the false-alarm rate, and exhibits strong generalization capability.

Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
1. Introduction
Solar flares are among the most energetic eruptive phenomena in solar activity and represent a rapid and explosive release of magnetic energy stored in the solar corona. This released magnetic energy is converted into electromagnetic radiation across a broad wavelength range and into the acceleration of high-energy particles. Intense solar flares are frequently accompanied by the production of solar energetic particles that can propagate through interplanetary space and impact the near-Earth environment (M. Messerotti et al. 2009), as well as by coronal mass ejections (CMEs) that expel large magnetized plasma into the heliosphere (e.g., X. Huang et al. 2018; Y. Zheng et al. 2023b). These eruptive processes can drive rapid and substantial disturbances in the near-Earth space environment, potentially leading to extreme space-weather events (Y. D. Liu et al. 2014) that disrupt satellite operations, communication, and power grids.
Consequently, the development of reliable solar flare forecasting capabilities is essential for mitigating the widespread and potentially severe impacts of solar eruptive events.
The prediction of solar flares is a widely studied topic, with researchers analyzing various prediction methods. R. G. Giovanelli (1939) once proposed a solar flare prediction model based on statistical relationships between solar flares and sunspots. Subsequently, an increasing number of statistical methods are applied to solar flare prediction models (e.g., K. Leka & G. Barnes 2003; G. Barnes et al. 2007; H. Song et al. 2009; D. S. Bloomfield et al. 2012; K. Lee et al. 2012; D. Falconer et al. 2011; G. Barnes et al. 2016; J. Park et al. 2017). In addition, there are also many researchers who predict the occurrence of solar flares by studying magnetic field parameters (e.g., K. Leka & G. Barnes 2003; G. Barnes et al. 2007; D. Yu et al. 2009; X. Huang et al. 2010; X. Yang et al. 2013; M. G. Bobra & S. Couvidat 2015; M. Korsós et al. 2015; J. Wang et al. 2020; D. Cicogna et al. 2021; F. Ribeiro & A. L. S. Gradvohl 2021; K. Yi et al. 2021; H. Ran et al. 2022). For instance, J. Wang et al. (2019) extracted SHARP parameters from the polarity inversion line (PIL) region of active regions (ARs) and employ a support vector machine (SVM) to predict solar flares based on the modified parameters.
Many researchers apply machine learning techniques to solar flare prediction, including artificial neural networks (ANNs) (e.g., O. W. Ahmed et al. 2013; R. Li & J. Zhu 2013; N. Nishizuka et al. 2018), SVMs (e.g., X. Yang et al. 2013; M. G. Bobra & S. Couvidat 2015; L. E. Boucheron et al. 2015; A. Raboonik et al. 2016; N. Nishizuka et al. 2017; V. M. Sadykov & A. G. Kosovichev 2017), random forests (RFs) (e.g., C. Liu et al. 2017; K. Florios et al. 2018), and multilayer perceptrons (MLPs) (e.g., R. Li & J. Zhu 2013; K. Florios et al. 2018).
Traditional machine learning models require manually designed feature extractors to select and extract relevant features. In contrast, convolutional neural networks (CNNs) are capable of automatically learning and extracting features by training convolutional kernels without human intervention. Consequently, deep learning methods have been increasingly applied to solar flare prediction (e.g., X. Huang et al. 2018; E. Park et al. 2018; Y. Zheng et al. 2019; X. Li et al. 2020; C. Pandey et al. 2024) and long short-term memory (LSTM) networks (e.g., A. E. McCloskey et al. 2018; Y. Chen et al. 2019; H. Liu et al. 2019; Z. Jiao et al. 2020; X. Wang et al. 2020; Z. Sun et al. 2022).
X. Huang et al. (2018) proposed a CNN model for binary-class solar flare prediction based on a classical architecture comprising two convolutional layers, each with 64 kernels of size 11 × 11. Z. Sun et al. (2022) trained and evaluated two heterogeneous deep learning networks—CNN and LSTM—as well as their stacked ensemble. V. Deshmukh et al. (2022) introduced a hybrid two-stage approach. In the first stage, a CNN model based on the VGG-16 architecture extracts features from a temporal stack of continuous magnetogram samples to generate flare probabilities. These probabilities are then combined with magnetogram-derived feature vectors to train an extremely randomized trees (ERT) model in the second stage, which produces a binary deterministic prediction (flare/no-flare) within the forecast window.
R. Tang et al. (2021) proposed a solar flare prediction method that fuses three independent deep learning models: deep neural networks (DNNs), bidirectional LSTM networks (Bi-LSTMs), and CNNs. The outputs of these models are integrated using a fully connected neural network to generate the final prediction results. K. Kaneda et al. (2022) proposed the Flare Transformer model, which utilizes a magnetogram module and a sunspot feature module to process image and physical features. A transformer attention mechanism is introduced to model temporal relationships between the input features.
In summary, deep learning has made significant breakthroughs and progress in solar flare prediction. Although many CNN-based methods achieve promising results, they still have certain limitations. Some approaches rely solely on magnetograms; others only use physical magnetic parameter features; and some are based on statistical modeling. Although R. Tang et al. (2021) attempted to combine the outputs of multiple deep learning models, the overall architecture still adopts a single-branch framework derived from individual networks. In contrast, the Flare Transformer model, proposed by K. Kaneda et al. (2022), employs a sparse attention mechanism that enables only shallow cross-modal interactions, potentially constraining the ability to capture higher-order semantic correlations and limiting effective utilization of magnetic parameters.
Although various deep learning approaches are adopted for solar flare prediction, most existing studies rely on single-branch architectures and shallow fusion strategies, which fail to fully exploit the complementary information between magnetograms and magnetic parameters. Currently, dual-branch multimodal fusion methods based on deep learning remain relatively underexplored for solar flare prediction. In response to these limitations, we propose a dual-branch architecture that encodes magnetograms and magnetic parameters in parallel and employs a cross-modal fusion module at the intermediate stage for deep joint modeling. Unlike traditional methods that merely concatenate features at higher layers or depend on sparse attention for shallow interactions, our approach enables deep fusion during representation learning, effectively capturing complementary features from both modalities. In addition, the model incorporates a multiscale processing mechanism to extract key features across different scales, enhancing its ability to represent complex ARs. By jointly optimizing modality fusion and multiscale feature representation, our design significantly improves the accuracy and robustness of solar flare prediction.
The structure of the remaining sections of the paper is as follows. Section 2 provides a detailed description of the dataset. Section 3 outlines our proposed methodology and the corresponding models. In Section 4, we evaluate the performance of the proposed solar flare prediction model. Finally, Section 5 concludes the paper with a discussion and summary.
2. Data Sources
The SHARP database is derived from full-disk observations by the Helioseismic and Magnetic Imager (HMI; J. Schou et al. 2012) on the Solar Dynamics Observatory (SDO). It contains the necessary information to automatically identify and track ARs in solar magnetograms. Each identified AR is assigned a unique number, and numerous physical parameters relevant to solar flare prediction are provided. Since 2010, these data cutouts have been referred to as Space-weather HMI Active Region Patches (SHARP), which continuously monitor ARs on the solar surface visible to SDO (M. G. Bobra et al. 2014).
In this study, we conduct an experimental analysis of solar flare prediction using SHARP data provided by the SDO/HMI instrument team. All magnetograms of recorded ARs from 2010 May to 2023 May, with a cadence of 96 minutes, and in the cylindrical equal-area (CEA) projection are selected.
In addition, based on the X-ray irradiance data recorded by the Geostationary Operational Environmental Satellites (GOES), operated by the National Oceanic and Atmospheric Administration (NOAA), information on the location, intensity, and timing (start, peak, and end) of solar flares is obtained. Solar flares are classified into five categories—A, B, C, M, and X—according to their peak soft X-ray flux. The A and B classes are generally considered minor solar flares, whereas C-, M-, and X-class flares are classified as major solar flare events, with M- and X-class flares being the most intense and thus of particular importance.
2.1. Feature Set of Physical Parameters
Each record in the SHARP dataset includes a set of metadata containing values of various physical magnetic parameters for the corresponding AR. These parameters are primarily derived from raw magnetic flux observations and have been continuously updated and refined. They are widely regarded as informative and physically meaningful indicators for solar flare prediction.
A number of studies have investigated photospheric magnetic field parameters to identify those most relevant to solar flare activity. K. Leka & G. Barnes (2003) found that parameters associated with free magnetic energy, total unsigned flux, vertical current, and shear angle are strongly correlated with flare occurrence. Similarly, O. W. Ahmed et al. (2013) showed that using only six magnetic parameters yields predictive performance comparable to that obtained with all 21 features.
M. G. Bobra & S. Couvidat (2015) systematically evaluated 25 magnetic parameters provided in the SHARP data products and identified several key features, such as TOTUSJH, TOTPOT, and TOTUSJZ, which are particularly effective in distinguishing flaring from nonflaring ARs, consistent with the findings of K. Leka & G. Barnes (2003). Building on this work, C. Liu et al. (2017) selected 13 SHARP parameters for solar flare prediction and further demonstrated that TOTUSJZ, TOTUSJH, and R_VALUE are especially important for distinguishing flares of different classes. Subsequently, H. Ran et al. (2022) analyzed that several SHARP parameters, including SAVNCPP, MEANPOT, USFLUX, SHRGT45, and ABSNJZH exhibit strong correlations with flare occurrence.
In this study, we adopt the SHARP magnetic field parameters listed in Table 1 as the input feature set for solar flare prediction. These parameters are selected because they are important for flare prediction and have been frequently used in recent studies (e.g., C. Liu et al. 2017; R. Tang et al. 2021; Y. Zheng et al. 2023a, 2023b; X. Li et al. 2024).
Table 1. List of SHARP Parameters Used in This Study and a Brief Description of the Parameters
| Keyword | Description |
|---|---|
| TOTUSJH | Total unsigned current helicity |
| TOTPOT | Total photospheric magnetic free energy density |
| TOTUSJZ | Total unsigned vertical current |
| ABSNJZH | Absolute value of the net current helicity |
| SAVNCPP | Sum of the modulus of the net current per polarity |
| USFLUX | Total unsigned flux |
| AREA_ACR | Area of strong field pixels in the active region (AR) |
| MEANPOT | Mean photospheric magnetic free energy |
| R_VALUE | Sum of flux near lolarity inversion line (PIL) |
| SHRGT45 | Fraction of area with shear >45° |
Download table as: ASCIITypeset image
2.2. Dataset Preprocessing
Despite the high quality of the SHARP data products, the raw dataset inevitably contains observational noise, missing values, and low-quality samples. To ensure data reliability, we adopt a strictly cascaded filtering strategy to construct a high-quality experimental dataset.
In the data cleaning phase, we first discard records lacking NOAA AR number and retain only high-quality samples based on the official SDO/HMI Quality Flags. Considering that regions near the solar limb are susceptible to projection effects, we select only ARs located within ±70° of the central meridian. Furthermore, to minimize irrelevant noise and exclude ARs with insufficient scale or information, we implement an adaptive screening strategy based on the solar cycle. Specifically, we independently enforce constraints on both physical scale and image resolution within each cycle: regions falling in the bottom 5% of the area distribution are first removed; subsequently, only samples with a pixel count exceeding 1000 are retained to ensure adequate spatial resolution. Simultaneously, all records containing missing values or physical anomalies are eliminated.
In the preprocessing phase, statistical analysis reveals that key SHARP parameters (e.g., total magnetic flux, current helicity) exhibit significant scale disparities. Using raw values directly can lead to training instability. Therefore, we apply a logarithmic transformation with a unit offset to highly skewed parameters, followed by a unified Z-score standardization for all physical parameters. This process effectively eliminates scale differences and ensures numerical stability during multimodal feature fusion.
2.3. Labeling and Sampling Process
After completing feature selection and data preprocessing, a standardized dataset of magnetogram samples is constructed. In this section, we detail the construction of the datasets used for model training and evaluation.
2.3.1. Sample Labeling
Due to the requirement for labeled training samples in deep learning, we utilize the GOES X-ray Flare Catalog provided by the National Centers for Environmental Information (NCEI).4 Since the NCEI catalog omits AR information for certain events, we implement a stepwise supplementation process.
Specifically, we initially crossmatch records lacking AR identification with the SolarSoft event archive5 to supplement the missing AR numbers. Subsequently, for events still missing AR associations, we manually inspect the corresponding records from SDO.6 After this procedure, 3440 flare records are successfully supplemented. Consequently, the proportion of records lacking AR information is reduced to only 3.1% (649 out of 21,067), which are excluded from the subsequent analysis. The supplemented flare catalog is publicly available on Flare Catalog.7
After a series of preprocessing steps—including the removal of samples with missing ARs, elimination of records with incomplete metadata, and normalization of magnetic parameters—a total of 134,250 magnetogram samples is constructed. This dataset comprises not only visual features from magnetograms but also features derived from their corresponding magnetic parameters, which are of critical importance for analyzing and predicting solar flare activity.
Solar flare prediction is treated as a classification task here. Building on the revised catalog, the study aims to predict whether a C-, M-, or X-class solar flare occurs in a given AR within the next 24 hr, focusing on events that may significantly impact Earth and human activities.
For the binary task of predicting whether a flare occurs, the sample is labeled accordingly. Specifically, based on the records from NCEI, if a C-class or stronger flare occurs within 24 hr following a given time stamp, the corresponding magnetogram sample is labeled as positive; otherwise, it is labeled as negative. The resulting dataset consists of 35,191 positive samples and 99,059 negative samples.
Building upon the binary classification task, the fine-grained prediction task further labels each positive sample according to the most intense flare that occurred within the 24 hr window following the time stamp. Specifically, samples are assigned to three classes: C-class, M-class, and X-class flares. The resulting dataset includes 29,327 C-class samples, 5337 M-class samples, and 527 X-class samples. The detailed distribution is shown in Table 2.
Table 2. List of Datasets Used in This Study
| Category | Dataset | Number | Forecast Window |
|---|---|---|---|
| Two | No-flare | 99,059 | 24 hr |
| Flare (C,M,X) | 35,191 | ||
| Three | C flare | 29,327 | 24 hr |
| M flare | 5337 | ||
| X flare | 527 | ||
Download table as: ASCIITypeset image
2.3.2. Data Sampling and Splitting Strategy
Solar flare prediction datasets inherently exhibit severe class imbalance and strong spatiotemporal correlations. To ensure a rigorous evaluation, we design a comprehensive data-processing pipeline that strictly enforces a splitting-before-sampling sequence implemented via AR-based grouping.
A critical step in prediction is dividing the dataset into training and testing sets. Traditional random splitting leads to information leakage due to highly correlated samples from the same AR, while year-based partitioning suffers from solar cycle variations that can result in an absence of major flares in certain testing periods. To overcome these limitations, we adopt a strict AR-based splitting strategy. All samples from a unique AR number are assigned exclusively to either the training or testing set, ensuring physical isolation between the subsets.
Beyond physical isolation, the sequence of data processing is critical. A common methodological pitfall is applying sampling strategies to the entire dataset prior to splitting. Even when AR-based splitting is subsequently applied, this procedure introduces statistical leakage, as the construction of the training set becomes implicitly coupled with the distribution of the testing set. This violates the fundamental principle that the training pipeline should remain independent of unseen test data.
To address this issue, we strictly adopt a splitting-before-sampling strategy. Under this approach, all samples from a given AR are assigned exclusively to either the training or testing set, with sampling operations applied independently to each subset only after the split is finalized. This approach offers two advantages: (1) it ensures statistical independence by keeping the sampling process entirely separate from the test data, thereby preventing any implicit information leakage; and (2) it prevents the model from memorizing region-specific evolutionary signatures, thereby providing a more reliable assessment of its true generalization capability.
To mitigate the severe class imbalance within the dataset, we employ a hybrid sampling strategy. For the majority class, we apply random undersampling to reduce data redundancy and computational cost. Conversely, for the minority class, we implement geometric data augmentation. These minority-class magnetogram samples undergo discrete rotations and random horizontal or vertical flipping. These transformations exploit the inherent rotational invariance of ARs, effectively expanding the feature space of rare events while encouraging the model to learn robust, orientation-independent features. This data strategy is consistent with methodologies widely adopted in previous solar flare prediction studies (e.g., Y. Zheng et al. 2019, 2021).
Based on these principles, specific datasets are constructed for binary and multiclass prediction tasks.
For the binary prediction task targeting flares of C class and above, we implement a stratified group hold-out strategy with an 8:2 training-testing ratio based on ARs. Based on this splitting, we establish two evaluation environments. Dataset D1 preserves the original imbalanced observational distribution, while Dataset D2 is constructed by applying our hybrid sampling techniques independently to the subsets after the initial split. Crucially, all sampling for D2 is performed independently within each subset after the split to ensure statistical isolation. The detailed class distributions for D1 and D2 are listed in Table 3.
Table 3. List of Binary Prediction Datasets
| Datasets | Forecast Window | No Flare | ≥C Class | |
|---|---|---|---|---|
| D1 | Training | 24 hr | 79,146 | 27,643 |
| Testing | ... | 19,913 | 7548 | |
| D2 | Training | 24 hr | 47,488 | 46,993 |
| Testing | ... | 11,948 | 12,832 |
Download table as: ASCIITypeset image
Building upon the binary framework, we further extend the analysis to a fine-grained multiclass prediction task to evaluate the model’s ability to discriminate among specific flare levels. However, given the extreme scarcity of M- and X-class flares, reliance on a single hold-out split may yield unreliable results and introduce substantial evaluation bias. In extreme cases, this can even result in the complete absence of rare flare classes in the testing set. To ensure evaluation robustness, we adopt a stratified group five-fold cross-validation scheme, splitting the entire dataset into five mutually exclusive folds based on ARs, denoted as F1–F5. In this strategy, each fold sequentially serves as the testing set while the remaining four form the training set. This stratified approach explicitly preserves the relative proportions of C-, M-, and X-class flares across all folds, ensuring that rare events are adequately represented. Consistent with our protocol, all class-balancing operations are performed independently within each fold after the splitting is finalized to prevent information leakage. The resulting distributions for each fold are detailed in Table 4.
Table 4. List of Stratified Group Five-fold Cross-validation Datasets
| Fold | Forecast Window | C | M | X | |
|---|---|---|---|---|---|
| F1 | Training | ... | 4663 | 5081 | 4300 |
| Testing | ... | 1202 | 1324 | 1261 | |
| F2 | Training | ... | 4762 | 5143 | 4410 |
| Testing | ... | 1104 | 1261 | 1118 | |
| F3 | Training | 24 hr | 4607 | 5225 | 4010 |
| Testing | ... | 1258 | 1180 | 1260 | |
| F4 | Training | ... | 4858 | 4925 | 4230 |
| Testing | ... | 1008 | 1480 | 1040 | |
| F5 | Training | ... | 4571 | 5244 | 4130 |
| Testing | ... | 1294 | 1160 | 1140 |
Download table as: ASCIITypeset image
3. Methods
This study proposes a transformer-based multimodal information fusion architecture for solar flare prediction. The model employs a parallel dual-branch transformer structure to enable cross-scale feature extraction and fuses multimodal information from solar magnetograms (image modality) and magnetic parameters (text modality) to predict solar flare activity within the next 24 hr.
3.1. Backbone
As illustrated in Figure 1, the proposed architecture is a dual-branch transformer designed to predict solar flares from multimodal inputs. The backbone consists of three core integrated modules, as follows.
Figure 1. The proposed network architecture consists of three main components: the Magnetogram and Magnetic Parameter Tokenizer (MMP-tokenizer), the Multi-Head Cross-Attention module (MMP-MHCA), and the Multimodal Cross-Scale Interaction Network (MCSI-network). It incorporates M = 3 multimodal transformer encoders, each containing two branches processing magnetogram patches at different scales. Multimodal fusion with the corresponding magnetic parameter tokens is performed via an attention mechanism in each branch. Cross-scale information fusion is achieved by using the CLS tokens from each branch. The extracted feature vectors from magnetogram patches and magnetic parameters are denoted as eimg and epara, respectively. Their fused multimodal representation is denoted as emmp. Here, e represents a feature vector, and the subscript indicates its corresponding modality.
Download figure:
Standard image High-resolution imageMagnetogram and Magnetic Parameter Tokenizer (MMP-Tokenizer). Located at the input stage, this module transforms the heterogeneous inputs into unified token embeddings. It specifically integrates CNN-extracted visual features from magnetograms with the projected physical parameters, mapping these different modalities into a shared feature space for subsequent processing.
Magnetogram and Magnetic Parameter Multi-Head Cross-Attention (MMP-MHCA). Embedded within the transformer encoder of each branch, this module performs deep multimodal fusion. It enables dynamic interaction between visual features and magnetic parameters to compensate for the limitations of single-modal data, ensuring the capture of complementary information from both modalities.
Multimodal Cross-Scale Interaction Network (MCSI-Network). To obtain a comprehensive multiscale representation, this module employs a cross-attention mechanism to fuse features from the parallel branches. It explicitly integrates fine-grained local details with coarse-grained global features, producing a robust embedding that combines the strengths of both spatial views.
3.2. MMP-Tokenizer
To enable transformer-based modeling of heterogeneous data, we propose a unified tokenization module termed MMP-Tokenizer. In our model, this module serves as a modality interface. Its primary function is to project both high-dimensional magnetograms and their associated scalar magnetic parameters into token representations of a consistent dimension (D). This alignment allows for the subsequent Transformer to perform cross-modal attention and fusion within a shared embedding space.
As illustrated in Figure 2, the module consists of two parallel branches. The top branch processes magnetograms into visual tokens using a convolutional network, while the bottom branch transforms magnetic parameters into token embeddings using a feedforward network.
Figure 2. The CNN module for extracting features from magnetograms and the structure for converting magnetic parameters into token representations.
Download figure:
Standard image High-resolution imageAfter obtaining token representations from both branches, these tokens are then utilized at different stages of the model: visual tokens are fed into a transformer encoder for high-level spatial representation learning, while parameter tokens are subsequently fused with encoded visual features through a multimodal fusion module.
For the visual branch, to enhance the representation of magnetogram features, this study adopts a convolution-based patch embedding strategy inspired by the Vision Transformer (ViT) framework (A. Dosovitskiy et al. 2021). Specifically, we replace the standard linear patch projection layer commonly used in ViT architectures with a three-layer convolutional network. Given an input image I, the network first employs a 7 × 7 convolutional layer to capture coarse-grained structural features. This is followed by a 3 × 3 convolutional layer to extract fine-grained local details. A final 3 × 3 convolutional layer projects the output channels to match the target embedding dimension (D). ReLU activation functions are applied after the first two convolutional layers to introduce nonlinearity. Finally, the resulting feature map is reshaped into a sequence of visual tokens only after these spatial features are captured, providing a structured input that can be directly fed into the transformer encoder.
The feature extraction process can be formally described as

Here,
denotes the input magnetogram, where B is the batch size, C is the number of input channels, and H, W represent the height and width of the image, respectively. The symbol σ denotes the ReLU activation function, and Wi and bi represent convolutional weights and biases. The reshape operation flattens the spatial feature map F3 into a sequence of N tokens, each of dimension D, which are directly fed into the transformer encoder.
This convolution-based tokenization strategy offers three main advantages. First, the local connectivity of convolutions helps preserve fine-grained features in magnetograms—such as local gradients and texture patterns—thus improving the quality of spatial representations. Second, the hierarchical structure of convolutional layers progressively expands the receptive field, allowing the network to capture multiscale features that enrich subsequent multimodal fusion. Finally, the inductive bias introduced by CNNs enhances the model’s generalization and stabilizes transformer optimization, particularly when training data is limited. Indeed, our empirical analysis confirms that replacing this module with a standard linear projection significantly reduces performance, showing the critical role of convolutional feature extraction in capturing local magnetic patterns. Simultaneously, magnetic parameters are processed in parallel through the parameter tokenization branch, described below.
In parallel with the CNN-based magnetogram tokenizer, the magnetic parameters are transformed into token representations. Let
denote the input parameter value matrix, where B is the batch size, and Np is the number of magnetic parameters. To enable embedding of parameter values, the input is reshaped to
, such that each parameter value can be individually encoded. The reshaped tensor is then passed through a lightweight feedforward network. Nonlinearity is introduced through ReLU activation, and a LayerNorm layer is applied at the end to stabilize the feature distribution. This results in a tensor of parameter tokens
, where D denotes the target embedding dimension consistent with the visual token space.
This tokenization strategy ensures that the parameter tokens are architecturally aligned with visual tokens, enabling effective multimodal interaction in the subsequent fusion stage.
3.3. Vision Transformer
The CNN transforms the magnetogram into a sequence of D-dimensional visual tokens, each corresponding to a local region of the image. These tokens are then combined with positional encodings (PEs) to retain spatial information and are subsequently fed into the transformer encoder for high-level representation learning.
As shown in Figure 3, the transformer encoder processes the sequence of tokens through multiple blocks. Each block includes layer normalization, multihead self-attention, and a feedforward network, along with residual connections that facilitate feature propagation. This architecture enables the model to capture high-level spatial representations across the entire magnetogram. Finally, the high-level visual features extracted from the transformer are fused with magnetic parameters to enhance the model’s predictive capability.
Figure 3. Architecture of the multibranch transformer encoder module.
Download figure:
Standard image High-resolution image3.4. Cross-attention Fusion
To effectively integrate information from both magnetograms and magnetic parameters, we propose a multimodal attention fusion module, termed MMP-MHCA. Drawing inspiration from the foundational attention architecture (A. Vaswani et al. 2017) and its successful extensions in multimodal learning (e.g., J. Lu et al. 2019; J. Li et al. 2021), our MMP-MHCA module employs a multihead cross-attention (MHCA) mechanism.
This design leverages the mechanism’s inherent ability to fuse heterogeneous features by allowing the query to originate from the visual modality (magnetogram) while the keys and values are derived from the physical parameter modality. This formulation enables fine-grained semantic interaction where magnetic parameters dictate attention-based weighting over the spatial tokens of the magnetograms, thereby allowing the model to selectively emphasize semantically relevant regions for effective feature enrichment.
During the multimodal fusion process, the magnetogram features and magnetic parameter features are denoted as
and
, respectively, where B is the batch size, N is the number of visual tokens, Np is the number of parameter tokens, and D is the feature dimension. Here, Xv serves as the query (Q), and Xt serves as both the key (K) and the value (V). Before computing attention weights via dot-product operations, the input features are linearly projected to obtain the Q , K, and V matrices.

where Wq, Wk, and Wv are the weight matrices for Q, K, and V, respectively.
In the multihead attention mechanism, each head independently learns linear projections to produce Q, K, and V matrices. Specifically, for the ith attention head:

where
,
, and
denote the weight matrices for the ith head, and dhead = D/Nhead is the dimension per head.
Mathematically, the cross-attention weight for each head can be formulated as

where A(i) represents the normalized attention weight matrix for the ith head, capturing the relevance between each spatial Q token and all key positions from the magnetic parameters.
The output of the ith attention head, denoted as Z(i), is computed via scaled dot-product attention as

Each of the N spatial tokens in Z(i) aggregates contextually relevant information based on these attention weights.
Finally, the outputs of all attention heads are concatenated and passed through a linear transformation to generate the final fused representation:

where Wo denotes the output projection matrix.
3.5. Multimodal Cross-scale Encoder
The ViT (A. Dosovitskiy et al. 2021) framework treats images as sequences of patches. Adopting the BERT-style (J. Devlin et al. 2019) sequence modeling paradigm, ViT prepends a learnable class token (CLS) to the input sequence within its bidirectional Transformer architecture. Through the standard self-attention mechanism, this CLS token gathers information from all image patches to construct a compact and discriminative global feature embedding, effectively serving as a global semantic representation.
In our proposed framework, given that multiscale images provide complementary spatial granularities, we employ two parallel branches to process features at different scales. Each branch adopts a similar architecture: the large-scale (L) branch processes larger image patches, while the small-scale (S) branch handles smaller image patches. Within each branch, as the magnetogram passes through the encoder layers, the CLS token first aggregates the specific visual features of that scale. In the subsequent MMP-MHCA module, the CLS token serves as the query, while the magnetic parameters are used as keys and values to enable cross-modal interaction. Through this process, each branch yields the global multimodal embedded magnetogram and magnetic parameters, denoted as
(where i ∈ {l, s}). This representation comprises the updated multimodal CLS token
and the spatial patch tokens
.
Although the multimodal representation emmp provides rich information at each specific scale, relying solely on single-scale features may limit the model’s ability to capture global context across different spatial resolutions. To address this limitation, drawing inspiration from the success of multiscale architectures such as HRNet (K. Sun et al. 2019), as well as recent dual-branch transformers (e.g., C.-F. Chen et al. 2021; Y. Lee et al. 2022; J. D. Havtorn et al. 2023; T. Yao et al. 2023; R. Chen et al. 2025), we introduce the MCSI-Network. It employs an attention-based token interaction strategy to fuse multiscale CLS and epatch representations.
In the subsequent cross-scale interaction stage, ecls serves as the query to interact with epatch from the other branch. Specifically, we first slice the feature vectors from the multimodal sequences emmp based on token positions. The token at the 0th position is extracted as the multimodal CLS token ecls, while the subsequent tokens constitute the spatial patch sequence epatch.
As illustrated in Figure 4, we take the update of the L branch as an example. The CLS token
from the L branch is first projected and then concatenated with the patch tokens
from the S branch. This forms a hybrid sequence
, which is input into the MHCA module. Then,
serves as the keys and values to interact with the query
. Finally, a residual connection is applied to stabilize the training and effectively integrate the cross-scale features. This process is formalized as

where N denotes the patch sequence length; Wq, Wk, and Wv are learnable projection parameters; and fl(·) denotes the projection function used for dimensional alignment. The small-scale branch acquires its corresponding updated CLS token
in a symmetric manner.
Figure 4. Cross-attention information fusion between different branches.
Download figure:
Standard image High-resolution imageSubsequently, the updated multimodal cross-scale CLS token is reinjected into its original position to reconstruct the sequence, preserving the structural integrity of the branch.

After processing by the MCSI-Network, the sequences effectively fuse multimodal and multiscale information and are utilized for the final solar flare prediction.
4. Experiments
In this section, we present the experimental evaluation of the proposed model using magnetograms and corresponding magnetic parameters as inputs. Based on the dataset and labeling strategy described in Section 2.3, the evaluation is structured around two tasks.
Specifically, we first conduct a binary prediction task to assess the model’s ability to distinguish flaring (≥C class) from no-flare samples, and subsequently perform a multiclass prediction task to evaluate its capability in discriminating specific flare classes among C-, M-, and X-class flares.
4.1. Evaluation Metrics
To quantitatively evaluate the model’s performance, key metrics are computed based on the four fundamental components derived from the confusion matrix: true positives (TP), where positive samples are correctly predicted as positive; true negatives (TN), where negative samples are correctly predicted as negative; false positives (FP), where negative samples are incorrectly predicted as positive; and false negatives (FN), where positive samples are incorrectly predicted as negative. These components form the foundation for computing key performance metrics, including accuracy, precision, recall, false-alarm rate (FAR), Heidke skill score (HSS), true skill statistic (TSS), frequency bias (FB), and F1-score, as defined in Table 5.
Table 5. Evaluation Metrics of the Model
| Metric | Definition |
|---|---|
| Accuracy |
|
| Precision |
|
| Recall |
|
| FAR |
|
| HSS |
|
| TSS |
|
| F1-score |
|
| Frequency bias |
|
Download table as: ASCIITypeset image
In solar flare prediction tasks, the severe class imbalance often results in uneven sample distributions, which, in turn, affects the effectiveness of conventional evaluation metrics such as accuracy and precision (e.g., D. S. Bloomfield et al. 2012; M. G. Bobra & S. Couvidat 2015; X. Li et al. 2024). To ensure a more reliable evaluation, this study adopts TSS, HSS, F1-score, and FB as the primary performance metrics. TSS, ranging from −1 to 1, measures the ability to distinguish between positive and negative classes and is insensitive to class imbalance. FB indicates systematic over- or underforecasting, a necessary complement to the TSS metric (K. Leka et al. 2019). HSS quantifies the improvement over random predictions, with scores ranging from −∞ to 1. The F1-score, defined as the harmonic mean of precision and recall, ranges from 0 to 1 and is widely adopted for evaluating performance in imbalanced prediction tasks due to its balanced consideration of false positives and false negatives.
4.2. Experimental Setup
All experiments are implemented using the pytorch framework and executed on an NVIDIA RTX 4090 GPU. The input magnetograms are uniformly resized to 240 × 240 pixels. For model optimization, we employ the AdamW optimizer with an initial learning rate of 10−5, a weight decay of 0.05, and a batch size of 32. We adopt a cosine annealing scheduler with linear warmup, where the learning rate gradually decays to a minimum. To further mitigate overfitting, we apply stochastic depth regularization with a drop-path rate of 0.2, and consistently employ focal loss as the loss function across all tasks. The maximum training epochs are set to 100, during which both training and testing losses are recorded to monitor learning dynamics.
4.3. Experimental Result
To validate the effectiveness of the model and the stability of the training process, we first visualize the training and testing loss curves across different datasets. Figures 5(a)–(b) illustrate the loss trajectories for the binary task of predicting ≥C-class flares, while Figures 5(c)–(d) display the results for the multiclass discrimination of C-, M-, and X-class flares. As the number of epochs increases, the overall loss across all tasks decreases steadily, indicating that the model is learning effective representations.
Figure 5. The curves show the loss dynamics of our proposed model during training and testing. Curves of different colors represent the loss trajectories across epochs for different datasets. Panels (a) and (c) depict the training loss variations for each dataset, while panels (b) and (d) show the corresponding testing loss variations.
Download figure:
Standard image High-resolution imageFor the binary prediction task, although the model achieves convergence on both datasets, the testing loss on the baseline dataset D1 is significantly higher than on D2, accompanied by noticeable fluctuations and a wider generalization gap, defined as the difference between training and testing losses. This indicates that on the imbalanced D1 dataset, the model predominantly learns the features of the majority no-flare samples, making it difficult to capture the key discriminative features of the minority flare class. In contrast, the significantly lower test loss observed on D2 implies that the model is able to focus more attention on the minority flare samples, thereby learning more distinguishable features rather than being dominated by the majority class.
In the multiclass task, Figures 5(c)–(d) present the loss distributions across the five stratified group folds (F1–F5). The curves exhibit highly consistent trends with minimal variance between folds in both training and testing phases. This uniformity confirms that the model achieves stable convergence and robust generalization performance, ensuring that the evaluation is not biased by specific data splits.
4.3.1. Binary Prediction Results
As shown in Figure 6(a), the model performance is first evaluated on D1. Although the model exhibits an upward trend throughout the training process, ultimately achieving an F1-score of 0.59, skill metrics remain relatively limited, with TSS and HSS scores of only 0.42 and 0.40, respectively. Figure 6(b) presents the evaluation results on the balanced D2. Compared with the results on D1, the discrepancy between skill scores and basic metrics narrows significantly. Specifically, recall rises to 0.79, and both TSS and HSS exceed 0.63, closely aligning with the high F1-score of 0.83. The comprehensive improvement across key metrics indicates that the model has successfully captured the intrinsic characteristics of the minority class.
Figure 6. The radar charts illustrate the evolution of recall, TSS, HSS, and F1 scores across training epochs on datasets D1 and D2.
Download figure:
Standard image High-resolution imageBased on these findings, Table 6 presents a quantitative comparison between our model and representative existing studies under unified prediction conditions. The results indicate that our model achieves superior performance across multiple key evaluation metrics. With TSS, HSS, and F1-score reaching 0.661, 0.658, and 0.827, respectively, our model significantly outperforms the corresponding results reported by R. Tang et al. (2021) and X. Huang et al. (2018). Although N. Nishizuka et al. (2018) reported a recall of 0.809, which is slightly higher than our 0.789, this sensitivity is achieved at the cost of precision, which drops to 0.529, indicating a tendency toward overforecasting and a high FAR. In contrast, our model achieves a high precision of 0.868, demonstrating its ability to effectively suppress false alarms while maintaining high sensitivity. This robust balance between recall and precision is further reflected in the superior F1-score of 0.827, which markedly exceeds those of the compared methods.
Table 6. Flare Prediction Results of Our Model for ≥C Class and a Comparison with Previous Studies (within 24 hr).
| Metric | Model | ≥C Class |
|---|---|---|
| Recall | This work | 0.789 |
| R. Tang et al. (2021) | 0.592 | |
| X. Huang et al. (2018) | 0.726 | |
| N. Nishizuka et al. (2018) | 0.809 | |
| Precision | This work | 0.868 |
| R. Tang et al. (2021) | 0.662 | |
| X. Huang et al. (2018) | 0.352 | |
| N. Nishizuka et al. (2018) | 0.529 | |
| Accuracy | This work | 0.829 |
| R. Tang et al. (2021) | 0.820 | |
| X. Huang et al. (2018) | 0.756 | |
| N. Nishizuka et al. (2018) | 0.822 | |
| HSS | This work | 0.658 |
| R. Tang et al. (2021) | ⋯ | |
| X. Huang et al. (2018) | 0.339 | |
| N. Nishizuka et al. (2018) | 0.528 | |
| TSS | This work | 0.661 |
| R. Tang et al. (2021) | 0.535 | |
| X. Huang et al. (2018) | 0.487 | |
| N. Nishizuka et al. (2018) | 0.634 | |
| F1-score | This work | 0.827 |
| R. Tang et al. (2021) | 0.624 | |
| X. Huang et al. (2018) | 0.475 | |
| N. Nishizuka et al. (2018) | ⋯ | |
| FB | This work | 0.909 |
| R. Tang et al. (2021) | ⋯ | |
| X. Huang et al. (2018) | ⋯ | |
| N. Nishizuka et al. (2018) | ⋯ | |
Download table as: ASCIITypeset image
Furthermore, our model exhibits an FB of 0.909, which is close to the ideal value of 1.0. This result indicates that the model has no significant systematic bias in its overall prediction tendency, and its forecasted frequency aligns closely with actual observations. Overall, the proposed method demonstrates a more balanced and robust comprehensive performance across both basic and skill-based metrics.
4.3.2. Multiclass Prediction Results
We analyze the performance of the model on five datasets (F1–F5). Figure 7 shows the trends of key performance metrics for each flare class across training epochs. All metrics across F1–F5 show consistent improvement with increasing epochs, suggesting the model’s robust capacity for continual learning.
Figure 7. The radar charts illustrate the evolution of TSS, HSS, and F1 scores for C-, M-, and X-class flares across training epochs under datasets F1–F5. The results are arranged vertically in the figure, with C class at the top, M class in the middle, and X class at the bottom.
Download figure:
Standard image High-resolution imageMost notably, for X-class flares, as illustrated in Figures 7(g)–(i), the model demonstrates superior and highly consistent performance. The metrics consistently fall within a high range, with TSS and HSS exceeding 0.80, and the F1-score peaking at approximately 0.88. Such outstanding performance validates the effectiveness of the multimodal fusion approach in extracting discriminative features from X-class magnetograms and their associated magnetic parameters. This may be attributed to the inherently distinctive and easily separable nature of X-class samples. Despite their scarcity, these samples exhibit pronounced differences from other flare types, thereby enabling more effective representation learning by the model.
For C-class flares, as illustrated in Figures 7(a)–(c), the model maintains a solid baseline performance. The F1-score reaches up to 0.77, while TSS and HSS generally stabilize in the range of 0.51–0.65. Notably, the F3 and F5 subsets consistently yield the best performance across all three indicators, indicating that their sample feature distributions are more favorable for modeling C-class flares. Although these scores are lower than those of the X class, they represent a reasonable balance between precision and recall in identifying C-class events.
However, the primary challenge lies in M-class prediction. As shown in Figures 7(d)–(f), the performance metrics exhibit noticeable fluctuations compared to the C and X classes, with HSS and TSS remaining in relatively low ranges and limited improvement in F1-score. This U-shaped performance profile suggests that M-class flares likely possess overlapping features with both weaker (C-class) and stronger (X-class) flares, making the decision boundaries more difficult to resolve. Nevertheless, the upward trend in metrics throughout the training epochs indicates that the model is gradually learning to mitigate this ambiguity over time.
To evaluate the model’s predictive performance across multiple classes (C, M, and X), our average results are compared with those from prior studies that conducted predictions (see Table 7).
Table 7. Multiclass Flare Prediction Results of Our Model and a Comparison with Previous Studies (within 24 hr).
| Metric | Model | C | M | X |
|---|---|---|---|---|
| Recall | This work | 0.742 ± 0.030 | 0.551 ± 0.058 | 0.857 ± 0.059 |
| Y. Zheng et al. (2021) | 0.644 ± 0.083 | 0.518 ± 0.272 | 0.523 ± 0.398 | |
| Y. Zheng et al. (2019) | 0.671 ± 0.059 | 0.617 ± 0.148 | 0.594 ± 0.394 | |
| C. Liu et al. (2017) | 0.526 ± 0.050 | 0.671 ± 0.037 | 0.297 ± 0.039 | |
| Precision | This work | 0.686 ± 0.066 | 0.608 ± 0.048 | 0.840 ± 0.035 |
| Y. Zheng et al. (2021) | 0.638 ± 0.041 | 0.659 ± 0.060 | 0.502 ± 0.346 | |
| Y. Zheng et al. (2019) | 0.670 ± 0.079 | 0.699 ± 0.087 | 0.562 ± 0.383 | |
| C. Liu et al. (2017) | 0.563 ± 0.054 | 0.656 ± 0.036 | 0.745 ± 0.152 | |
| Accuracy | This work | 0.806 ± 0.023 | 0.714 ± 0.025 | 0.902 ± 0.015 |
| Y. Zheng et al. (2021) | 0.791 ± 0.039 | 0.830 ± 0.031 | 0.893 ± 0.065 | |
| Y. Zheng et al. (2019) | 0.812 ± 0.029 | 0.849 ± 0.034 | 0.933 ± 0.041 | |
| C. Liu et al. (2017) | 0.712 ± 0.026 | 0.778 ± 0.019 | 0.957 ± 0.005 | |
| FAR | This work | 0.314 ± 0.066 | 0.392 ± 0.048 | 0.160 ± 0.035 |
| Y. Zheng et al. (2021) | 0.361 ± 0.041 | 0.340 ± 0.060 | 0.297 ± 0.281 | |
| Y. Zheng et al. (2019) | 0.330 ± 0.079 | 0.301 ± 0.087 | 0.138 ± 0.140 | |
| C. Liu et al. (2017) | 0.437 ± 0.016 | 0.344 ± 0.020 | 0.255 ± 0.126 | |
| HSS | This work | 0.566 ± 0.056 | 0.361 ± 0.047 | 0.775 ± 0.041 |
| Y. Zheng et al. (2021) | 0.488 ± 0.047 | 0.444 ± 0.194 | 0.419 ± 0.309 | |
| Y. Zheng et al. (2019) | 0.535 ± 0.061 | 0.551 ± 0.120 | 0.539 ± 0.366 | |
| C. Liu et al. (2017) | 0.334 ± 0.028 | 0.497 ± 0.031 | 0.406 ± 0.014 | |
| TSS | This work | 0.579 ± 0.045 | 0.356 ± 0.051 | 0.780 ± 0.051 |
| Y. Zheng et al. (2021) | 0.489 ± 0.049 | 0.432 ± 0.222 | 0.436 ± 0.330 | |
| Y. Zheng et al. (2019) | 0.538 ± 0.059 | 0.534 ± 0.137 | 0.552 ± 0.370 | |
| C. Liu et al. (2017) | 0.328 ± 0.050 | 0.500 ± 0.037 | 0.291 ± 0.039 | |
| FB | This work | 1.087 ± 0.091 | 0.912 ± 0.131 | 1.022 ± 0.088 |
| Y. Zheng et al. (2021) | ⋯ | ⋯ | ⋯ | |
| Y. Zheng et al. (2019) | ⋯ | ⋯ | ⋯ | |
| C. Liu et al. (2017) | ⋯ | ⋯ | ⋯ | |
| F1-score | This work | 0.712 ± 0.043 | 0.576 ± 0.037 | 0.847 ± 0.032 |
| Y. Zheng et al. (2021) | 0.489 ± 0.049 | 0.432 ± 0.222 | 0.436 ± 0.330 | |
| Y. Zheng et al. (2019) | ⋯ | ⋯ | ⋯ | |
| C. Liu et al. (2017) | ⋯ | ⋯ | ⋯ | |
Download table as: ASCIITypeset image
For X-class flares, the proposed method demonstrates superior performance, significantly outperforming existing approaches (e.g., C. Liu et al. 2017; Y. Zheng et al. 2019, 2021) in terms of HSS (0.775) and TSS (0.780). Notably, the FAR is remarkably low at 0.160, indicating a minimal false-positive rate. Moreover, the FB of 1.022 ± 0.088 suggests that the model exhibits negligible systematic bias, effectively avoiding the tendency toward significant over- or underforecasting. Additionally, the F1-score reaches 0.847, reflecting a well-balanced performance between precision and recall. These results highlight the model’s high sensitivity and robust predictive capability for intense flare events, underscoring its potential utility in severe space-weather forecasting.
For C-class flares, the model achieves robust results, with an HSS of 0.566 and a TSS of 0.579—both significantly higher than those reported by Y. Zheng et al. (2021, 2019) and C. Liu et al. (2017). The F1-score reaches 0.712 ± 0.043, notably outperforming the 0.489 reported by Y. Zheng et al. (2021). Furthermore, the FAR is reduced to 0.314 ± 0.066, which is markedly lower than the values reported by Y. Zheng et al. (2019) (0.330 ± 0.079), C. Liu et al. (2017) (0.437 ± 0.016), and Y. Zheng et al. (2021) (0.361 ± 0.041). These metrics indicate that the model not only effectively identifies C-class flares but also substantially suppresses false alarms, demonstrating its practical applicability. While M-class performance shows a slight compromise compared to previous works, this trade-off is strategically acceptable. The slight dip in M-class performance has enabled the model to prioritize and maximize the detection capabilities for the most destructive X-class events, which is the primary objective of operational space-weather forecasting.
In conclusion, the proposed dual-branch multimodal fusion model demonstrates exceptional robustness and discriminative ability, particularly in identifying X-class flares. Compared with existing approaches (Y. Zheng et al. 2021, 2019; C. Liu et al. 2017), our model achieves consistently superior performance across multiple metrics for X-class and C-class events, while effectively reducing the FAR. Although the model exhibits slightly lower performance on certain metrics for M-class flares compared to some previous studies, its overall performance remains competitive. These findings are well aligned with the core research objective of focusing on major solar flare eruptions.
It should be noted that the current field lacks a standardized and publicly available test set, making it difficult to conduct fair and quantitative comparisons across different methods (e.g., G. Barnes et al. 2016; R. Tang et al. 2021). Therefore, the results presented are intended to serve as a valuable point of reference rather than a definitive performance comparison, demonstrating the proposed method’s potential for operational early warning applications.
4.4. Ablation Studies
To thoroughly verify the effectiveness of the model and investigate the specific contributions of key components and experimental settings, we design and implement four sets of rigorous ablation studies.
First, we establish a model relying solely on line-of-sight magnetograms (corresponding to the visual branch in Figure 2) as the magnetogram-only baseline. By comparing this baseline with the complete fusion model, we quantify the performance gains introduced by the physical parameter branch. Subsequently, we evaluate the interaction efficiency of different feature fusion strategies. Furthermore, addressing details of data preprocessing and model architecture, we analyze the effects of input image resolution and the distinct roles of PE within the magnetogram and physical parameter branches. In conclusion, these experiments verify the effectiveness and robustness of the proposed dual-branch multimodal fusion model from multiple aspects.
4.4.1. Multimodal Inputs
To validate the effectiveness of the proposed multimodal fusion strategy, we conduct a comprehensive performance comparison among the complete fusion model and the two single-modality baselines, namely the magnetogram-only model with the parameter branch removed and the magnetic parameter-only model using only the SHARP parameters as input. This experiment investigates whether integrating strong physical priors as a supplement to implicit visual features can provide critical complementary information to better capture flare precursor signatures.
The experimental results are presented in Table 8. The fusion model achieves the best overall performance with a peak TSS of 0.661. While both single-modality baselines exhibit slightly better FB values, their independent predictive capabilities are compromised by elevated FARs, which results in a decline in key metrics such as ACC, TSS, and HSS. Compared to the magnetogram-only baseline, the multimodal approach indicates that magnetic parameters, acting as highly condensed physical representations, provide stable global constraints for the model. This synergy between the two modalities effectively suppresses local noise interference, thereby markedly improving the accuracy and stability of flare predictions.
Table 8. Performance Comparison among the Multimodal Fusion Model, Magnetogram-only Baseline, and Physical Parameter-only Baseline
| Model | Mag. | Param. | PEMag | PEParam | HSS | TSS | FB | FAR | ACC |
|---|---|---|---|---|---|---|---|---|---|
| Fusion | ✓ | ✓ | ✓ | ✗ | 0.658 | 0.661 | 0.909 | 0.132 | 0.829 |
| Mag-only | ✓ | ✗ | ✓ | ✗ | 0.619 | 0.646 | 0.919 | 0.143 | 0.822 |
| Param-only | ✗ | ✓ | ✗ | ✗ | 0.527 | 0.527 | 1.0002 | 0.228 | 0.764 |
Note. Mag. and Param. denote the magnetogram and magnetic parameter modalities, respectively. PEMag and PEParam indicate that positional encoding is applied only to the magnetogram branch and the physical parameter branch, respectively.
Download table as: ASCIITypeset image
4.4.2. Feature Fusion Strategies
Serving as a critical bridge between the magnetogram and physical parameter modalities, the multimodal interaction module plays a core role in enhancing the discriminability and feature alignment of cross-modal representations. As illustrated in Figure 8, we systematically explore various feature fusion strategies. Despite variations in their specific architectures, all strategies follow a unified design principle: using the magnetogram modality as the primary information stream and incorporating parameter information to enrich feature representation and achieve cross-modal coordination. The corresponding quantitative comparison is summarized in Table 9.
Figure 8. Comparison of multimodal interaction strategies between magnetograms and physical parameters.
Download figure:
Standard image High-resolution imageTable 9. Performance Comparison of Different Feature Fusion Strategies
| Model | Mag. | Param. | PEMag | PEParam | HSS | TSS | FB | FAR | ACC |
|---|---|---|---|---|---|---|---|---|---|
| CA | ✓ | ✓ | ✓ | ✗ | 0.658 | 0.661 | 0.909 | 0.132 | 0.829 |
| Concat. | ✓ | ✓ | ✓ | ✗ | 0.635 | 0.653 | 0.947 | 0.150 | 0.826 |
| Add. | ✓ | ✓ | ✓ | ✗ | 0.620 | 0.648 | 0.914 | 0.140 | 0.823 |
| Inv-CA | ✓ | ✓ | ✓ | ✗ | 0.623 | 0.646 | 0.931 | 0.147 | 0.822 |
Note. Same as in Table 8. CA, Concat., and Add. denote the cross-attention, concatenation, and direct addition fusion strategies, respectively. Inv-CA denotes the inverse cross-attention strategy, where physical parameters serve as queries and magnetograms serve as keys/values.
Download table as: ASCIITypeset image
We first compare the effectiveness of different fusion operations. The concatenation strategy achieves modality integration by feature concatenation and linear projection, followed by self-attention within the joint feature space to obtain fused representations. Although its TSS (0.653) is relatively close to that of the cross-attention fusion strategy (0.661), it exhibits a noticeably higher FAR, accompanied by a decline in overall accuracy. This suggests that simple linear concatenation has limitations in modeling cross-modal dependencies and may introduce additional false alarms. In contrast, the direct addition strategy learns fused features by combining linear mappings and nonlinear transformations. However, the results indicate that this approach offers no distinct advantages across key metrics such as TSS, FAR, and ACC. Compared with the cross-attention strategy, it fails to yield benefits in either predictive accuracy or computational efficiency.
Furthermore, to investigate the impact of cross-modal interaction direction on performance, we compare two attention mechanism designs. In the standard cross-attention, we use the magnetogram as the query, while physical parameter features serve as keys and values, thereby embedding physical prior information into the spatial magnetogram representations. To validate the necessity of this design, we construct a variant model named Param-Q (row 4), which conversely treats physical parameters as the query, attempting to integrate the visual information of magnetograms into parameter features. However, experimental results indicate a significant performance drop for Param-Q, even falling below the direct addition strategy. This result strongly suggests that, within the proposed model, using the magnetogram as the primary feature carrier is more conducive to preserving discriminative spatial structural features while embedding complementary magnetic parameter information.
Based on this comparative analysis, we adopt the cross-attention mechanism as the fusion strategy for the multimodal interaction module.
4.4.3. Positional Encoding
Positional information plays a crucial role in characterizing the spatial morphological evolution of ARs. To systematically analyze the role of PE across different modalities, as shown in Table 10, we design and compare three PE configuration strategies, applying PE only to the magnetogram branch (Mag-PE), applying PE to neither branch (No-PE), and applying PE to both branches simultaneously (All-PE).
Table 10. Performance Comparison of the Model with and without Positional Encoding on Magnetogram and Parameter Branches
| Model | Mag. | Param. | PEMag | PEParam | HSS | TSS | FB | FAR | ACC |
|---|---|---|---|---|---|---|---|---|---|
| Mag-PE | ✓ | ✓ | ✓ | ✗ | 0.658 | 0.661 | 0.909 | 0.132 | 0.8288 |
| No-PE | ✓ | ✓ | ✗ | ✗ | 0.645 | 0.658 | 0.964 | 0.153 | 0.8284 |
| All-PE | ✓ | ✓ | ✓ | ✓ | 0.635 | 0.659 | 0.927 | 0.139 | 0.8285 |
Note. Same as in Table 8. Mag-PE denotes positional encoding only to the magnetogram branch; No-PE represents the setting without positional encoding, and All-PE applies positional encoding to all modalities.
Download table as: ASCIITypeset image
The experimental results demonstrate that the Mag-PE strategy achieves the best overall performance balance. Although the No-PE strategy obtains a marginally higher HSS (0.645) and an FB closer to the ideal value (0.964), this improvement comes at the cost of an elevated FAR (0.153). In contrast, the Mag-PE strategy achieves the highest TSS of 0.661 while suppressing the FAR to 0.132. This indicates that explicit spatial positional information enables the model to more precisely focus on critical structural features associated with flare triggering within magnetograms, thereby effectively filtering background noise while maintaining stable overall accuracy.
Conversely, when PE is applied to both branches (All-PE), model performance degrades compared to Mag-PE, with TSS dropping to 0.659 and FAR rising to 0.139. This phenomenon suggests that physical parameters, serving as highly condensed global statistics, inherently lack the spatial arrangement dependencies characteristic of image pixels. Imposing positional constraints on this modality may disrupt its original semantic structure and introduce redundant interference, thereby weakening the cross-modal feature fusion.
In summary, to achieve a balance between flare capture capability (high TSS) and warning reliability (low FAR), the Mag-PE strategy is adopted as the default PE configuration in this work.
4.4.4. Image Resolution
To evaluate the impact of magnetogram resolution on model performance, we conduct a comparative analysis of three resolution settings 160 × 160, 240 × 240, and 320 × 320 pixels. The experimental dataset spans the observation period from 2010 May 1, to 2023 May 1, comprising N ARs for training and M ARs for testing. The results are summarized in Table 11.
Table 11. Performance Comparison of Models Using Different Input Image Resolutions
| Resolution | Mag. | Param. | PEMag | PEParam | HSS | TSS | FB | FAR | ACC |
|---|---|---|---|---|---|---|---|---|---|
| 160 × 160 | ✓ | ✓ | ✓ | ✗ | 0.613 | 0.653 | 0.876 | 0.123 | 0.825 |
| 240 × 240 | ✓ | ✓ | ✓ | ✗ | 0.658 | 0.661 | 0.909 | 0.132 | 0.829 |
| 320 × 320 | ✓ | ✓ | ✓ | ✗ | 0.621 | 0.660 | 0.884 | 0.122 | 0.828 |
Note. Same as in Table 8.
Download table as: ASCIITypeset image
The results indicate that the resolution to 160 × 160 leads to the loss of fine-grained textural details, thereby diminishing the predictive accuracy of the model. Conversely, increasing the magnetograms resolution to 320 × 320 yields no additional gains, suggesting that performance reaches saturation. This phenomenon is likely attributable to two primary factors: (1) computational constraints force a reduction in batch size, leading to prolonged training duration and potential optimization instability; and (2) the receptive field or parameter capacity of the current architecture appears insufficient to fully exploit fine-grained features beyond the 240 × 240 scale. Consequently, based on the trade-off between computational efficiency and predictive precision, we adopt the input resolution of magnetograms to 240 × 240 pixels.
4.5. Feature Visualization
To address the challenge of interpretability in deep learning models, we conduct a multidimensional investigation into the feature layers (e.g., N. Nishizuka et al. 2018; Y. Zheng et al. 2021). In the domain of computer vision, the gradients of feature maps typically characterize texture variation rates or edge sharpness; in photospheric magnetograms, however, this corresponds directly to the intensity of magnetic field gradients, a critical physical quantity for flare eruptions (D. Falconer et al. 2011). As shown in Figure 9, we visualize the median gradient and total intensity across feature channels for different flare classes. The analysis reveals that the model does not treat all channels equally; instead, sharp response peaks emerged only in specific dominant channels.
Figure 9. Statistical comparison of feature channel responses across flare classes. The upper and lower panels display the median gradient and total intensity of the feature extraction layer across channels, respectively. Curves of different colors correspond to different flare classes.
Download figure:
Standard image High-resolution imageNotably, in these key channels, the feature response intensity exhibits a strict hierarchical distribution (i.e., X class > M class > C class). This significant differentiation suggests that the model has autonomously learned a feature representation strategy, relying on a set of highly discriminative feature representations to identify complex magnetic structures within ARs, rather than weighting all feature channels uniformly. Crucially, this strong activation in specific channels reflects the model’s acute capture of sharp variations in magnetic structures. Consequently, it directly demonstrates that the learned feature extraction mechanism is highly consistent with the physical nature of energy release.
To further quantify the connection between the internal representations of the model and physics mechanisms, we analyze the correlation between the network median gradient extracted by the model and the classic physical parameter. These parameters are MEANPOT, TOTPOT, and R_VALUE (e.g., M. G. Bobra et al. 2014; J. Wang et al. 2019). As illustrated in Figure 10, unlike channel-specific metrics, this value is derived by calculating the median of gradient magnitudes across the entire feature tensor, which encompasses all spatial dimensions and channels. This global aggregation strategy ensures a holistic assessment of the topological complexity of the sample.
Figure 10. Scatter plots illustrate the relationships between the global network median gradient and MEANPOT, TOTPOT, and R_VALUE. Black dashed lines indicate linear regression trends with confidence intervals denoted by gray shading. Pearson correlation coefficients are annotated at the top of each panel.
Download figure:
Standard image High-resolution imageThe results demonstrate a significant positive correlation between the network gradient response and all three physical parameters, with Pearson correlation coefficients of 0.44, 0.43, and 0.39, respectively. This finding indicates that the feature representations of the model are more strongly activated when the input magnetogram exhibits higher magnetic free energy density or stronger magnetic flux near the PIL (e.g., K. Leka & G. Barnes 2003; D. Cicogna et al. 2021). Such statistical consistency confirms that the internal feature extraction mechanism aligns closely with the physical nature of magnetic parameters (H. Ran et al. 2022).
To verify whether the model spatially locks onto the physical trigger source of eruptions, we visualize the attention maps generated by the model. We then compare them spatially with the calculated strong vertical magnetic field gradient and the strong-gradient PIL (e.g., D. Cicogna et al. 2021; K. Yi et al. 2021). Figures 11–13 illustrate the spatiotemporal trajectory of the attention evolution for different flare classes.
As shown in the first row of Figures 11–13, the attention map exhibits low-intensity activation features characterized by cool colors, with a relatively divergent focus. Consistent with this physical state, the magnetic topology of the corresponding AR is relatively loose, and the PIL presents a fragmented morphology. This indicates that the model maintains a low alert level at this stage.
Figure 11. Spatiotemporal evolution of model attention. Rows illustrate the evolution of model attention within the prediction window for an X-class flare event. Columns from left to right display: (a) the original line-of-sight magnetogram; (b) the model attention heatmap, where warm colors indicate high activation; (c) the overlay of the magnetogram and heatmap; and (d) the magnetogram superimposed with features where red shading marks strong magnetic gradient regions, and green contours delineate the strong-gradient polarity inversion line (PIL). Each block is 240 × 240 pixels in size.
Download figure:
Standard image High-resolution imageFigure 12. Same as Figure 11, but for an M-class flare event.
Download figure:
Standard image High-resolution imageFigure 13. Same as Figure 11, but for a C-class flare event.
Download figure:
Standard image High-resolution imageThe second row reveals a distinct shift in the perception of the model as the attention focus automatically converges on the central region where positive and negative magnetic poles approach each other. At this moment, the area of high-gradient regions indicated by red shading expands, and the PIL begins to elongate and complicate, signaling the injection of nonpotential energy (X. Yang et al. 2013). This synchronous enhancement proves that the model keenly captures the dynamic process of flux emergence and responds in real-time to the accumulation of magnetic stress (M. Korsós et al. 2015).
In the third row, the AR displays a highly nonpotential configuration where a compact high-gradient region tightly wraps around a twisted PIL. The high-intensity hotspot in the attention map spatially coincides precisely with the physical strong-gradient PIL. This high degree of morphological consistency confirms that the model has successfully locked onto the physical core region responsible for the eruption.
In summary, the evolutionary process from dispersed low activation to focused high activation provides compelling evidence that the proposed model is not a black box. Instead, it effectively integrates local magnetogram textures into semantic physical precursors, thereby establishing its physical credibility in solar flare forecasting.
5. Discussions and Conclusions
This study demonstrates the efficacy of a dual-branch multimodal fusion model for 24 hr solar flare prediction. The analysis encompasses binary classification and extends to a multiclass task targeting fine-grained C-, M-, and X-class flare intensities. To strictly evaluate the model’s robustness, we implement a stratified group five-fold cross-validation strategy across five constructed subsets (F1–F5), reporting the average metrics to demonstrate robust generalization.
The reliability of solar flare prediction models depends heavily on the integrity of the data splitting and evaluation strategy. In this study, we emphasize the necessity of the splitting-before-sampling strategy, which fundamentally departs from the sampling-before-splitting approach common in many previous studies (e.g., Y. Zheng et al. 2019, 2021). By ensuring that all class-balancing operations are performed independently within each partition only after the split is finalized, we eliminate potential statistical coupling between training and testing data. This rigor is essential for providing an unbiased assessment of a model’s true generalization capability on unseen solar ARs.
Furthermore, our dual-dataset evaluation (D1 and D2) for binary tasks offers a more comprehensive perspective than traditional balanced-only evaluations. While the balanced dataset D2 provides a controlled environment to assess the model’s sensitivity to flare features, the original-distribution dataset D1 serves as a critical operational benchmark. By maintaining the inherent class imbalance of real-world observations in D1, we can better estimate the model’s performance in practical forecasting scenarios, where the overwhelming majority of samples are nonflaring.
For multiclass discrimination, the adoption of stratified group five-fold cross-validation addresses the statistical instability inherent in rare event prediction. Unlike standard k-fold or year-based splitting (e.g., X. Li et al. 2020; N. Nishizuka et al. 2021), our stratified approach ensures that extremely rare M- and X-class flares are proportionally represented in every fold. This prevents the empty class problem in testing partitions and ensures that the reported average metrics across F1–F5 are statistically robust. Ultimately, this framework provides a rigorous standard for evaluating flare intensity discrimination, ensuring that the model learns feature representation rather than memorizing region-specific evolutionary signatures or benefiting from information leakage.
Among existing approaches, prior work such as N. Nishizuka et al. (2018) relies on manual feature selection based on domain knowledge, which limits the ability to capture potentially critical features and reduces adaptability to changes in data distribution. Furthermore, some studies use single-modal inputs, utilizing either magnetograms (e.g., X. Huang et al. 2018; X. Li et al. 2020; Y. Zheng et al. 2023a) or magnetic parameters (e.g., Y. Zheng et al. 2023b; X. Li et al. 2024), thereby overlooking the complementary information inherent across modalities. Although R. Tang et al. (2021) incorporated multiple architectures and multimodal data, fusion is performed only at the decision level, resulting in limited interaction. In addition, while Y. Zheng et al. (2023a) compared magnetogram-based and parameter-based models on a unified dataset, their physical model relied solely on time-series magnetic parameters, and the magnetogram-based model employed only single-scale magnetograms. These simplified designs neither enable effective complementary fusion between magnetogram and parameter modalities nor exploit multiscale feature representations.
In contrast, the proposed dual-branch multimodal fusion model separately extracts features from magnetograms and magnetic parameters integrated with cross-attention mechanisms. Subsequently, we introduce a cross-scale interactions mechanism to facilitate multiscale feature fusion. Trained in an end-to-end manner, the model significantly enhances both feature representation capability and generalization performance. The consistent superiority of our model, particularly in predicting X-class flares, validates the efficacy of the multimodal fusion framework. As shown in the ablation study in Table 8, the integration of the physical parameter branch not only enhances prediction accuracy but, more critically, suppresses the FAR, thereby boosting overall performance metrics.
This improvement underscores the intrinsic complementary mechanism between the two modalities. While networks are adept at extracting implicit spatial patterns from magnetograms, their sensitivity to pixel-level details renders them susceptible to interference from projection effects and high-frequency observational noise. Addressing this, SHARP parameters serve as a vital supplement. By providing a condensed representation of magnetic properties—such as total magnetic flux and current helicity—these parameters directly characterize the topological complexity and nonpotential energy accumulation of ARs. Consequently, incorporating these strong physical priors acts as a form of regularization, making the model robust against local image perturbations. Fundamentally, this fusion strategy anchors the decision-making process to the core physics of flare eruptions rather than mere image pattern matching, effectively preventing the network from overfitting to superficial morphological features.
The main results of this work are summarized as follows. (1) We propose a dual-branch framework that utilizes multimodal and cross-scale interaction mechanisms to deeply fuse magnetograms with magnetic parameters. This end-to-end architecture effectively integrates complementary spatial and physical features, significantly enhancing representation learning and performance of solar flare prediction compared to single-modal designs. (2) This study introduces a stratified group five-fold cross-validation scheme and adopts a splitting-before-sampling strategy. The scheme ensures nonoverlapping data sources between training and testing sets to prevent data leakage, maintains class proportions consistent with the overall dataset across all folds, and guarantees the presence of representative samples from all classes in both subsets. (3) The model demonstrates superior predictive performance on X-class flares, significantly outperforming C- and M-class predictions. Despite the scarcity of X-class samples, the model achieves high TSS and HSS scores while maintaining a low FAR, surpassing several existing methods. These results indicate that the proposed multimodal fusion architecture possesses a distinct advantage in capturing the discriminative features of strong solar flare eruptions.
In conclusion, while the proposed model shows robust performance on X-class and C-class flares prediction, reducing the false-positive rate for M-class flares remains an important task. In future work, we aim to advance beyond the current framework by designing a spatiotemporal prediction model that integrates time-series physical evolution with magnetogram evolution. Specifically, we plan to incorporate physics-informed semantic guidance to better direct the magnetogram learning process. Additionally, we will focus on optimizing the feature extraction mechanism and introducing strategies such as class-aware loss functions or dynamic adjustments. These enhancements are expected to resolve the ambiguity in M-class events and further elevate the model’s overall discriminative performance.
Acknowledgments
We sincerely thank the reviewers for their thorough evaluation of our manuscript and for their valuable comments and suggestions. The data used in this study are provided by NASA/SDO, the HMI science team, and the GOES team. We gratefully acknowledge the support of all members of the Joint Science Operations Center (JSOC) at Stanford University and thank them for granting us access to the relevant data. We also express our sincere appreciation to the SDO/HMI team members for their dedicated efforts and significant contributions to the SDO mission. This work is supported by the National Key R&D Program of China (2021YFA1600500, 2025YFF0510700, 2025YFE0202500, 2022YFF0503800), the National Natural Science Foundation of China (42274201), and the Specialized Research Fund for State Key Laboratory of Solar Activity and Space Weather.
Author Contributions
L.Z. proposed the methodology, performed the experiments, and was responsible for writing and submitting the manuscript. Y.Y provided overall supervision and critical input on complex aspects of the research. All other authors contributed to the revision through constructive feedback and helped improve the manuscript.
Footnotes
- 4
- 5
- 6
- 7





















