Uncertainty of Coupled Model Intercomparison Projects 6 (CMIP6) in Indonesia’s maritime continental region for the historical period

Climate Model is a tool for studying climate and climate change, In climate models there is uncertainty that occurs due to the inability of the model to simulate climate in an area. This study aims to find the best CMIP6 model in the IMC and look at the uncertainty of the model. The data used in this study are precipitation data on the CMIP6 model, as well as the use of CRU data as reference data and observation data. Uncertainty is calculated for historical periods using CDF to see random values from the model. Validation of historical CRU and CMIP6 data is performed by performing statistical calculations that include Pearson correlation, standard deviation, RMSE, KGE, TSS, and relative bias. CRU model testing in Indonesia shows quite good performance at all points, the CRU model has a pattern that follows observational data based on correlation values. the uncertainty of CMIP6 is large in high precipitation and low in moderate precipitation, and there is moderate uncertainty in low precipitation, The analysis of historical data indicates that the EC-Earth3-Veg-LR model has greater performance in simulating precipitation within the IMC region, closely aligning with the values and patterns observed in the CRU model. Furthermore, the EC-Earth3-Veg-LR model displays a strong correlation, as well as good values for the KGE and relative Bias, measuring at 0.33, 0.175, and 1.101% respectively.


Introduction
Climate change is a global-scale disaster that occurs over a long period and affects various aspects of human life [1], [2].The climate is governed by a complex climate system, where in alterations to this system have globally consequences on the climate.Therefore, climate change occurs as a result of the changes to the climate system [3].
The Coupled Model Intercomparison Projects (CMIP) were created to help with the study and comparison of climate simulations created by combining global circulation models of the atmosphere, oceans, cryosphere, and land.The aim of CMIP is to improve our understanding of past, present, and future climate changes resulting from both natural climate variability and external radiative forcing.Such is achieved through the application of a collection of simulations conducted by several climate models [4], [5].CMIP6 is the latest CMIP model variant today, in this model poor quantification and understanding of radiative forcing have been a problem for a long time in CMIP and this will be a better approach in CMIP6 [6].The Indonesian Maritime Continent (IMC) is a region consisting of several countries, IMC has a ratio between land and sea of about 2.7: 7.3.This value is almost equal to the global ratio of land and sea of 3: 7. IMC is an archipelago of warm waters that is assumed to act as the primary heat source for driving atmospheric circulation by inducing extremely severe convective activity [7].The IMC region consists of various variables, specifically the dynamics of the Intertropical Convergence Zone (ITCZ), Indonesian Throughflow (ITF), El Niño-Southern Oscillation (ENSO), Indian Ocean Dipole (IOD), Monsoon, Cold Surge, Madden-Julian Oscillation (MJO), and diurnal variability [8].
Climate models contain uncertainties that originate from different factors, including the model's limited ability to accurately capture complex climate systems, Additionally, uncertainty can arise due to computational limitations within the model [9], [10].The objective of this study is to examine the degree of uncertainty present in the CMIP6 model in the IMC region and also see the model with the best performance in simulating precipitation in the IMC region during the historical period.

Study Area
IMC is an area with coverage of several countries such as Indonesia, Papua New Guinea, East Timor, the Philippines, parts of Australia, and others.The IMC region is predominantly occupied by the ocean [11].there are many broad definitions of IMC, in this study we take the area of study from latitude 20ºS -20º N, and longitude 90ºE -160ºE, the selection of this study area is based on the definition of maritime continent by the Australian Bureau of Meteorology as well as the consideration of the grid size of the model.Figure 1 shows the IMC area which is composed of several countries, in this study the area used is only land.

Figure 1. Study area of the Indonesian maritime continent
Observation data collection was carried out for 5 meteorological stations located on five major islands of Indonesia, Table 1 shows the name of the station and the province where the station is located.Observation coverage is not comprehensive making CRU TS V4.03 data as a substitute for observation data, CRU data has a monthly temporal and spatial resolution of 0.5° × 0.5° with a period from 1901 -2014.CRU data retrieved from https://crudata.uea.ac.uk/cru/data/hrg/CRU is built based on measurement data interpolated with the angular-distance weighting method, the amount of measurement data used in the CRU affects the output [15].there is a difference in spatial resolution between CRU and CMIP6 therefore interpolation is carried out for CMIP6 data using the nearest neighbour method, and interpolation is carried out following CRU resolution 0.5° × 0.5° to obtain consistent results.Observation data taken for ten years from 2001 -2010, this data is an observation of the Indonesian Agency for Meteorological, Climatological, and Geophysics (BMKG) which is obtained from https://dataonline.bmkg.go.id/, the selection of this data period is adjusted to the availability of data on the BMKG website.

Validation Methode
Statistical calculation can provide an assessment of the degree to which the model accurately replicates the climatic conditions in the study region.The statistical techniques used in this study contain Pearson correlation, standard deviation, root mean square error, Kling-Gupta efficiency, Taylor skill score, and relative bias.The selection of this validation method relies upon a comprehensive analysis of many studies that have been undertaken [9], [14], [16]- [18].

Pearson correlation.
The correlation is used to demonstrate the linear association between two variables, representing the relationship between the sample and the population.The correlation value measures the strength and direction of the relationship between variables.A positive correlation indicates that if one variable grows, the other variable tends to increase as well.Negative correlation refers to the presence of an inverse relationship between two sets of data.[19], [20], Pearson Correlation Defined as.
where x is the test data and y is the reference/observation data.The range of correlation values is -1 to 1 where in that range there is a division of correlation strength, Table 3 shows the range of correlation strength.Standard deviation indicates the amount of deviation from the data to its mean, a large standard deviation value tells that the distribution of data is far from the mean while a small standard deviation is the opposite [22].Standard deviation is defined by.
In the equation above s is the standard deviation, x is the data tested and n is the amount of data.

Root Mean Square Error (RMSE).
Root Mean Square Error (RMSE) is a statistical metric utilised to quantify the errors inherent in a given dataset.By employing RMSE, the resulting output provides an assessment of the model's performance [23].RMSE is defined as.
where y is the value of the model and n is the amount of data

Kling-Gupta Efficiency (KGE).
The KGE is a statistical measure that integrates the Pearson correlation, coefficient of variation, and bias.The objective of the KGE measure is to identify commonalities between the two datasets.KGE values are bounded between negative infinity and 1, with a value of 1 indicating optimal performance.KGE defined as.
Where r is the Pearson correlation, γ is the fraction of the coefficient of variance, and β is the normalized bias with standard deviation [16], [24].

Relative Bias.
Relative bias is related to the deviation between the model data and observation/reference, the value of positive relative bias indicates that the model data is worth more than observation/reference (overestimate) and relative negative bias has the meaning of the model is lower than observation/reference (underestimate) [25], The relative bias defined as. ) × 100% In the relative bias equation, xi is model data and yi is observation/reference data.Relative bias in this study used percentages, this was done to see the percent difference between the two data.
4.1.6.Taylor Skill Score.The Taylor Skill Score is a measurement used to evaluate the spatial pattern performance of a model.It is determined with a range of values from 0 to 1, with a higher value indicating better performance, Taylor Skill Score is defined as.

𝑆 =
(1+) 4   4(+ In this equation, R is the correlation between the model and the reference data, while SDR is the ratio of the spatial standard deviation of the model to the reference data [18], [26], [27].In Taylor, the skill score is also done using a Taylor diagram to see the degree of proximity of reference data to the model [28].

Cumulative Distribution Function
Uncertainty relates to random variables obtained from the model.Uncertainty analysis involves the process of identification, where the Cumulative Distribution Function (CDF), often referred to as the distribution function of F, is defined for the random variable X with respect to its initial value x.
A random variable on a CDF can be a finite value or a value at most calculated in a discrete variable [29].

Validation of CRU
The verification of CRU revealed different results across the five study stations.The outcomes of CRU statistical validation can offer information about the efficacy of CRU models in replicating the climate at the selected study area.The results of CRU model validation calculations at the study station are presented in Table 4.The table reveals a notable correlation between the CRU model across all study stations, indicating a high level of correlation.Specifically, the correlation value for each station above 0.5, with the Sultan Hasanuddin station showing the strongest correlation.In the RMSE value, there is a large error information between the model data and the average linear regression, the CRU model has a large error at Tanjung Priok Station with a value of 125.576 mm/ month and the smallest is at Sultan Mahmud Badaruddin II Station which is valued at 65.867 mm/ month.The KGE value in each CRU model is fairly good, with the best value being Sultan Mahmud Badaruddin II with a value of -0.373, and the lowest KGE value is in the Sentani Station area with a value of -1.425, from the results of this KGE it can be seen that the CRU model for Sultan Mahmud Badaruddin II Station has a good pattern match and accuracy.The standard deviation value shown in the table shows that the deviation of the CRU model data reaches >100 mm/month, except for Sentani Station which is only 87.071 mm/month, The largest standard deviation intersection is at Sultan Hasanuddin Station with a value of 225.314 mm/month.The relative bias results in the table provide information that the CRU model for Tanjung Priok and Sentani Stations is overestimate with the largest percentage in Sentani valued at 44.921%, while Sultan Mahmud Badaruddin II and Beringin Stations are underestimated with a percentage that is not too large (-1.945% and 0.969%).This indicates that the CRU models for Sultan Mahmud Badaruddin II, Beringin, and Sultan Hasanuddin Stations do have not too big a difference with observational data.
The results of the CRU model rainfall test are of good value because the measurement network that is widely spread makes the angular-distance weighting (ADW) interpolation method have good performance [15].The CRU model results for all five stations performed well in simulating precipitation, which is similar to research conducted by [30] for the Philippines region and [31] for three rivers in China (Yellow River, Yangste, and Lancang).

Climatology.
Here we conducted a precipitation analysis based on climatology during the period 1901 -2014 by doing a spatial average.Figure 2 and 3 shows the average results for CRU and all ten CMIP6 models.Nor-ESM2-MM.These models tend to overestimate precipitation, especially in the Borneo, Sulawesi, and Papua regions.On the other hand, the MIROC6 and MPI-ESM-1-2HR models tend to underestimate precipitation.In the region of Asia and Australia, the CMIP6 model generally demonstrates similar values, but tend to be more underestimated.
The EC-Earth3-Veg-LR model looks to have the closest value to CRU when compared to the other nine models, Specifically, for the Indonesian region, the EC-Earth3-Veg-LR model demonstrates a range of 200 -400 mm/month, which is relatively higher.However, the pattern observed in this model closely resembles to CRU model.Moreover, both the Asia and Australia regions in the EC-Earth3-Veg-LR model exhibit notable similarities to the CRU model in terms of both value and pattern.The monsoonal rainfall pattern (region I) characterised by a peak occurring during the months of December, January, and February (DJF), and a trough occurring during the months of June, July, and August (JJA).The equatorial pattern (region II) shows bimodal rainfall distribution throughout the months of March, April, May (MAM) and September, October, November (SON), while experiencing reduced precipitation during DJF and JJA.The local pattern (region III) has one peak rainfall in the month of JJA with low rainfall throughout the month [32].
Figure 5 shows the rainfall pattern formed by the CRU and CMIP6 models, based on this figure it can be seen for region I (Figure 5a) all models are capable of simulating the monsoonal rainfall pattern, However, there are differences in the timing of the minimum rainfall among certain models, such as ACCESS-ESM1-5 which has the lowest rainfall in SON.The equatorial rainfall pattern in region II (Figure 5b) looks well simulated by all models, with 2 rainfall peaks and 2 rainfall troughs, the difference in peak time and equatorial pattern trough can also be seen in some models such as MIROC6.Region III (Figure 5c) is a local rainfall pattern that can only be simulated by MIROC6 and NorESM2-MM, both models have peak rainfall in the month of JJA which is characteristic of local rainfall patterns, the region III rainfall pattern is poorly captured is likely due to the use of a grid that is too large, thereby limiting their ability to capture the local influences within region II.
Statistical calculations for rainfall patterns can be seen in Table 5, The result of the table which the calculation of the CMIP6 against CRU can be known for region I the highest correlation is found in the IPSL-CM6A-LR model and relative bias for the ACCESS-ESM1-  5, it is known that the NorESM2-MM model is the best model for matching the rainfall pattern, having a high correlation value, and having a low relative bias for region I, while the CNRM-CM6-1 model is the best model for region II, and the MIROC6 model is the best model for region III.6 shows the relationship between CMIP6 and CRU data, from these results it can be seen that the correlation value of the Asian region is dominated by moderate to strong correlation (0.5 -1), However, it is seen that the BCC-CSM2-MR, EC-Earth3-Veg-LR, MIROC6, MPI-ESM-2-HR, and NorESM2-MM climate models have a very low correlation value, particularly within the Malaysian region.The Philippines region in all models has a high positive correlation value in the West, while the resulting Australian region tends to be dominated by positive correlation across regions in each model.The correlation value in the Indonesian region shows differences in all models.Figure 6 show that out of the ten models examined, only Java and South Papua display a similar correlation pattern, characterised by a moderate to strong positive correlation.In the BCC-CSM2-MR, MIROC6, MPI-ESM1-2-HR, MRI-ESM2-0, and NorESM2-MM models, the correlation pattern shows a higher value in the southern region of the large Indonesian island.On the other hand, the ACCESS-ESM1-5, EC-Earth3-Veg-LR, and CNRM-CM6-1 models display a mostly positive correlation throughout most regions of Indonesia.
Based on the correlation findings presented in Figure 6, it is clear that the ACCESS-ESM1-5 and EC-Earth3-Veg-LR models demonstrate the strongest correlation within the IMC region.This is seen in the large number of positive correlation values across all regions.Table 6 provides the outcomes of statistical validation calculations conducted on the ten CMIP6 models.The result reveals that the ACCESS_ESM1-5 model has the highest correlation among all models, with a correlation of 0.353.Additionally, the IPSL-CM6A-LR model demonstrates the lowest RMSE at 24.823.The EC-Earth3-Veg-LR model has the highest KGE score, although negative, with a value of -0.175.The CNRM-CM6-1 model achieves a Taylor Skill Score of 0.116.Lastly, the EC-Earth3-Veg-LR and MRI-ESM2-0 models have the best Relative Bias, with values of 1.01%.
The statistical validation of the CMIP6 in the IMC region indicates that the EC-Earth3-Veg-LR model demonstrates best performance.This is known from the model's close rank on the Taylor diagram and the results presented in Table 6, which reveal that EC-Earth3-Veg-LR has the strongest correlation, KGE, and relative bias compared to the remaining ten models.

CMIP6 uncertainty
Uncertainty in the model shows how much random value the model has against the average data of all models and references, Uncertainty results from the CDF which can be seen as the chance of rain events in a certain range based on its distribution.8 show CDF and the variance of the precipitation possibility distribution within the model, from these results it can be seen that the variance is the high value when precipitation is high (Variance >900) and the presence of variance is medium value at low precipitation.Variance in CDF can represent uncertainty [29], therefore it can be said that the uncertainty of ten CMIP6 is high when precipitation is high and is quantile 1.0, while when precipitation levels are low, the resulting uncertainty is of a moderate nature.The value of uncertainty shows a decrease when precipitation levels range between 100 and 200 mm/month.

Figure 4 .
Figure 4. Regional division of Indonesia's rainfall pattern

11 Figure 6 .
Figure 6.Spatial correlation of ten CMIP6 models to CRU data for the period 1901 -2014

Figure 7 .
Figure 7. Taylor diagram of ten CMIP6 models, red stars indicate CRU data.

Figure 8
Figure 8 CDF of CMIP6 and CRU precipitation

Figure
Figure8show CDF and the variance of the precipitation possibility distribution within the model, from these results it can be seen that the variance is the high value when precipitation is high (Variance >900) and the presence of variance is medium value at low precipitation.Variance in CDF can represent uncertainty[29], therefore it can be said that the uncertainty of ten CMIP6 is high when precipitation is high and is quantile 1.0, while when precipitation levels are low, the resulting uncertainty is of a moderate nature.The value of uncertainty shows a decrease when precipitation levels range between 100 and 200 mm/month.

Table 1 .
[14]tion of meteorological stations in the observation data.precipitationoverIMC,CMIP6data was obtained from https://esgfnode.llnl.gov/search/cmip6/.Ten CMIP6 were used in this study, The selection of these models was based on several studies that have been carried out specifically for the Southeast Asian region[9],[13],[14]. table 2 shows information from 10 CMIP6 models used in IMC uncertainty analysis and region validation, The ensemble used by the nine models is r1i1p1f1, and for the tenth model r1i1p1f2 is the ensemble used, the selection of different ensembles in this model due to the availability of ensembles in the database.
[12]ataCMIP6 is the latest CMIP model to date, it contains a set of historical data derived from the evaluation of CMIP5[12].This study used monthly precipitation data to evaluate the performance of the CMIP6 3 model in simulating

Table 2 .
Data of 10 CMIP6 models used in the research

Table 4 .
Statistical validation of CRU model at 5 meteorological stations

Table 5 .
Statistical validation of Indonesia's rainfall pattern

Table 6 .
Statistical validation of CMIP6 model against CRU for the period 1901 -2014