Evaluating skill of BMKG wave model forecast (Wavewatch-3) with observation data in Indian Ocean (5 – 31 December 2017).

Providing Maritime meteorological forecasts (including ocean wave information) is one of BMKG duties. Currently, BMKG employs Wavewatch-3 (WW3) model to forecast ocean waves in Indonesia. Evaluating the wave forecasts is very important to improve the forecasts skill. This paper presents the evaluation of 7-days ahead BMKG’s wave forecast. The evaluation was performed by comparing wave data observation and BMKG wave forecast. The observation data were obtained from RV Mirai 1708 cruise on December 5th to 31st 2017 at the Indian Ocean around 04°14'S and 101°31'E. Some statistical properties and Relative Operating Characteristics (ROC) curve were utilized to assess the model performance. The evaluation processes were carried out on model’s parameters: Significant Wave Height (Hs) and Wind surface for each 7-days forecast started from 00 UTC. The comparation results show that, in average, WW3 forecasts are over-estimate the wave height than that of the observation. The forecast skills determined from the correlation and ROC curves are good for the first- and second-day forecast, while the third until seventh day decrease to fair. This phenomenon is suspected to be caused by the wind data characteristics provided by the Global Forecasts System (GFS) as the input of the model. Nevertheless, although statistical correlation is good for up to 2 days forecast, the average value of Root Mean Square Error (RMSE), absolute bias, and relative error are high. In general, this verifies the overestimate results of the model output and should be taken into consideration to improve BMKG’s wave model performance and forecast accuracy.


Introduction
Marine meteorological information services are important and must be available continuously for public information, especially for ocean wave information [ [11]. Ocean wave forecast is very vital in supporting many activities in maritime sector for example, safety of voyage, coastal development, etc[12] [13] [14][15] [16] [17] [18] [19][20] [21]. The Indonesia Meteorology Climatology and Geophysics Agency (BMKG) is an institution which is responsible for providing maritime forecast information to the public; including ocean wave forecast over Indonesian waters for 7 days ahead. BMKG utilizes WW3 model to provide the ocean wave forecast because the model is able to provide more complete and continuous data. The data serve as complement to observation and measurement records. The other problem is the observation and measurement data in the ocean are very limited and, 2 rarely continuous, based on WMO-No.702, the ocean wave information can be provided from three types of wave data that are, observed, measured and hindcast [22]. BMKG used WW3 model since 2014, the model parametrization, configuration and validation based on Ramdhani 2015 [23], this model is run using 10-meter level height from GFS (Global Forecasting System), NCEP, NOAA as an input model. Some wave forecast users informed that the results of the wave forecast were higher than reality, this condition will be an important concern to conduct a validation study on the BMKG's wave forecast results, so the aim of this research is to evaluate BMKG's wave model (WW3) performance for 7 days forecast, and to analyzes the forecast skill for recommendation and improvement the WW3 model.

Data
The significant wave height (Hs) and wind surface data observation time series obtained from RV Mirai 1708 cruise, located in the Indian ocean arround 04°14'S, 101°31'E (figure 1), and the measurement period is carried out from 5 to 31 December, 2017. Significant wave height and wind surface dataset was produced from surface meteorological measurement systems (SMET/MIRAI Surface Meteorological observation system).
Wavewatch3 model output obtained from BMKG Ocean Forecast System [24], the data point position collected at 04°14'S, 101°31'E (following the position of observation data), the parameters assessed covers are significant wave height (Hs) and Wind surface for 7 days forecast, start from 00 UTC in each day, the spatial resolution is 0.0625 deg (7km), and the spatial temporal is 3 hours, the time series data period from 5 to 31, December 2017, and provides 7 days forecast information in each date.

Methods
The correlation coefficient (r) is a value that indicates the strength of the linier relationship between variables. The coefficient values from -1 to 1, and the interpretations of the values are -1 (Perfect negative correlation), 0 (No correlation), 1 (Perfect positive correlation), The closer that the absolute value of r is to one, the better that the data are described by a linear equation [25].
where: r = the correlation coefficient relationship between the variables x and y x -the values of the x-variable y -the values of the y-variable To identify skill performance from the Wavwacth3 model forecast against observation data used the ROC (Relative Operating Characteristic) curve, ROC curves is highly flexible method for representing the quality of the forecasts to discriminate observations in each category [26], [27].
ROC is a means of testing the skill of categorical forecast, the ROC curve is the visualization methods used for classifying quality. ROC is a signal detection curve for binary forecasts obtained by plotting a graph of the Hit rate (y-axis) versus the False alarm rate (x-axis) over a range of different thresholds (figure 2). It shows the dependency between the HR (Hit Rate) and the FAR (False Alarm Rate). Hit Rate (HR) = hits / (hits + misses), False Alarm Rate (FAR) = false alarm / (correct negative + false alarm) (table 1). The ROC curve represents a skill forecast of a system wherein the hit rate and the false alarm rate are compared. A rough guide for classifying the accuracy of a diagnostic test is given by the traditional   Threshold's categories were based on Douglas scale. The categorical method used for analyzing which wave categories were successfully predicted. Douglas Sea Scale is a scale which measures the height of the sea waves. The scale is very simple to follow and is expressed in one of 10 levels [28].  3). The points of this study shows that the WW3 model is accurate enough having correlation more than 0.7. However, it should be considered that the results of this evaluation use at the same time series data between observation and WW3 model (not a forecast data). On this study compared the WW3 model output for seventh days forecast with the observation data ( figure 4), the pattern of Significant Wave Height (Hs) chart line shows that the WW3 model provide provided a good forecast until 2 days forecast, even for the extreme conditions (the wave height more than 2 meters), and the forecast accuracy begun to deteriorate from 3 days up to 7 days forecast.  The scatter plot indicates the relationship of Significant Wave Height (Hs) between WW3 and observation data (figure 5). The figure of 8 scatters plots represent the correlation between WW3 analysis and WW3 forecast for day-1 until day-7 respectively are given above. It is shown that the first day WW3 forecast has the best correlation with a value is 0,79, followed by WW3 analysis with a correlation value of 0,72, with a better absolute bias average value of 0,20 m and RMSE value 0,24 compared to day-1 forecast value. In contrast, the absolute bias average and RMSE value showed relatively high value namely 0.48 for abs bias average and 0.52 for RMSE value, on day-1 WW3 forecast. In general, this verifies that even though the WW3 model can capture the observation pattern but it's still overestimated the observation value. The correlation is decreasing from day-2 forecast until 7-day forecast with correlation value 0,63 in day-2 to 0,59-0,57 in day-3 to day-7 forecast, with absolute bias average range between 0,42 m -0,49 m and RMSE range between 0,51 -0,47, this is should be taken into consideration for wave model forecast improvement. The graph of WW3 temporal bias and observational data (figure 6) shows that the WW3 analysis has the smallest bias value compared to the WW3 model of day-1 until day-7. This chart line shows that the WW3 forecast from day-1 to day-7 came out with an overestimated value compared to observations. The bias from the WW3 forecast day 1 shows relatively better results than the WW3 forecast day-2 to day-7. The pattern of this temporal bias also shows that the model's behavior has decreased in performance further away from the initial time. The ROC curve represents a skill forecast of a system wherein the hit rate and the false alarm rate are compared. We can analize the ROC curve with 2 methods, first we can see the curve line. When the curve line getting away from the base line, the skill forecast is good. The second method is by analyze the area under curve (AUC), wider AUC shows the better skill of forecast, but both of them are have similar value. The result of ROC between WW3 versus observation data (figure 7), shows that at the first day forecast has the best skill forecast whose AUC value is 0.87. The AUC in second day is decrease, but it is not significant, the AUC value is 0,86. Both of skill forecast in first and second day still in good level. The skill forcast is decreasing significantly starting in day-3 until day-7 forcast, presented by AUC's value decreased from 0.86 in second day to 0.78-0.77 in day-3 until day-7 day forecast. The ROC curve determine that operated wave model in BMKG have a good skill forecast in two days, after that, it is decreasing significantly representing the skill forecast getting worst.  The line graph shows the UV wind correlation of WW3 model with observation data ( figure 9). The start of the comparison WW3 analysis with observation data obtained a good correlation, it is 0.77, and V-Wind is 0.82, and following the correlation has decreased from begun the first day forecast until the day-7 forecast.

Conclusion
The comparison results between WW3 and observation data show that, in average, BMKG's wave model forecasts (WW3) are over-estimate in significant wave height. The comparison of both on time series data shows that the forecast by the WW3 model until two days provided good skill forecast even for the extreme conditions, after that the forecast begun to deteriorate, the correlation value shows that after two days forecast decreased significantly. For the forecast skills determined from the ROC curves are good at the first-and second-day, while the third until seventh day decreased to fair.
BMKG's wave forecasts provides good results for the first-and second-day forecast, this phenomenon is suspected to be caused by the wind surface data characteristics provided by the Global Forecasts System (GFS) as the input of the model. Nevertheless, although statistical correlation is good for up to 2 days forecast, the average value of Root Mean Square Error (RMSE), absolute bias, and relative error are high, therefore, these results could be taken into consideration to improve BMKG's wave model performance and forecast accuracy.
For suggestion, prior to performing WW3 model enhancement, it is advisable to carry out preliminary research to some materials as follows; (1) the forecast accuracy can be maintained or improved by updating the forecast every day or more frequent using the improved data input, (2) verifies and analysis of Global Forecast Data (GFS) wind output data as the input utilized by WW3, (3) attempting to use the other wind data products as the input of WW3, e.g., wind data of WRF output.