Implementation of Chaotic Gaussian Particle Swarm Optimization for Optimize Learning-to-Rank Software Defect Prediction Model Construction

Finding the existence of software defect as early as possible is the purpose of research about software defect prediction. Software defect prediction activity is required to not only state the existence of defects, but also to be able to give a list of priorities which modules require a more intensive test. Therefore, the allocation of test resources can be managed efficiently. Learning to rank is one of the approach that can provide defect module ranking data for the purposes of software testing. In this study, we propose a meta-heuristic chaotic Gaussian particle swarm optimization to improve the accuracy of learning to rank software defect prediction approach. We have used 11 public benchmark data sets as experimental data. Our overall results has demonstrated that the prediction models construct using Chaotic Gaussian Particle Swarm Optimization gets better accuracy on 5 data sets, ties in 5 data sets and gets worse in 1 data sets. Thus, we conclude that the application of Chaotic Gaussian Particle Swarm Optimization in Learning-to-Rank approach can improve the accuracy of the defect module ranking in data sets that have high-dimensional features.


Introduction
Software defect prediction helps us in identifying defect prone software modules. This process is really helpful for software testing activity [1]- [3]. There are 3 different techniques on software defect prediction, that are classification purposes, defect number prediction and construct the defect modules ranking.
Defect ranking prediction and defect classification are the most used technique in the software defect domain. Defect modules ranking is important for the software testing phase, as they can be used to ensure that test resources are allocated efficiently [4]. There are two approaches to construct a ranking defect prediction. The first one is predicting the module defect number, then order it and the second one is Learning-to-Rank approach that directly optimizes the ranking performance [5].
Learning-to-Rank approach uses simple linear model as a prediction model with software metrics [6] as a linear model variable and α as the corresponding parameter for software metrics. The value for 2 1234567890 ''"" α is obtained by training the software module's data with known defect number. Learning-to-Rank approach utilizes Composite Differential Evolution (CoDE) to find the optimal value for α [5]. If α have already been trained, the model is ready to use for predicting the defect modules ranking. Because α is a corresponding parameter for software metrics in learning-to-Rank linear model, the number of α increases as the number of usages of software metrics in the development of predictive models. CoDE is a meta-heuristic proposed by Wang et al [7], in their research CoDE was evaluated using 25 test function and each function run at 30 dimensional search space. The software defect prediction data sets used nowadays has more than 190 software metrics [3], [8].
Chaotic Gaussian Particle Swarm Optimization is meta-heuristic proposed by Jia et al [9] has good test results in solution search space with dimensions of 100 to 500. We propose Chaotic Gaussian Particle Swarm Optimization as an α determinant for the Learning-to-Rank approach by considering that improving the performance of prediction model construction can increase the prediction accuracy.
The remaining of this paper is organized as follows. In Section 2, we mention research that relate to our research. Section 3 describes the implementation of Chaotic Gaussian Particle Swarm Optimization in the Learning to Rank approach. We report and discuss our empirical evaluation in Section 4. We conclude our research in the last section.

Meta-heuristic in software defect prediction
Can et al [10] constructed software defect prediction model using SVM and utilize Particle Swarm Optimization for determine some parameter value in SVM. Their research showed better accuracy for classification purpose. Coelho et al [11] applied Particle Swarm Optimization for both ensemble constructor and consensus function for ensemble clustering technique. This technique showed that it could predict defects as good as or better than the other algorithms. Wahono and Suryana [12] applied Particle Swarm Optimization and Bagging for noise feature and data imbalance problem. Their result showed that the classification accuracy had significantly improvement in almost classification method after they implement PSO and Bagging for noise attribute and data imbalance problem.

2.2.
Ranking approach software defect prediction Li et al [13] arranged top 4 subsystem that suspected to have most defect ratio for testing priority purposes. Zimmermann et al [8] used linear regression to predict defect number for further analysis correlation post and pre-release defect modules ranking. They used a combination of complexity metrics in constructing prediction models. The result showed that the complexity metrics were good indicators for software defect prediction. Li et al [14] classified the data to 3 classifications, defect prone modules, non-defect prone modules and deferment modules. They performed ranking method for each class separately. The result showed that the proposed method gave a good ranking prediction. Yang et al [5] showed to us that directly optimize the ranking performance can give good defect modules ranking. They used simple linear model with meta-heuristic algorithm to optimize the α for the model.

Chaotic Gaussian Particle Swarm Optimization
Particle Swarm Optimization was proposed by Kennedy and Eberhart in 1995 [15]. PSO had simple and efficient characteristics when looking for solutions in the search space. However, Particle Swarm Optimization often experienced premature convergence when searching for solutions in high dimensional search space. Therefore Jia et al [9] performed a hybrid algorithm between Particle Swarm Optimization, Chaotic and Gaussian Local Search.

Methodology and implementation
In this section, we discuss the implementation of Learning-to-Rank approach with Chaotic Gaussian Particle Swarm Optimization. In this research we use Chaotic Gaussian Particle Swarm Optimization based on research conduct in [9].

Data sets
In this research, we have used 2 public benchmark data sets. The first data sets is 3 version of ECLIPSE data sets [8] and the second data sets is 5 open-source project AEEEM data sets [3]. The characteristics of these data sets are shown in table 1.
The first data sets is public benchmark data that was submitted by D'Ambros et al consisted of 5 software projects Eclipse JDT Core, Eclipse PDE UI, Equinox Framework, Mylyn and Apache Lucene. In each software project D'Ambros extracted 6 categories of software metrics: process metrics, previous defect metrics, source code metrics, entropy of change metrics, metric source code and entropy (metric source code). This data can be used for defect prediction at the software class level.
The second data sets is ECLIPSE software project that was proposed by Zimmermann et al consisted of various source code metrics and divided into 3 versions of release i.e. version 2.0, 2.1 and 3.0. This data can be used to predict defects-prone at the file and software package level.

Chaotic Gaussian Particle Swarm Optimization Implementation for Learning-to-Rank Approach
Learning-to-Rank [5] approach calculates relative defect number of a module that denote as f(x) use the following equation: x_i is the i-th metric from software metrics vector that represent module x = (x1,x2, . . . . . , xd), when d is total software metrics and represent for the dimension of solution search space on Chaotic Gaussian PSO that is utilized to determine the value of vector α. For Chaotic Gaussian PSO fitness function, we have use Fault Percentile Average (FPA) [16] as suggested in [5]. FPA is also used for evaluating ranking prediction results. The Fault Percentile is defined as below where n_i is actual defect number in module i, k is k-th module from top i actual defect modules ranking and m is top m relative defect modules ranking. Vector α is translated into the position of each particle or usually called solution candidate in the swarm in order to determine the optimal value of vector α. Then Chaotic Gaussian PSO [9] begins the optimizing process for vector α using the parameter setting from table 2 and FPA as the fitness function.

Implementation scenario
Our experiment scenario for Learning-to-Rank software defect prediction approach is based on [5].
The implementation of Learning-to-Rank software defect prediction is divided into 2 scenarios. This division is based on different treatments on the data that will be used for construct prediction model and data used to predict the ranking of modules. Five data sets are single-release software project data (AEEEM data sets) and 6 other data sets are multiple-release software project data (Eclipse V2.0, 2.1, 3.0 in file and package level). The detail scenario is shown in figure 1 and 2.   The first step taken on the implementation of Learning-to-Rank is to normalize the data sets. It then transforms all values in each column to the range zero to one. After normalizing the data, we divide each AEEEM project data set based on 75% -25% ratio. 75% of the data is used to train prediction models, the remaining 25% is used to evaluate the prediction model as test data. We then perform the splitting, because the 5 data sets from the AEEEM data set had only 1 version of each data set.
The next step is to train prediction models using Chaotic Gaussian PSO. With the vector α that is produced by Chaotic Gaussian PSO, model is ready to be used to predict modules rank.We use the prediction model that constructs from former release data sets to predict next release data sets such as a prediction model construct from Eclipse_File2.0 data sets will predict the defect modules ranking for Eclipse_File2.1 and Eclipse3.0 data sets. For single release data sets we predict the defect modules ranking of data test with the prediction model construct by corresponding training data sets.

CoDE (predict) Ideal
Eclipse_File2.0-2.  Eclipse_File2.0-2.1 in table 3 means the prediction model is trained using Eclipse_File2.0 data sets and then the prediction model is used to predict defect modules ranking for Eclipse_File2.1 data sets and so on. We have trained 10 prediction models for each data set. Each value in table column represents the average of 10 FPA value from training process or average of 10 predicted result along with its deviation standard, while the number in the bracket is p-value of nonparametric test results of Wilcoxon rank-sum. Table 3 shows that the prediction model trained using Chaotic Gaussian Particle Swarm Optimization has significantly better accuracy on 3 data sets and has an equivalent accuracy in 3 other data sets. This proves that Chaotic Gaussian Particle Swarm Optimization has a better ability than CoDE to find optimal solutions in data sets with large dimensions. It is confirmed by CGPSO FPA train value that is statistically better than CoDE FPA train value in all data sets. Table 4 represents the experimental results of Learning-to-Rank software defect prediction implementation in single-release data sets. All prediction models in table 4 have been trained with 75% of the total data sets, and then the trained models have been used to predict the defect module ranking for 25% of unused data on the model training process.
In Table 4, Chaotic Gaussian Particle Swarm Optimization has significantly better accuracy in 2 data sets, has an equivalent accuracy on 2 data sets and has worse accuracy results in 1 data sets. The result is approximately the same as the result in table 3. However, in Table 4 Chaotic Gaussian PSO has worse accuracy than CoDE on eclipse data sets. However, we still have not been able to find the exact reason CoDE has a higher accuracy on predicted defect modules ranking on eclipse data sets. We suspect that the model that we have trained with eclipse data sets using Chaotic Gaussian Optimization is overfitting.

Conclusion
In this research, we have presented and evaluated the implementation of Chaotic Gaussian Particle Swarm Optimization on the Learning-to-Rank software defect prediction approach for train model parameter. The results show the prediction model construct using Chaotic Gaussian Particle Swarm Optimization has better FPA train value on all data sets used in the model training process. An improved prediction accuracy is also seen in some data sets. Although one of the prediction results on Eclipse data sets produces worse accuracy than the prediction results produces by CoDE.