Comparison of entropy measures in generalized maximum entropy estimation

This study aims at comparing the efficiency of different Entropy measures in generalized maximum entropy estimation of the unknown parameters in macroeconomic panel data models by investigating the impacts of foreign direct investment on economic growth of 10 ASEAN countries over the period of 16 years (2001-2016). The three entropy measures (Shannon, Tsallis and Renyi entropy) are used as ordinary least squares estimator and generalized maximum entropy estimator. The results show that Tsallis entropy is the most appropriate to describe the effect of FDI on real GDP in ASEAN countries as it provides the maximum Max Entropy value and the lowest mean square error (MSE). The results of FDI indicates its positive impact on real GDP with the coefficient equal to 0.194 meaning when the FDI increases by 1 percent then the real GDP will increase by 0.194%. FDI is considered as one of the main factors affecting inward real GDP as an increase in production of goods and services leads to a rise in GDP This, in turn, makes the economy more attractive to the foreign investors.


Introduction
Limited data will bring about an underdetermined, or ill-posed problem for the observed data, or for regressions when small data set is applied [1]. [2] mentioned that we could obtained the significant parameter when we have a large sample size of data. [3] suggested that when the sample is limited, it is often difficult to obtain meaningful results. Therefore, the conventional estimations are difficult to reach the optimal solution. Due to the traditional estimation techniques, such as least squares and maximum likelihood, are difficult to find the optimal solution, the Generalized Maximum Entropy (GME) approach is introduced in the light of these problems.
Regarding data problem, this study is well aware that a large sample size is not always obtainable in any fields of economic research since the data collection systems of some countries, particularly the underdeveloped and developing, are poor and bring about the lack of reliable data. These facts can give rise to ill-posed problem when the amount of sample information is insufficient for model estimation; in other words, when the observed data is less than the parameter estimates [4] and [5]. In addition to the limitation of the data, the time series approach to modeling typically involves a set of strong normality assumptions. Therefore, it might not be true that error is assumed to be an i.i.d. normal distribution [6]. Moreover, [7] also confirmed that the time series data has conditionally nonnormal distribution in various empirical studies and the normal distribution assumption can yield biased estimation for parameter estimates. Therefore, in the real data analysis, such data might be illbehaved leading to either ill-posed or ill-conditioned problem; that is the correct distribution of the data is not easy to specify in the underlying economic and financial models [8].
To solve these problems, [9] suggested that it seems reasonable to estimate the unknown parameters using GME approach since it is the most appropriate choice available to the estimation of the model when we want to get away from any parametric assumptions and mitigate the limited data problems. [8] mentioned that the objective of this estimator is to excerpt the available information from the observed data and functional form. This approach then became one of the most effective estimators in econometrics and economic modelling. Previous works have provided evidences of the superiority of entropy estimation over conventional estimations by maximum likelihood (ML) and least squares (LS). [10] conducted a Monte Carlo experiment study of GME estimation under nonnormal assumption. They investigated and compared GME performance with ML estimator. They observed that GME estimation has lower risk than ML in terms of Mean Square Error (MSE), It was found that, when the data was small, the MSE of parameter in the GME is smaller than the ones in ML estimator, indicating that GME tends to be more accurate. Moreover, the recent work of [11] also made comparison between GME approach and LS approach using the dataset on Portland cement and the GME estimator was found to have relatively higher performance as its MSE was lower than that of OLS.
As mentioned earlier, GME estimator can play a vital role in the econometric model estimation. It can be conducted as an alternative estimator to LS and ML. In this study, we consider panel data regression models with strict exogeneity as well as collinearity and endogeneity. To the best of our knowledge, the GME estimator for panel regression has already proposed in the works of [12], [13], and [14]. However, these studies considered only shannon entropy [15] measure as the objective function in the GME approach. Thus, we contribute to the literature by suggesting two more entropy measures, Renyi entropy [16] and Tsallis entropy [17].
The primary task in using the GME approach as an estimator is to choose an appropriate entropy measure which reflects the uncertainty (state of knowledge) that we have about the occurrence of a collection of events. In the literature, empirical applications have so far been restricted to using Shannon's entropy. But as will be shown later, different entropy measure may reflect different uncertainty structure, hence it is not reasonable to apply Shannon's entropy to all datasets. In this study, we replace the Shannon entropy with either the Renyi or Tsallis as generalized entropy measure. By carefully selecting the entropy measure in the GME, we can estimate the Panel regression efficiently and can get a model flexible enough to reflect all possible patterns in the data.
The reminder of this article is organized as follows. In Section 2, we briefly summarize the Shannon, Renyi and Tsallis entropy measures for the Panel regression model. Then we derive the two GME estimation methods. In Section 3, sampling experiments is presented. Section 4 is on economic application of the proposed methods. We summarize our results in Section 5.

Panel data regression models
Panel regression is one of the important tools for estimating the relationships between independent and dependent variables. The model is based on panel data, consisting of both cross-sectional and time series data. The appropriate estimation method for this model depends on the two error components. More formally, the most general formulation of a panel data regression model can be expressed by the following equation: The assumption about this error term is that

Entropy approach
In this study, maximum entropy estimator is introduced to estimate unknown parameters in equation (2.1). This estimation involves inferring the probability distribution that maximizes information entropy given a set of various constraints.
The entropy measures are briefly presented below. Shannon entropy measure Renyi entropy measure Tsallis entropy measure where k p is a proper probability and q is a positive constant and depends on the particular units used.. These two entropy measures are indexed by a single parameter q , which we restrict to be strictly positive: We then generalize the maximum entropy to the inverse problem to the regression framework.
Rather than search for the point estimates  , we can view these unknown parameters as expectations of random variables with M support value for each estimated parameter value () k ,

Simulation study
The simulation and experiment study is conducted for assessing the finite sample performance of the various GME estimators . Our dataset is simulated from We aim to examine the accuracy of the estimator using the bias approach . where r  and r  are the estimated parameter value and true parameter value at iteration rth , respectively .In this experiment, the iteration R =100 is used .
We consider the number of support as M =5, as it was confirmed to be more robust and efficient, see    In the economics perspective, the results show that FDI creates a significant impact on the real GDP .We find a positive relationship between real GDP and FDI in which the coefficient is equal to 0.491 meaning when the FDI increases by 1%, the real GDP will increase by 0.194%. Typically, FDI is considered as one of the main factors affecting inward real GDP as an increase in production of goods and services leads to a rise in GDP This, in turn, makes the economy more attractive to the foreign investors.

Conclusions
In this study, we introduce generalized maximum Tsallis and Renyi entropy as an estimator for Panel regression model. We conduct an experiment simulation study to investigate the accuracy of our estimators and also test whether our robust estimator is resistant to multicollinearity or not .In this simulation study, the conventional Shannon entropy is also applied to compare its performance with the other two estimators .The results show that given Tsallis and Renyi as the entropy measures, the bias of estimated parameters are not much different in terms of the sign and magnitude of the estimates. We show that each of these GME entropy measures can be used as an objective function of estimation procedure in much the same way as [19] used the Shannon entropy measure to formulate the GME estimation method .Then, we apply all estimators to study the effect of Foreign Direct Investment on Real Gross Domestic Product in ASEAN countries .We vary the number of support and examine the performance of each estimator using MSE value .We find that FDI creates a positive significant effect to the real GDP.
However, this application study is just a simple application and the one variable employed is not enough, the further study would also consider other macroeconomic variables in order to obtain a better insight about the contribution of various economic indicators to the GDP.