Bayesian Computational Methods of the Logistic Regression Model

In this paper, we will discuss the performance of Bayesian computational approaches for estimating the parameters of a Logistic Regression model. Markov Chain Monte Carlo (MCMC) algorithms was the base estimation procedure. We present two algorithms: Random Walk Metropolis (RWM) and Hamiltonian Monte Carlo (HMC). We also applied these approaches to a real data set.


INTRODUCTION
MCMC methods are set of algorithms used in efficient counting, optimization, dynamic simulation, integral approximation, and sampling. These techniques are commonly used in problems relating to statistics, combinatorial, physics, Chemistry, probability, optimization, numerical analysis. Because of its ease of implementation, in other cases fast convergences as well as numerical stability applied mathematicians and statisticians prefer MCMC methods. But, due to its complex theoretical basis, and shortage of theoretic convergence diagnostic. MCMC methods are sometimes referred to as black boxes for sampling and posterior estimation [1].
The application of Bayesian methods in applied problems expanded during in 1990s, the basic idea of Markov chain estimation is generating approximate samples from posterior distribution of interest by generating Markov chain whose stationary distribution of which is the desired posterior. Revolutionary approach for Monte Carlo was started in the particle Physics studies by Metropolis (1953), then Hastings (1970) generalized it via more statistical setting. [2] Gibbs sampling and Metropolis-Hastings algorithms are classical approaches to implement MCMC algorithms. A special case of Metropolis-Hastings sampling is Gibbs sampling in which all the candidate value is often accepted. Gibbs sampling can only be used if the posterior distribution has conditional distribution that is standard distribution such as Dirichlet, Gaussian, or discrete distribution. Whereas the Metropolis-Hastings (MH) algorithm is also applied to a wide range of distributions and based on the candidate values being proposed sampled of the proposal distribution. These are either rejected or accepted according to the probability rule [3].
Another common algorithm is HMC that introduces momentum variable and employs leapfrog discretization of deterministic Hamiltonian dynamics as the proposal scheme in combination with momentum resampling [4] Difference of MCMC approaches are designed by simulation irreversible Markov chains that converge to the target distribution. One group of algorithms includes s irreversible MALA [5]and non-  [6], another group of algorithms is included the bouncy particle [7]and Zig-Zag samplers using Poisson jump processes [8].
We are focusing in this study on estimating parameter for the logistic regression model where the dependent variable be binary data, also we discussed RWM and HMC methods to determine the parameter estimation.

LOGISTIC REGRESSION MODEL
The name of logistic regression exists from that the function 1 ⁄ is named logistic function. Logistic regression model is the special case of statistical models named generalized linear models which also include linear regression, log-linear models, Poisson regression, etc.
Logistic regression model is very commonly used in the analyzing data including binary or binomial responses as well as several explanatory variables. This model offers a strong technique analogous to ANOVA of continuous responses and multiple regression [9].
Suppose that 1,2, … , independent observations on a binary response variable y, then for k-dimensional covariate X, the model is defined as: The value of be a number between 0 and 1, also (1) equivalent: ∑ where

BAYESIAN LOGISTIC REGRESSION
Suppose we have a normal prior distribution for all the parameters, which is often used as a prior distribution for logistic regression. ~ , 1,2, … , The posterior distribution is: It seems, this expression has no closed form, and the marginal distribution of each coefficient by integration is difficult to obtain. For logistic regression, the exact numerical solution is hard to obtain. In the statistics software, the most common method used for estimating parameters is the MCMC simulation, which provides an approximate solution.

Bayesian Computational Methods
In the following we will presented the computation Bayesian algorithms in the process of obtaining Bayesian parameter estimation in Logistic regression model:

Metropolis-Hastings Algorithm
Its important algorithm can provide a general approach to generating correlated sequence drawing from the target density which may be difficult to sample using the classical independent methods.
The basic Metropolis-Hastings algorithm can be given in the following: Generate the candidate state at step t from the proposal distribution . | , for the next state in the chain the candidate state is accepted or rejected with the probabilities given by , This means the candidate which has a higher value of target distribution than target distribution of the current value always acceptable. In contrast, the candidate with lower target distribution value will only be accepted with a probability equal to the ratio of the target distribution value to the current distribution value. However, a chain with random-walk proposal distribution will generally have several accepted candidate points, but that most moves are a short distance, that is the accepted candidate point will also be close to the previous current value . So moving around the whole state space it could take a long time. [10] Hamiltonian Monte Carlo algorithm HMC or (Hybrid Monte Carlo) algorithm is a MCMC technique, which uses a combination of Metropolis Monte Carlo approach and advantages of Hamiltonian dynamics [6], [11]for sampling of complex distribution.
In view of the observed data , , … , , we are interested in sampling from the posterior distribution of the model parameters .
, ∝ exp Where function of potential energy, defined by: ∑ | Negative log-likelihood in the first term, is assumed prior on the parameters of model and it is almost always difficult to solve the posterior distribution.
HMC expose Hamiltonian dynamics system with the auxiliary momentum variables to propose samples of in the Metropolis framework which explores parameter space more efficiently than proposals of the standard random walk.
HMC generates the proposals jointly for ( , ), by using the system of differential equations as follows: , where is quadratic kinetic energy function which corresponds to negative log-density of the zero-mean multivariate Gaussian distribution with covariance ,where is called the mass matrix, which is often set to the identity matrix, but it can be used with Fisher information to precondition the sampler [12]  : heart disease (value 0: no ; value 1: yes) Package statistical software R is used for Bayesian analysis that provide a convenient environment for the simulation of MCMC, we simulate the posterior density of the logistic regression model by using RWM and HMC algorithms .First we derived the β posterior simulation of RWM algorithm . The summary of the sample is given as in the following: Table1 show the posterior summaries which consist of variables, mean, standard deviation and quantiles of posterior distribution, where significant variables could be determined at the 5% significance level. Values from 2.5% to 97.5% quantiles provide 95% credibility interval for every given   Table 2.  Table 2 shows that variables thalach , oldpeak , cp , exang , ca, thal and slop have |z| 2. Thus, these variables have not converged. All other variables have converged, according to Geweke diagnostic.

RESULTS AND DISCUSSION
Then, we simulated the posterior density of the logistic regression model by using HMC algorithm. Five of independent variables are continuous, with wide range of the values. When tuning this model, the step size of these types of variables is tuned separately for the specific application of the HMC. We summarize the results and plot the trace and density distribution of the simulated posteriors in the following table 3 and Figure 3: