In this paper, we study sampling from a posterior derived from a neural network. We propose a new probabilistic model consisting of adding noise at every pre- and post-activation in the network, arguing that the resulting posterior can be sampled using an efficient Gibbs sampler. For small models, the Gibbs sampler attains similar performances as the state-of-the-art Markov chain Monte Carlo methods, such as the Hamiltonian Monte Carlo or the Metropolis adjusted Langevin algorithm, both on real and synthetic data. By framing our analysis in the teacher-student setting, we introduce a thermalization criterion that allows us to detect when an algorithm, when run on data with synthetic labels, fails to sample from the posterior. The criterion is based on the fact that in the teacher-student setting we can initialize an algorithm directly at equilibrium.
Bayesian Statistics for Complex Systems
Guest Editors
- Michael A. Lomholt, University of Southern Denmark, Denmark
- Ralf Metzler, University of Potsdam, Germany
- Samudrajit Thapa, Max Planck Institute for the Physics of Complex Systems, Germany
Scope
The physical understanding of complex systems poses major challenges due to factors such as large numbers of system variables, many of which might be interdependent; memory effects and heterogeneity in the system; lack of feasibility to repeat experiments and limited sampling of the data space often in the presence of one or more sources of noise. Bayesian statistics provides a framework for parameter estimation, model comparison and uncertainty quantification which has proven and continues to be useful in overcoming many of these challenges.
This issue will feature theoretical and/or computational studies that focus on the development and/or application of Bayesian methods as a data-driven approach to model complex systems across several fields including soft matter, climate, movement ecology, biology and epidemiology. The issue is intended to provide a platform for exchanges within and between diverse scientific communities which share a common interest in Bayesian statistics.
Submission process
We encourage submissions from all authors whose work fits with the scope of this special issue. This is intended to be a high quality issue and all special issue articles will be subject to the same review process as regular articles. Authors are invited to contact the journal team to discuss the suitability of their work prior to submission.
Please submit your article via our online submission form. You should submit the appropriate article type for your submission then choose 'Bayesian Statistics for Complex Systems' from the drop-down menu.
Deadline for submissions
The target deadline for submissions is 31 October 2024. Articles will be published on acceptance, without being delayed by other papers in the collection.
Participating Journals
Paper
We use statistical mechanics techniques, viz. the replica method, to model the effect of censoring on overfitting in Cox's proportional hazards model, the dominant regression method for time-to-event data. In the overfitting regime, Maximum Likelihood (ML) parameter estimators are known to be biased already for small values of the ratio of the number of covariates over the number of samples. The inclusion of censoring was avoided in previous overfitting analyses for mathematical convenience, but is vital to make any theory applicable to real-world medical data, where censoring is ubiquitous. Upon constructing efficient algorithms for solving the new (and more complex) Replica Symmetric (RS) equations and comparing the solutions with numerical simulation data, we find excellent agreement, even for large censoring rates. We then address the practical problem of using the theory to correct the biased ML estimators without knowledge of the data-generating distribution. This is achieved via a novel numerical algorithm that self-consistently approximates all relevant parameters of the data generating distribution while simultaneously solving the RS equations. We investigate numerically the statistics of the corrected estimators, and show that the proposed new algorithm indeed succeeds in removing the bias of the ML estimators, for both the association parameters and for the cumulative hazard.