R Willink 2007 Metrologia 44 73 doi:10.1088/0026-1394/44/1/011
R Willink
Show affiliationsTwo equations relating to the effect of finite resolution on the uncertainty of a sample mean give the impression of being contradictory. One suggests that knowing the data to be digitized increases uncertainty while the other might be thought to imply the opposite. This issue is clarified and relevant concepts are discussed. This discussion leads to the development of a new approximation relating to the uncertainty of an estimate obtained by taking the mean of a set of digital measurement results. This result is consistent with others published recently in implying that resolution error can usually be ignored in such a situation. The solution to this problem obtained in a Bayesian analysis with the usual non-informative prior distribution is shown to give an inappropriate result when all the observations are equal and to exhibit anomalous behaviour in certain other circumstances. One of the appendices describes an estimator that has smaller mean-square error than the sample mean when averaging over the set of measurement problems of this type.
06.20.Dk Measurement and error theory
02.50.-r Probability theory, stochastic processes, and statistics
Issue 1 (February 2007)
Received 3 November 2006
Published 16 January 2007
R Willink 2007 Metrologia 44 73
Wu Bi-ru et al 1992 J. Phys. B: At. Mol. Opt. Phys. 25 355
P D Fainstein and L Gulyás 2005 J. Phys. B: At. Mol. Opt. Phys. 38 317
Eric Sheldon 2003 Eur. J. Phys. 24 91
A. N. Semenov and A. V. Subbotin 1992 Europhys. Lett. 19 161
C Quesne 2008 J. Phys. A: Math. Theor. 41 392001
2002 Nanotechnology 13
K Kirsten 1990 Class. Quantum Grav. 7 2169
J F Sutcliffe et al 1991 Phys. Med. Biol. 36 87
K Hirano et al 2007 Phys. Med. Biol. 52 2545