A PROBLEM WITH THE CLUSTERING OF RECENT MEASURES OF THE DISTANCE TO THE LARGE MAGELLANIC CLOUD

The distance modulus (μ) of the Large Magellanic Cloud (LMC) has long been a key question for the distance ladder and all extragalactic distances. For the Hubble Space Telescope Key Project (HSTKP) to measure the Hubble constant (H₀), "the uncertainty in the distance to the LMC is one of the largest remaining uncertainties in the overall error budget for the determination of H₀" (Freedman et al. 2001). If the LMC μ is changed by 0.1 mag, then the H₀ value will change by 5%.

Hundreds of papers have been published which present independent measurements of the LMC distance. Recent compilations of these values appear in Westerlund (1997), Cole (1998), Gibson (2000), Freedman et al. (2001), Benedict et al. (2002a), Clementini et al. (2003), Walker (2003), and Alves (2004). These have used many different standard candles (including Cepheids, RR Lyrae stars, SN1987A, eclipsing binaries, the tip of the red giant branch, the red clump, and Mira variables), which have each been calibrated with many independent methods. For example, Cepheids have been calibrated with the Baade–Wesselink method, main sequence fitting in galactic clusters, nonlinear pulsation modeling, Hipparcos parallaxes, and HST parallaxes.

Before the year 2001, the published μ values were spread from 18.1 to 18.8 mag with the stated error bars being much smaller than the spread. This was generally regarded as a symptom that many measures had large unrecognized systematic uncertainties. For example, "It is clear from the wide range of moduli compared to the quoted internal errors in Figure 5 (of the quoted paper) that systematic errors affecting individual methods are still dominating the determinations of LMC distances." (Freedman et al. 2001). Gibson (2000) concluded "it is clear from even a cursory examination of (his) Figure 1 that significant unappreciated systematic uncertainties exist in many published values of μ_LMC."

Zoom In Zoom Out Reset image size

In 2001, the final report of the HSTKP (Freedman et al. 2001) made a detailed analysis of the earlier LMC distance measures and concluded (based largely on Madore & Freedman 1991) that μ = 18.50 ± 0.10. This conclusion is quite reasonable and has been widely publicized and adopted as a consensus of our community.

After 2001, the literature still contains many reported new measures of the LMC distance by many different methods. But suddenly the wide scatter has disappeared. Alves (2004) is the only compilation that reports on post-HSTKP measures alone, and he finds that "The average of 14 recent measurements of [μ] implies a true distance modulus of 18.50 ± 0.02 mag, and demonstrates a trend in the past two years of convergence towards a standard value." He even goes so far as to say "Regarding the convergence of published LMC distance results, I suggest to you that the fat lady has begun to sing" (his italics). Popular press reports repeat the claim that there is "a trend in the last two years of, finally, convergence toward this standard value" (MacRobert 2004).

But no one has offered any explanation as to why the situation should suddenly improve from one where most methods were being dominated by large systematic errors to one where all these same methods have their systematic errors essentially vanish. Equally disturbing is the fact that Alves' compilation shows 14 values that are too consistent with the HSTKP value. That is, all 14 values have 18.50 within their quoted 1-sigma error bars. A chi-square test for the hypothesis that μ = 18.50 has a reduced χ² of 0.23 (with 13 degrees of freedom), which is improbably low (with a probability of 0.0022) if the error estimates are correct. (Alves only added one sentence saying "The reduced chi-square is less than 1, which suggests that the adopted error bars may be too conservative" after I had pointed out the problem as it appeared in his preprint.) I next performed a cursory literature search and quickly turned up seven more measures, with the entire list of 21 measures having a reduced χ² = 0.19 and a chance probability of such a low value of 0.00003. This is a disturbing situation, so I have made a thorough analysis, and this is what I report on in this paper.

I have collected published measures of the LMC distance modulus that have appeared after 1997 January 1. For dates before 2002, most of the publications were compiled by Benedict et al. (2002a) and Clementini et al. (2003), with these not being listed separately in this paper. My search for additional measures was made with the ADS database (http://adsabs.Harvard.edu/), the astro-ph preprint server (http://www.arxiv.org/astro-ph/), and the indexes of various journals. I suspect that I have not found all reported measures of μ, but likely there are only a few missing, and the completeness is not correlated with the reported μ value. In all, I have found 45 papers that report on μ with error bars, and these are tabulated in Table 1. The additional 99 values were collected from the papers of Benedict et al. (2002a) and Clementini et al. (2003). The first column gives a date for each paper, usually its acceptance date. The second and third columns cite the reference and a short identification of the method used. The fourth column gives the quoted μ value and its 1-sigma uncertainty (σ). The fifth column gives the deviation of the reported μ value from that of the HSTKP value (18.50 mag) divided by the 1-sigma uncertainty, that is D = (μ − 18.50)/σ. If the true μ value is 18.50 mag and if the various reported uncertainties are correct, then D should be a Gaussian distribution centered at zero with an RMS scatter of unity.

A variety of minor dilemmas occurs when trying to summarize a long paper into one number for Table 1. For example, in Fitzpatrick et al. (2003), the distance to the eclipsing binary HV5936 is given as μ = 18.18 ± 0.09, but they then correct this distance to the middle of the LMC galaxy as reported in Table 1. Benedict et al. (2002a) report their value as 18.38–18.53 with error bars of −0.11 and +0.10, while I have represented this as 18.455 ± 0.13. Keller & Wood (2002, 2006) only include one source of uncertainty into their very small quoted error bars (±0.02 mag and ±0.018 mag, respectively) despite discussing additional uncertainties that would greatly increase the error bars, yet I have used their published error bar. Bono et al. (2002b) present two results (18.53 ± 0.08 based on a theoretical calibration and 18.48 ± 0.13 based on an empirical calibration) which I have combined as 18.51 ± 0.11 in Table 1. Sebo et al. (2002) report on values for two cuts for which their best results (de-reddened and with a plausible galaxy tilt) are 18.53 and 18.54 mag, but no error bars are quoted so this measure was not included in my Table 1.

Some of the papers report multiple values for μ based on alternative reasonable assumptions or ranges of acceptable inputs. (As pointed out in the last section of this paper, this is a good practice.) Should these multiple values be presented in Table 1 as individual lines or as some combined measure? With the alternative μ values usually being based on analyses that differ by only a small fraction of their input, the values are highly non-independent, and as such they should not be separately listed in Table 1. Rather, the range of alternative values is an expression of the systematic errors not included in the quoted error bars. As such, this systematic uncertainty was added in quadrature with the quoted error bars to yield a total uncertainty for inclusion in Table 1.

Alves (2004) claims that the post-HSTKP measures of the LMC distance have converged. But is this convergence too good?

A quick scan through Table 1 shows many cases where the measured value is close to the HSTKP value. Of the 31 measures dated after 2002, all but one (i.e., 97%) are within 0.10 mag of the HSTKP value. (This is to be compared to the 46% between 18.40 and 18.60 for the pre-HSTKP compilation of 84 papers from 1997–00 (Benedict et al. 2002a). This demonstrates the dramatic change around the time of publication of the HSTKP results.) A formal weighted average of the post-2002 measures returns a value of 18.51 ± 0.01 mag (an unweighted average gives 18.49 ± 0.01 mag), which is very close to the HSTKP value.

The disturbing property of Table 1 is the over-abundance of measures that agree with the HSTKP to much greater accuracy than the quoted error bars. For the post-2002 measures, 87% (27 out of 31) of the measures have the reported μ value within 1-sigma of the HSTKP value (i.e., |D| ≤ 1), whereas the expected fraction is 68%. (Indeed, the fraction would be 90% had not Keller & Wood quoted an artificially low error bar.) Also, 68% (21 out of 31) of the measures have the reported μ value within 0.5-sigma of the HSTKP value (i.e., |D| ≤ 0.5), whereas the expected fraction is 38.3% (12 out of 31). These numbers are startling deviations from the expectations.

A chi-square analysis can be used to test whether the overall spread of the deviations (D) is as expected for a model that the true μ value is that given by the HSTKP (18.50 mag). The χ² value will simply be the sum of the squares of the D values in the last column of Table 1. The number of degrees of freedom equals the number of reported measures (31) as there are no adjustable model parameters since I calculated D with the model of the HSTKP μ rather than the weighted average of the values. I find that χ² = 20.3 for a reduced chi-square of 0.65. This value is not exceedingly small, being surprising at only the 2-sigma level. (If the Keller & Wood (2006) measure is excluded due to their not including their stated large systematic errors, then the reduced chi-square becomes 0.51, with this being improbable at the 1% probability level.)

However, the chi-square test is not the most sensitive test for the question of whether the reported values cluster too tightly around the HSTKP value. The reason is that we already know that historically there are large systematic errors unaccounted for (cf. Benedict et al. 2002a) and just a few of these included in the post-2002 sample will provide a high chi-square contribution that would mask an anomalous concentration. An additional reason is that any exclusion of systematic uncertainties will also provide an erroneous high chi-square contribution. Indeed, just three of the post-2002 measures (one with an admitted low error bar of ± 0.018 mag) provide 14.0 out of the total χ² = 20.3.

A more sensitive test than the chi-square for the question in hand is the Kolmogorov–Smirnov (K–S) test (Press et al. 1986). Here, the observed distribution of differences from the HSTKP value is compared to the ideal distribution arising if the LMC distance measures have a Gaussian distribution with the quoted error bars. As the deviations from the Gaussian shape are symmetric, I will examine the observed distribution of |D| as this combines any effects of clustering on both the positive and negative sides. This K–S test is quite general in that it makes no assumption as to the fraction or size scale in the cluster, and the effects of outliers is minimal.

The K–S test starts with the construction of cumulative distributions where the fraction of values less than |D| are tabulated from small to large |D|. For small |D| the fraction will be near zero while for large |D| the fraction will approach unity. Two such cumulative distributions are constructed, one for the 31 observed post-2002 values and the other for the Gaussian model (see Figure 1). With normal fluctuations, the two cumulative distributions will deviate from each other, and they will have some maximum difference. The greater the maximum difference, the lower the probability that the observed distribution is correctly modeled by the theoretical distribution. The maximum difference between the two cumulative distributions of |D| is 0.33 at |D| = 0.59. This large difference is unlikely to occur, with the probability of good data and a good model producing such a large difference equal to 0.0023. (With the exclusion of the Keller & Wood report (owing to its artificially small error bar), the probability is reduced to 0.0010.) The probability of 0.0023 corresponds to worse than a 3.0-sigma discrepancy, and constitutes strong evidence that the observed distribution does not match the model.

So here we have it, the post-2002 measures of μ are too-tightly clustered to the HSTKP value (compared to what is possible given the quoted error bars) with a >3-sigma confidence level. There are only two ways that D = (μ − 18.50)/σ can be systematically near zero, and that is either if the published μ values are artificially selected/adjusted to be near 18.50 mag or if the published σ values are systematically too large by about a factor of 2. Either possibility strikes at the heart of the reliability of the post-2002 reports.

For understanding the artificial clustering of LMC distances, it is helpful to track how this clustering has evolved in time. We already know that the measures before ∼2001 have large unrecognized systematic errors while the measures after ∼2002 have a clustering too tight to be consistent with their reported errors. But exactly when did this transition occur and how long did it take?

To answer these questions, I have assembled all the published measures of LMC distances from 1997 to the present. These come from Benedict et al. (2002a), Clementini et al. (2003) and Table 1. These measures of μ are plotted as a function of date in Figure 2. In addition, the values of |D| are plotted as a function of date in Figure 3.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

These 144 measures have also been divided into various time intervals, with their statistical properties tabulated in Table 2. The first column gives the years or year ranges for the input. The second column gives the number of published values in the date range. The third and fourth columns give the unweighted average and the RMS scatter for the reported μ values in the time interval. The last column gives the median |D| value.

From 1997 to 2001, the average LMC distance modulus, 〈μ〉, was bouncing up and down from 18.33 to 18.55. After 2002, 〈μ〉 was relatively constant (18.46–18.52) and was close to the HSTKP value. Apparently, the transition occurred around the end of 2001.

From 1997 to 1999, the RMS scatter in the reported μ values was relatively large (i.e., ≥0.13 mag). From 2003 to present, the RMS scatter has been relatively low (i.e., ≤0.06 mag). Based on this measure, the transition occurred between 2000 and 2002, with the RMS scatter having intermediate size (i.e., 0.10– 0.11 mag).

The median |D| value is a reasonable measure of the degree of too-tight clustering toward the central value. This measure is relatively insensitive to the existence of outliers and it can be directly compared to the expectation for a good Gaussian distribution. (for which the median |D| should be 0.68). From 1997 to 2002, the median |D| was larger than the value expected for good Gaussian errors. From 2003 to present, the median |D| has been much smaller than expected for good Gaussian errors. From this, it looks like the transition occurred in 2002.

From these three statistics, it appears that the transition from widely scattered to tightly clustered (when compared to the quoted error bars) occurred sometime around 2002. The data do not have enough resolution in time to set tight limits on the duration of the transition, so the transition might be fast and sometime in 2002 or it might be a two-year-long interval from around 2001–2.

The timing and duration of the transition allows us to reject some possible causes. For example, various substantial advances in data that have widespread application amongst all the methods do not correspond in time to the observed transition. In particular, the sudden availability of Hipparcos parallaxes and the OGLE and MACHO photometry both happened long before the year 2000 and hence cannot be responsible for the transition. In addition, the relatively short duration of the transition indicates that the transition is not caused by the steady accumulation of improvements in methods, whether applicable to individual techniques or applicable to multiple techniques. The short unresolved transition is consistent with some single cause dating to around 2001–2.

The HSTKP paper was submitted on 2000.6, accepted on 2001.0, and published on 2001.4. The tremendous publicity for the HSTKP was throughout 2001. If the transition was caused by the HSTKP, then the transition should appear sometime around 2001 plus the average time for new research to be performed plus the average time resulting from the publication process. As such, we have a prediction that if the transition is caused by the HSTKP, then the transition should last for a year or so starting sometime in 2001.

We have a close match between the observed transition date (2002 or perhaps 2001–2) and the predicted transition date (2001–2) if the HSTKP is a primary cause of the transition. This coincidence in time does not prove a causal connection. However, given that we know independently about the massive publicity and the instant 'consensus' caused by the HSTKP, the coincidence in time does provide reasonable evidence that the transition is related to the HSTKP.

The 31 post-2002 measures of the LMC distance use six classes of standard candles, with many different methods for calibrating each. Nevertheless, there is certainly overlap in data, methods, and assumptions between the papers. Are the 31 measures largely independent? If not, then the effective number of degrees of freedom used in Section 3 might be substantially smaller than 31. Another way to look at this is that the existence of substantial correlations amongst the 31 measures could cause individual reports to be clustered more tightly than their quoted error bars.

The most commonly used class of standard candles in the 31 post-2002 papers is the Cepheids, with 13 papers. This is the case where correlations would be the most pervasive. So in this Section, I will examine in detail the correlations amongst the measured LMC distances based on Cepheids.

I have examined the 13 post-2002 Cepheid papers and I found that they have a little overlap. These papers calibrate the absolute magnitude of Cepheids by many different methods, including theoretical models, parallaxes, the Baade–Wesselink method, the hydrogen maser distance to NGC4258, and main sequence fitting of galactic clusters with Cepheids. This very wide range of calibration methods provides confidence that any correlations will be substantially restricted. To provide a detailed analysis, I will closely describe the relations between the groups of Cepheid papers that share the same calibration methods.

The first group is the three papers that calibrate their LMC Cepheid distances with theoretical models (Moskalik & Dziembowski 2005; Keller & Wood 2006; Testa et al. 2007). In all cases, the theoretical models are completely independent by widely separated groups. Indeed, the models are for completely different classes of Cepheids, with the three papers modeling triple-mode Cepheids, bump Cepheids, and normal Cepheids, respectively. The input light curves of the LMC Cepheids were taken from OGLE-II I-band data, MACHO BR data, and Las Campanas JK data, respectively. There is no overlap between the target stars. As such, these three apparently similar papers have no correlation.

The second group is the two papers (Abrahamyan 2004; Benedict et al. 2007) that calibrate the Cepheid absolute magnitudes with parallaxes of galactic Cepheids. The basic inputs are completely different as Abrahamyan uses Hipparcos parallaxes while Benedict et al. use HST parallaxes. On top of that, the two papers use completely different LMC data and substantially different analysis assumptions. So despite the use of a few shared targets for calibration, the data and analyses are completely independent.

The third group is the three papers (Storm et al. 2004, 2006; Gieren et al. 2005) that all share a core of collaborators and the same technique (the Baade–Wesselink method) to calibrate the LMC Cepheid distances. These three papers have a greatly different input. Storm et al. (2004) apply the Baade–Wesselink method to galactic Cepheids so as to calibrate a period–luminosity (P–L) relation, Gieren et al. (2005) apply the Baade–Wesselink method to 13 Cepheids across the LMC bar, and Storm et al. (2006) apply the Baade–Wesselink method to six Cepheids in the LMC cluster NGC1866. The same three papers use completely independent input data in the VIWJHK, VIWJK, and K bands, respectively. As for assumptions, these three papers use a metallicity correction but no projection correction, no metallity correction but with a projection correction (p = 1.58–0.15 log P), and no metallicity correction but with a different projection correction (p = 1.39–0.03 log P), respectively. In all, these three papers are completely independent in input.

Nevertheless, the 13 post-2002 Cepheid papers do carry some specific overlap. The paper of Storm et al. (2004) simply reports the result of Fouqué et al. (2003) plus a correction for metallicity effects, and so their two results are clearly strongly correlated. Both Fouqué et al. (2003) and Macri et al. (2006) adopt the OGLE P–L relations as their LMC information, although they have greatly different methods of calibration. Ngeow & Kanbur (2007) base their calibration on the Gieren et al. (2005) distances from the Baade–Wesselink method, but they also use 630 other LMC Cepheid V-band light curves from OGLE and completely different methods and assumptions for analysis. These three cases are the only significant overlap that I can find. So, out of the 13 Cepheid papers, only two (Fouqué et al. 2003; Ngeow & Kanbur 2007) are not completely independent of each other.

A detailed examination of all the other post-2002 papers shows no significant overlap of data or targets. Out of all the 31 papers, I can only point to two that have any substantial overlap with other papers. In all, I conclude from these detailed analyses of the post-2002 papers that correlations are indeed present but only at a rather small level.

An alternative test for paper-to-paper correlations is to look at the median |D| values for just the Cepheid papers from 1997 to the present. Any correlations amongst the Cepheid papers will certainly be stronger than the correlations amongst all 31 papers. As such, if the correlations are significant, then the Cepheid median |D| values should be more tightly clustered than those for all papers. If the correlations are not significant, then the Cepheid median |D| values should be similar to those of all the papers. So we have a test for the significance of correlations in creating the too-tight clustering reported in Section 3.

I have separated out the Cepheid-based measures from the entire data base. This results in 10, 13, and 13 measures in the year ranges of 1997–9, 2000–2, and 2003–7. The median |D| values are 1.03, 0.60, and 0.55 for before/during/after the transition. This is to be compared with 1.24, 0.76, and 0.43 respectively for all papers (see Table 2). The typical uncertainty in the median |D| for the Cepheids will be roughly ±0.3 (the RMS scatter in |D| divided by the square root of the number of items). As such, the two series of median |D| values are consistent. This is interpreted as implying that paper-to-paper correlations are small and do not cause the basic dilemma from Section 3.

Further insight into the cause of the too-tight clustering of LMC distances can be obtained by comparison with the situation for the Small Magellanic Cloud (SMC) distance modulus (μ_SMC). The task of measuring the distance to the SMC is largely identical with that of measuring the LMC distance. As such, if there are any systematic advances that cause the clustering of LMC distances then these same advances should also apply to the SMC distances resulting in a similar too-tight clustering. Alternatively, if the LMC distance clustering is caused by 'sociological' influences connected with the HSTKP, then the SMC distances should display little clustering. As such, we have a means of distinguishing the cause of the transition.

Unfortunately, the literature contains much fewer reports of SMC distances than of LMC distances. (The reasons are the SMC is not the lynchpin for the entire extragalactic distance scale, its slightly farther distance makes it somewhat harder to get adequate observations, and its smaller mass means that there are fewer targets.) From 1991 to the present, I find only 29 papers that report a derived SMC distance with error bars. I have divided them into time intervals 1991–9 (with 14 measures), 2000–2 (with 10 measures), and 2003–7 (with 5 measures) so as to correspond to before/during/after the LMC transition.

The average (and median and weighted average) is close to μ_SMC = 18.9 mag. So I take the D value for the SMC to be D_SMC = (μ_SMC − 18.9)/σ. The RMS scatter of μ_SMC is 0.21, 0.14, and 0.11 mag for before/during/after the LMC transition. The median value of |D_SMC| is 1.0, 1.0, and 1.2 for before/during/after the LMC transition.

If the LMC transition (with the median |D| going from 1.24 to 0.43) is caused by advances in techniques, then we would expect the median |D_SMC| to undergo a similar transition. But this is not seen, as we have the median |D_SMC| always ≥1 and even increasing somewhat after the transition. We do see a long-term improvement in the RMS scatter of the SMC distances (perhaps caused by the various real advances and improvements over the last decade), but this is not a clustering with respect to the reported error bars.

If the LMC transition is caused by "sociological" effects connected with the widespread publicity for the HSTKP, then we would expect little effect on the SMC distances. The reason is that the SMC was not included in the HSTKP. Indeed, as predicted by this possibility, the |D_SMC| does not undergo a transition from ≥1 to ≤0.5. If anything, |D_SMC| grows even larger after the LMC transition. As such, the SMC distance measures provide reasonable evidence that the cause of the LMC transition is somehow associated with the HSTKP.

We are faced with a dilemma because the post-2002 distance measures for the LMC are greatly more concentrated around the HSTKP value than is plausible given the published error bars. Here are three possible interpretations.

7.1. Correlations Between Papers

7.2. Most Quoted Error Bars Have Large Over-Estimates

7.3. The Band-Wagon Effect

The LMC μ measurements from before the HSTKP are widely scattered with the scatter being much larger than expected based on the reported error bars. After the HSTKP, the 31 LMC μ measurements are tightly concentrated around the HSTKP value of 18.50 mag, with this scatter being much smaller than expected based on the reported error bars. The post-2002 deviation from a Gaussian distribution of errors is significant at the >3-sigma confidence level. The date of this transition is unresolved and around the year 2002 or perhaps in 2001–2.

Three interpretations were considered, although other interpretations might be possible. Correlations among the 31 post-2002 papers are certainly present, but they cannot account for a significant fraction of the sudden post-HSTKP concentration. The over-estimation of error bars is unlikely to provide any substantial amount of concentration. The band-wagon effect has strong positive evidence, including the unresolved duration of the transition, the coincidence in time with the HSTKP, the successful prediction that the Cepheid papers would behave similarly with the whole data set, and the successful prediction that the SMC distances would not suffer the same too-tight clustering. This band-wagon effect has many strong precedents even amongst the best astronomers and physicists from the last decade and century. In all, the evidence strongly points to the cause of the too-tight clustering of the post-2002 LMC distances as being due to a band-wagon effect.

All of these interpretations are worrisome, as they strike at the heart of the measurement process for one of the most important astronomical parameters. So what do I recommend to help this situation? The obvious suggestion is a very close examination of all 31 post-2002 papers for correlations and possible alternative inputs. Ideally, a network containing all plausible sets of assumptions (perhaps with assigned probabilities) can be applied uniformly to the data from the 31 papers so as to produce a single μ value based on all the data along with error bars that reflect the true range of reasonable inputs. However, such a large analysis is beyond the scope of this paper or indeed of any individual researcher. As a start, some group of knowledgeable experts might take all the calibration data and apply it to all available LMC data for one particular method, say the Cepheids or the RR Lyrae stars. With such comprehensive analyses for the various methods, they can then be combined to produce some final value with a realistic error bar. In the meantime, individual researchers should report their own analysis in such a way that correlations and assumptions are explicitly given. Also, to avoid selection effects, future papers should quantitatively detail the effects on the derived μ of all available data sources, all plausible data cuts, and all reasonable input assumptions in all combinations. Alternatively, a researcher could in advance specify the experimental choices in complete detail (without regard for how these selections will change the answer) and then stick with these choices all the way to the end.

The purpose of this paper is to alert workers in the field to the problem. Part of this alert is simply to let us know that the current best estimate of μ is more like the HSTKP value of 18.50 ± 0.10 mag, instead of something like 18.50 ± 0.02 mag. Another part of this alert is to encourage groups of experts to synthesize a global approach for individual methods as well as to combine the various methods. A final part of this alert is to encourage all LMC-distance workers to draft their future papers either with all selections specified rigidly in advance or with the derived μ values for all reasonable combinations of selections explicitly given in the paper.

I thank Chris Britt and the referee for help with this paper.