Spectral fit residuals as an indicator to increase model complexity

Spectral fitting of X-ray data usually involves minimizing statistics like the chi-square and the Cash statistic. Here we discuss their limitations and introduce two measures based on the cumulative sum (CuSum) of model residuals to evaluate whether model complexity could be increased: the percentage of bins exceeding a nominal threshold in a CuSum array (pct$_{CuSum}$), and the excess area under the CuSum compared to the nominal (p$_\textit{area}$). We demonstrate their use with an application to a $\textit{Chandra}$ ACIS spectral fit.


INTRODUCTION
Spectral fitting of X-ray data has been usually done by minimizing a statistic like χ 2 or cstat (Cash 1979;Avni 1976).The value of the statistic is useful to evaluate the goodness of the fit, which is also used as a stopping rule in evaluating model complexity.Fits are deemed acceptable if χ 2 /ν < 1 + 3 • 2 ν (where ν is the degrees of freedom in the fit) or ∆cstat< 3 • σ cstat (where σ 2 cstat is the estimated variance for the nominal model as given by Kaastra 2017).When the fitting statistics or F-test values drop below the threshold defined by the null distributions, it is commonly acknowledged that further increases in model complexity (or the number of free parameters in the fit) are statistically unsupported.However, these measures are global measures of fit, and leave contiguous ranges in data space with correlated deviations in the residuals.Here we introduce summary statistics that identify the presence of such structures in the residuals and describe how to use them to decide whether the model complexity should be increased.
We use a Chandra/ACIS-S spectrum of the corona of an exoplanet hosting star (HD 179949, ObsID 6122;Acharya et al. 2023) to illustrate the measures (see Figure 1).The data and the spectral fits made with different emission models are shown in the left column of the Figure .The models increase in complexity from top to bottom (see Acharya et al. 2023 for details), with none of the statistical measures (cstat, pct CuSum , p area ) acceptable for the simplest model, cstat being acceptable for the model in the middle row, and all measures acceptable for the model in the bottom row.Below, we define pct CuSum and p area , describe their rationale, and how to compute them and their null distributions.

CUSUM: DEFINITION, CALIBRATION, AND SIGNIFICANCE
Spectral fit residuals can be expressed as with c i the observed counts in bin i of N and m i the predicted model counts in the same bin, generated for an astrophysical model defined by the best-fit parameters θ.The cumulative sum (CuSum) at the j th bin is (1) Corresponding author: Anshuman Acharya anshuman@mpa-garching.mpg.de While the ordering of indices can be reversed, CuSum preserves the sequential order of the bins, thus incorporating extra information ignored in χ 2 or cstat statistics.The null distribution needed to evaluate the quality of the CuSum of a best-fit can be built in one of two ways: − generate K mock datasets using the fake pha() function in Sherpa, and fit using the same model, such that for each mock data set k, we obtain corresponding best-fit parameters θ k ; or − obtain K post burn-in draws {θ k , k = 1, . . ., K} as iterations during a Markov Chain Monte Carlo (MCMC) fit using the get draws() tool in Sherpa (based on BLoCXS; van Dyk et al. 2001).Each θ k yields a CuSum array {CuSum(θ k ) j , j = 1, . . ., N }, and the resulting sample defines the null distribution.This null distribution characterizes the strength of the deviations present in CuSum( θ) j .We construct two statistics that probe deviation at different scales, pct CuSum to detect biases in the model continuum and p area to detect presence/absence of narrow lines.
1. pct CuSum : We compute the equal-tail 90% point-wise range [C 05 J , C 95 J ] of CuSum(θ k ) j=J for each detector channel J in the specified passband.Then we evaluate the percentage of bins, pct CuSum , where the observed CuSum exceeds the 90% bounds, i.e., if n channels have CuSum( θ) J < C 05 J or CuSum( θ) J > C 95 J , then pct CuSum = 100 • n N .If pct CuSum ≳10%, the model used is considered inadequate and we suggest that more complex models with more free parameters should be explored.If, on the other hand, pct CuSum ≪10% this is a sign of overfitting, and thus less complex models are favored (see the middle column of Figure 1).
2. p area : For each of the k = 1, . . ., K CuSum arrays in the null, we calculate the total area that falls outside the 5-95% bounds across all the channels, where the indicator indices Z + = Z − = 1 if the corresponding terms in the summation are +ve, and 0 otherwise.We then compute a similar area measure for the best-fit spectrum, The set {area k , k = 1, . . ., K} now forms a null distribution of excess areas to which the value of area( θ) can be compared to obtain a p-value.The statistic p area is the probability that area( θ) > area k .We consider the best-fit model an inadequate fit if p area ≪0.05, as an indication that a large deviation is present in CuSum( θ) j (see the right column of Figure 1).This process is similar to the posterior predictive p-value calibration procedure described by Protassov et al. (2002).

APPLICATION
In Figure 1, we show the results of the residuals analysis for three best-fit models for an exemplar dataset of HD 179949 (Acharya et al. 2023).The top row represents a single temperature APEC model with variable metallicity (1m).The ∆cstat = +3.3• σ cstat , indicates a poor fit.This is supported by the pct CuSum = 40.6%and a p-value = 0.0.The middle row represents a 2-temperature model with metallicities scaling together (2m).The ∆cstat = +2.1 • σ cstat , indicates an acceptable model.The argument for improvements is solidified by pct CuSum = 34.2% and p-value = 0.0.The bottom row represents a 2-temperature model with abundances grouped by First Ionization Potential and varying in several groups (2v).This gives ∆cstat = +1.3• σ cstat , pct CuSum = 13.5% and p-value = 0.09.This model is therefore accepted as the best representation of the data.
pct CuSum is sensitive to broad differences such as would be present in a biased fit of the continuum, and p area to small scale changes such as appear when spectral lines or abundance differences are mis-estimated.Our method guards against overfitting to statistical fluctuations in the data by specifying criteria for acceptability that are tied to the null distributions.We note that our arguments and method are general, and can be extended to any non-linear fitting scenario to supplement fitting procedures.

Figure 1 .
Figure 1.Demonstrating the improvements in the measures of fit quality as model complexity increases from top to bottom (see text).Each column highlights a different statistic: ∆cstat on left, pctCuSum in the middle, and parea on right.The simplest model in the top row fails in all measures; the ∆cstat value in the middle row is borderline acceptable, but the CuSum based measures are not; and in the bottom row all the measures indicate that the fit is adequate.Increasing model complexity further risks overfitting.