A novel predictive analytics score reflecting accumulating disease burden—an investigation of the cumulative CoMET score

Objective. Predictive analytics tools variably take into account data from the electronic medical record, lab tests, nursing charted vital signs and continuous cardiorespiratory monitoring to deliver an instantaneous prediction of patient risk or instability. Few, if any, of these tools reflect the risk to a patient accumulated over the course of an entire hospital stay. Approach. We have expanded on our instantaneous CoMET predictive analytics score to generate the cumulative CoMET score (cCoMET), which sums all of the instantaneous CoMET scores throughout a hospital admission relative to a baseline expected risk unique to that patient. Main results. We have shown that higher cCoMET scores predict mortality, but not length of stay, and that higher baseline CoMET scores predict higher cCoMET scores at discharge/death. cCoMET scores were higher in males in our cohort, and added information to the final CoMET when it came to the prediction of death. Significance. We have shown that the inclusion of all repeated measures of risk estimation performed throughout a patients hospital stay adds information to instantaneous predictive analytics, and could improve the ability of clinicians to predict deterioration, and improve patient outcomes in so doing.


Introduction
The early detection of clinical deterioration in hospitalized patients has benefits in terms of morbidity, mortality and healthcare economy (Young et al 2003, Liu et al 2012, Mardini et al 2012, Mokart et al 2013. Utilizing the 'HeRO' score, based on heart rate characteristics, our group has demonstrated, in the setting of a randomized controlled trial, that using only data from the ECG, an instantaneous risk score can be calculated that predicts the future onset of sepsis in very low birth weight infants. Such a score allows clinicians to intervene earlier, significantly decreasing mortality in this most vulnerable group of patients . All such scores are predicated on the finding of a 'signature of illness' (Griffin et al 2004) in the continuous cardiorespiratory monitoring data, which can be detected using mathematical time-series analyses, and mapped to the risk of an event in the next time window.
The ethos underlying the HeRO score was translated to the discovery of signatures of illness in adult (Politano et al 2013, Moss et al 2016, Blackwell et al 2020 and pediatric ICU (Spaeder et al 2019) patients, and adult ward patients (Blackwell et al 2020). In these settings, risk is displayed as the 'CoMET' ('Continuous Monitoring of Event Trajectories') score. In addition to parameters extracted from continuous cardiorespiratory monitoring data, the score takes account of vital signs and lab results from the electronic health record (EHR).
The outputs that generate CoMET, which are the products of logistic regression, are instantaneous metrics of 'cardiorespiratory' and 'cardiovascular instability', and the output display charts the trajectory of this instability over the prior 3 h to allow identification of patients with a deteriorating trajectory (see figure 1) (Ruminski et  A potential shortcoming of this approach is that, while excellent at reflecting patient trajectory now, i.e. based on the prior several hours of data, the CoMET score is unable to reflect the cumulative burden of illness sustained throughout the days and sometimes weeks or longer of their hospital stay. As such, the score may fail to reflect all that we have learned about a given patient and their illness to date. We hypothesize that it would be better to develop a predictive analytics tool that is reflective of all accrued data since the patient's admission, to better refine our ability to predict outcomes and forewarn of deterioration. Figure 1. The CoMET display as presented to treating teams. Panel a shows 48 h data from a patient in room 89, whose instantaneous cardiovascular and respiratory risk scores are low. Panel a,i shows the current CoMET for this patient (bed number is displayed in the head of the CoMET, i.e. the circular marker at coordinates 0.2, 0.9 in the chart shown), which is small, and pale, reflecting the fact that the instantaneous risk is low. It is plotted on a graph with 'cardiovascular instability' on the x-axis, and 'respiratory instability' on the y-axis. The numbers are the fold increase in risk of an event within the next 6-12 h. A score of 1 means that the risk of an event is at the average risk for that unit, with higher numbers meaning higher risk. The tail of the CoMET is barely visible, suggesting that the patient has been at this level of risk for the prior 3 h. The contours on the graph delineate the expected percentile of patients in each contour of the graph, with darker shades of gray indicating more frequently populated areas-this patient is in the darkest gray area, and as such is exhibiting expected levels of risk for that unit-'nothing out of the ordinary'. Panel a,ii shows the 'leaderboard'-all of the patients on a given unit ordered from top to bottom based on their current CoMET score, highest to lowest. The 3 h tail of the CoMET is also shown to demonstrate the degree of recent stability or instability. Individual patients can be selected for a deeper dive by checking the box to the left of their bed number (the check next to the patient in bed 89 can be seen). This brings up the graph shown in panel a,iii, which is a 24-, 48-or 72 h graph showing the progress of the patient's CoMET scores over the defined period, with cardiovascular instability shown as the red line, and respiratory instability shown as the green line. Panel b reflects the data from a different patient in bed 61, who appears much less stable. Their CoMET (b,i) is large and bright red, emphasizing the current high level of risk, mainly in respect of their risk of respiratory instability. The tail of the CoMET demonstrates that this instability has substantially progressed over the prior 3 h, and that the patients CoMET now occupies a position on the graph which is rarely occupied (very pale contour), and ought to draw clinical attention to the patient if it has not done so already. The leaderboard in b,ii has this patient at the top (most unstable on the unit), along with the CoMET tail which is long and increasing, underscoring the increasing degree of risk and instability. Selecting this patient using the check box next to their bed number brings up the graph shown in panel b,iii, which demonstrates the evolution of CoMET over the prior 48 h, showing recurrent and at times progressive respiratory instability in the face of relative stability of the cardiovascular risk. The instantaneous CoMET graph is deliberately selected at a time of peak respiratory instability (occurring at around 1 am on the day shown).
Herein, we investigate the value of the novel cumulative CoMET ('cCoMET') score, which represents a continuous summation of the q15 minute CoMET scores throughout a patient's hospital stay. We hypothesize that this will reflect the accumulating burden of physiological instability, and in doing so will be related to morbidity and mortality. This particular metric is akin to the HbA1 C measured in diabetes, and is a 'test with memory', so that regardless of whether there is little or much instantaneous or recent cardiorespiratory instability, the events that have gone before will still be reflected in the output predictive metric. We have previously explored this concept using the cumulative HeRO score in very low birth weight infants, demonstrating that a high cumulative HeRO score is significantly associated with in-hospital mortality (Griffin et al 2004). We hypothesize that the cCoMET will have the ability to summarize the entire hospital stay to date, and thereby give insights into the accumulated physiological insult suffered by a patient, which should assist in accurately predicting their trajectory and ultimately their outcome, and assist clinicians to direct their efforts to the most vulnerable patients under their care.

Methods
The study population We studied adult (age 18 years) patients consecutively admitted to acute care beds for whom continuous ECG data was available at the University of Virginia Medical Center. The 71 monitored beds are arranged in 3 units and are under the care of a variety of hospital services, principally Cardiovascular Medicine and Cardiothoracic Surgery. An institutional electronic data warehouse archived the electronic medical record (EMR) data, including admission, discharge, and transfer information. Patients with a length of stay of <72 h were excluded from the analysis due to a lack of accumulated data to facilitate meaningful cumulative predictive analytics.

EMR vital signs and laboratory results
At 15 min increments, we recorded the most recent charted vital signs measurements and laboratory tests as described elsewhere (Moss et al 2017). We excluded observations occurring after 'Do Not Resuscitate' (DNR) or 'Do Not Intubate' (DNI) orders or after transition to comfort measures-only (CMO).

Cardiorespiratory dynamics measured from continuous ECG monitoring
Heart rate dynamics We processed the continuous ECG with multiple QRS detection algorithms on the ECG lead with the highest signal to noise ratio. The three resulting heart beat time series were combined to determine the probability density of each detected heart beat. Low confidence beats were excluded from the analysis. We made observations every 15 min of the preceding 30 min and calculated the mean interbeat interval, the standard deviation or HR variability, and nonlinear dynamics of HR (Pena et al 2009, Lake and Moorman 2011, Moss et al 2014.

ECG derived respiratory rate and respiratory sinus arrhythmia
We estimated the respiratory rate (RR) from both the cyclic variation in ECG waveform characteristics that result from respiratory movements and, when present, from the respiratory sinus arrhythmia peak in the frequency spectrum of heart beat intervals. We analyzed 60 s interbeat interval time series windows containing 20 or more heartbeats. Windows overlapped by 75%. Details of how calculations were performed can be found in our previous publications ( Calculating the CoMET score CoMET uses continuous cardiorespiratory monitoring data and waveforms sampled every 2 s to perform mathematical measurements, such as measures of entropy and heart rate variability. R-R intervals and electrocardiogram-derived breathing rate were obtained from 200 Hz electrocardiogram waveforms; laboratory data and nurse-entered vital signs were obtained from the electronic medical records. Together these are used to derive an estimate of the fold increase in the risk of clinical deterioration. The models that have informed CoMET development have been described previously ( Comprehensive clinical data that is incorporated into the CoMET score are shown in table 1. The CoMET score is the fold-increase in probability of an adverse event occurring in the next 8 h, with a score of 1 meaning that the risk of the event occurring is the average risk for that patient in that particular unit. CoMET scores are calculated every 15 min to give a frequently updated estimation of the risk of imminent events, for example emergent transfer to the ICU. The CoMET display as presented to clinicians is shown in figure 1 for 2 patients-a stable patient, and an unstable deteriorating patient.

Calculating cCoMET
Different to the CoMET score described above, which is an instantaneous estimate of risk of an imminent event, the cCoMET was developed to be a dynamic risk marker with memory. Calculation of the cCoMET was performed every 15 min by comparing the instantaneous CoMET score to a dynamic baseline predicted risk, individualized for that patient: This calculation is made every 15 min using up-to-date real-time data, and the q15 minute score is progressively summed throughout the patients stay until discharge or death (see figure 2(c)). The baseline risk model is a logistic regression model where the outcome is death and the predictors are as follows: age, time since admission, admitting service, race, and sex. The baseline risk score is used as a 'reference' at every time-point with which to compare the CoMET score at that time-point. Positive or negative contributions to cCoMET are calculated based on whether the instantaneous CoMET risk score is above or below the instantaneous baseline risk. In this way, the risk attributable to the demographic and other factors included in the baseline risk score are taken into account at each stage, before positive or negative contributions are given to the cCoMET. This means that cCoMET, and its evolution, can be compared across patients regardless of the contributions to the risk of the age, admitting service and the other components of the baseline risk. For clarity, if the instantaneous CoMET score was below the baseline risk, the patient received a negative contribution to their cCoMET reflecting the fold-risk amount that the CoMET was below the baseline predicted risk. If the patient's CoMET scores remained below the baseline predicted risk, then their cCoMET would become progressively more negative over time. Conversely, if the instantaneous CoMET score was above the baseline predicted risk, the patient received a positive contribution to their cCoMET proportionate to the amount the CoMET was above the baseline predicted risk. If the patient remained above the baseline predicted risk, they would develop a progressively more positive cCoMET. Negative or positive contributions to the cCoMET were tallied every 15 min from the

Results
We retrospectively calculated the cCoMET scores of 8105 patients admitted to the acute cardiology care floor of the University of Virginia Medical Center from October 11, 2013 to September 1, 2015. Typical reasons for admission to these beds included acute coronary syndromes, heart failure exacerbations, arrhythmias and sudden cardiac deaths, step downs from intensive care units, and both pre-and post-solid organ transplant. After removal of patients whose stay was <72 h, 5363 patients remained for analysis. cCoMET's relationship to length of stay Since cCoMET is cumulative over time, we examined whether there was a marked relationship between cCoMET and length of stay. If cCoMET increased monotically with length of stay, then it would not fit our purpose as an indicator of illness burden. However, figure 3(a) demonstrates there is no large dependence of cCoMET on length of stay. While the final cCoMET score was higher in those patients in the quintile of longest length of stay, and was progressively lower with each shorter length of stay quintile, these differences were small. Length of stay (LOS) quintiles 1 thru 5 had median cCoMET scores of −0.44, −0.18, −0.12, −0.12 and 0 respectively, (P = <0.001, Kruskal-Wallis test). Similar trends were seen in the relationship between baseline CoMET ( figure 3(b)) and baseline risk ( figure 3(c)) and length of stay. Figure 2. Examples of cCoMET score evolution in two example patients. (a) 4 d evolution of cCoMET score in a patient who survived hospitalization to discharge. Baseline risk is shown in blue. Actual CoMET score through admission is shown in green. cCoMET is shown in red. Because the CoMET score is persistently below the baseline predicted risk throughout the admission, the cCoMET score becomes progressively more negative than zero throughout the admission, being around -5 on the day of discharge. (b) 4 d evolution of cCoMET score in a patient who died during this hospitalization. Colors are the same as in panel a. The CoMET score is persistently above the baseline predicted risk, and so cCoMET becomes progressively more positive through the admission, being around 23 just before death. x-axis numbers are probabilities of an event occurring in the selected CoMET model. (c) Equation for the calculation of cCoMET.
Distinct patterns of cCoMET evolution Exemplar patterns of cCoMET evolution are shown in figure 2. Panel a demonstrates the data from a patient who survived to discharge. Their CoMET score (green) was lower than the baseline model (blue) for the majority of their stay, and accordingly the cCoMET (red) became a progressively more negative value as time passed. This patient's final cCoMET score was more negative than −5 on the day of discharge, suggesting that this was a 'low risk hospitalization'. Panel b demonstrates another pattern in cCoMET score evolution-that of a patient who did not survive their hospitalization. In this patient, the CoMET score was persistently above the baseline model, and accordingly the cCoMET score became progressively more positive, until ultimately they died. At the point of death, the cCoMET was above 20.
Increasing cCoMET score was associated with mortality Higher cCoMET scores were significantly associated with mortality. Figure 4(a) demonstrates a rightward tail to the histogram in patients who died during this hospitalization, with a median cCoMET of 2.49 in patients who died versus −0.25 in those who survived (P < 0.0001, Kolmogorov-Smirnov test). There was a similar trend in the results for final CoMET score before discharge or death ( figure 4(b)), but the opposite trend in terms of baseline risk score, where patients who survived had a significantly higher baseline risk score compared to those who died ( figure 4(c)). Figure 5 demonstrates that in both patients who died (green circles) and those who survived (red circles), higher baseline CoMET scores were positively associated with higher final cCoMETs. Lower numbers on the xaxis suggest a lower baseline risk compared to expected on admission, while higher values suggest a higher baseline risk compared to expected on admission.
cCoMET adds information to the CoMET score when predicting mortality Figure 6 shows the log odds of death as a function of a logistic regression model using baseline factors (left panel), the cCoMET (central panel), and the last CoMET score reflecting the final 30 min of data (right panel)-the relationship between death and any of these parameters is steepest for the last CoMET prior to death (P < 0.001), and as such this is the strongest predictor, while cCoMET is less effective (yet still has a significant P = 0.03) and baseline CoMET is the least effective (non-significant relationship: P = 0.10). One might speculate that the reason that last CoMET score outperforms cCoMET in this circumstance is that significant numbers of patients die from sudden and unpredictable illnesses that could not have been foreseen using a tool such as cCoMET. Such diagnoses include ventricular arrhythmia, aneurysmal rupture, intracranial hemorrhage, acute myocardial infarction, and pulmonary emboli. Figure 7 is a scatter plot of final cCoMET by sex, demonstrating that males in the studied cohort are more likely to have higher (more positive) cCoMET scores, and hence to have high risk hospitalizations, than their female counterparts (median −0.16 versus −0.33, P < 0.001 on Kolmogorov-Smirnov test). . The median final cCoMET in each group is depicted by the broken line, and was significantly higher in those who died (2.49 versus −0.25, P < 0.01, Kolmogorov-Smirnov test); (b) Histogram depicting the relative frequency of individual final CoMET scores by whether the patient survived (red bars) or died (green bars). The median final CoMET in each group is depicted by the broken line, and was significantly higher in those who died (0.030 versus 0.007, P < 0.01, Kolmogorov-Smirnov test); (c) Histogram depicting the relative frequency of individual baseline risk scores by whether the patient survived (red bars) or died (green bars). The median baseline risk score in each group is depicted by the broken line, and was significantly higher in those who survived (1.70 × 10 −2 versus 1.63 × 10 −2 , P < 0.01, Kolmogorov-Smirnov test); Figure 5. The relationship between baseline CoMET (the first CoMET score adjusted for by the baseline risk) and final cCoMET in patients who survived (red dots) and those who died (green dots) during the index hospitalization. In both groups, higher baseline CoMET scores were associated with higher final cCoMET scores.

Discussion
The development of risk prediction models to assist clinicians evaluate high versus low risk hospitalizations, and weigh this information to avoid potentially preventable deteriorations and attain optimal outcomes in patients, is a developing and rapidly expanding field (Monfredi et al 2021). The CoMET model referred to in this paper was developed from 146 patient-years of vital sign and ECG waveform time series, encompassing 9232 ICU admissions, and 1206 clinician chart review identified episodes of unplanned intubation, sepsis or hemorrhage (Moss et al 2016). The multivariate models that were developed from this data demonstrated C-statistics of 0.61-0.88 with respect to their ability to predict such events up to 24 h prior to their occurrence, and give the clinician a dynamic mean fold-increase in the risk of that event occurring.  In this new work, we have studied how to use the CoMET score to produce a new metric, the cumulative CoMET, or cCoMET, score, by comparing each individual 15 min CoMET score with respect to a 'reference' baseline risk model, and then summing the results throughout the patients stay. The baseline risk model here was simplistic, comprising age, time since admission, admitting service, race, and sex. The blue lines on figures 2(a) and (b) show that the baseline risk model varied relatively little with time, while every-15 min CoMET scores varied more markedly. CoMET scores above the baseline risk model added to the cCoMET score, while instantaneous CoMET scores below the baseline risk model subtracted from the cCoMET score. Thus, unlike the instantaneous CoMET score, cCoMET represents to some degree a historical record of risk during a patient's stay, and reflects not only on recent developments in the patients CoMET score compared to the baseline model, but on all CoMET scores since admission. One way to look at it is that the cCoMET reflects accruing risk or indeed damage, or lack thereof, occurring to a patient's health status during a given hospital admission. A very positive cCoMET score suggests that a lot of damage was accrued to a patient's health during a hospital admission, while a very negative cCoMET suggests that much less damage than expected occurred to a patient's health during a hospital admission.
Our main findings are that cCoMET is crucially little affected by length of stay, and that higher (more positive) cCoMET scores portend a higher risk of dying in a given hospital admission. We have also found that a high baseline CoMET is related to an ultimately higher cCoMET in both patients who survived or died during the index hospital stay. Furthermore, final cCoMET scores are higher in males than females in our patient group. In performing this work, we have sought to extend the value of instantaneous risk scores, and to ensure that all potentially valuable data accrued during a patient's hospital stay is brought to bear when assessing their ultimate risk, and see a role for cCoMET being used alongside CoMET scores to give clinicians both short and long term risk estimations, aiding their management of patients. It can also be inferred that a potential utility of cCoMET that is worthy of future study is the titrating of nurse:patient or nurse:physician ratios, or for electively (rather than emergently) moving a patient from the floor to an ICU if there is a cCoMET threshold that is crossed, or if the cCoMET increases by a certain amount over a defined period of time, even without any overt clinical deterioration that would otherwise trigger this. Risk scores with 'memory', that is the ability to reflect all of the risk faced by the patient throughout a hospital admission, are likely to reflect some degree of accumulating physiological insult, which would seem likely to add to our present ability to predict instantaneous risk. There is likely to have been a good physiological reason that the risk score was elevated, regardless of whether the event occurred or not. While cCoMET did not outperform instantaneous CoMET scores in the current work in terms of mortality prediction, future work is planned to assess its ability to predict other, less extreme end-points, including transfer to the ICU, sepsis, emergent intubation, blood transfusion etc, and to compare this with instantaneous CoMET scores. Whether the score with memory will outperform the instantaneous risk score in such a circumstance is, at present, unclear.

Relationship to the work of others
The prediction of deterioration in hospitalized patients has traditionally utilized track-and-trigger systems, including the National Early Warning Score (NEWS), the Modified Early Warning Score (MEWS), the Rothman Index (RI) and others (Rothman et  . These scoring systems have benefits including their relative ease of use and accessibility, yet suffer because they largely rely on vital signs and lab tests, which are variously intermittent, delayed, incorrect, unvalidated or indeed never taken (Grant 2018, Grant and Crimmons 2018, Grant 2019, Gerry et al 2020. Few take account of crucial information contained within continuous cardiorespiratory monitoring, like the CoMET score does. Furthermore these scores give an instantaneous idea of risk, without taking into account the events that have occurred earlier in a patients hospital stay. They have been shown to only marginally improve outcomes, while substantially increasing provider workloads, false alarms, alert fatigue and inappropriate resource utilization (Alam et al 2014). A contemporaneous review of digital tools for predicting patient deterioration can be found here (Mann et al 2021)-it is clear that when comparing risk prediction tools that 'models of greater complexityKhave superior performance in predicting deterioration' (Mann et al 2021). Potential benefits of the CoMET score are that it includes all available patient data, including continuous cardiorespiratory monitoring, and furthermore utilizes multiple unit-specific logistic regression-based models that have been trained on patients with specific causes of clinical deterioration identified by clinician review (Moss et al 2016), and in doing so it is capable of learning the signatures preceding the development of specific clinical illnesses (Blackwell et al 2020). However, the CoMET score is also an instantaneous score, based on vital signs, lab values and continuous cardiorespiratory monitoring parameters that are happening now (Ruminski et al 2019). The novelty of the cCoMET score explored here is that it includes the 'now', but adds this to all of the predicted risk that has occurred from admission to now, to add potentially important information to current risk prediction. None of the digital tools compared in the paper by Mann et al (Mann et al 2021). included the ability to reflect all of the physiological insults sustained by the patient since admission like the cCoMET can. While one can argue that the instantaneous CoMET score (and the factors that it takes into account) has some degree of short term memory ( i.e. lab values, vital signs and cardiorespiratory parameters do not vary with time at random, or stochastically, instead do bear some relation to the values that went before), it is not able to take account of all aspects of the risk since admission in as comprehensive a way that the cCoMET does. As such, the demonstrated relationships between cCoMET and mortality, baseline CoMET and gender are both novel and important, and emphasize the need for development of predictive analytics scores that incorporate 'memory', since, without this, important characteristics of the patient's condition may be overlooked, negatively impacting on the performance of predictive analytics tools.

Limitations
We omitted all patients with lengths of stay <72 h, since it was felt that these patients did not have enough time to develop a meaningful cCoMET score, though this may have affected the data presented. The numbers of surviving patients was substantially more than the patients who died, limiting the analysis in this group, and limiting the statistical significance of the findings.