Is smart water meter temporal resolution a limiting factor to residential water end-use classification? A quantitative experimental analysis

In this study, we use data from a single-point smart water metering system installed at a four-person, single-family, fully-detached residence in the Midwest United States, collecting 1-s resolution flow rate data over a 4-week study period from September 3 to October 1, 2021. Aggregate indoor household water use data were collected from a custom ally^® electromagnetic flow meter provided by Sensus, installed on the main water supply pipe into the residents' home. In addition to measuring flow rate (gal/min), the meter also sensed temperature (K) and pressure (psi) data at a 1-s resolution. Although these pressure and temperature data are useful to water system operations, they are not as valuable to demand disaggregation due the large impact the distribution system has on these variables. We validated this assumption through feature analysis based on correlation and data visualization (see figures S15–S18 in the supporting information (https://stacks.iop.org/ERIS/2/045004/mmedia)). Consequently, we focus our analysis on flow rate data. The water meter writes data to a computer running a script that parses the raw data into a suitable format for further analysis. A data acquisition system connected to the water meter parses the raw data into a timestamped comma separated value format for further analysis.

To examine the effects of data sampling temporal resolution on water end-use classification, we aggregated the 1-s resolution time series to resolutions of 5 s, 10 s, 30 s, and 1 min. The 1-min resolution has been recognized as a critical threshold for certain end-use data analytics in the electricity sector (Carrie Armel et al 2013), Similarly, a previous study based on analysis of synthetic data identified the same threshold as critical for end-use disaggregation in the water sector (Cominola et al 2018). Here, we test these findings with an experimental study based on real-world data and aim to identify a similar critical data sampling resolution threshold for water end-use classification in the residential sector. Meanwhile, since the study is only based on a four-person household, we preliminarily compare water consumption patterns with a larger study to ensure the study home is representative of larger scale behavioral patterns.

During the study period, the home occupants manually recorded a water diary of labeled end uses. In this study, six types of indoor water end uses contributed to the total household water demand: faucets, toilets, showers, refrigerator, dishwasher, and washing machine. We used a written water diary over the 4-week study period to collect ground truth end use data for model training and validation. The 4-week period was selected based on previous studies and practicality (Beal et al 2011, DeOreo et al 2016, Horsburgh et al 2017). The water diary included end-use labels, start time, and date that were completed by the household occupants. More details about the diary are reported in the supporting information, including the water diary template (figure S19) and an example of completed recordings (figure S20). This data collection included only factual data such that this work was determined not to meet the definition of human subjects research and, therefore, did not require Institutional Review Board (IRB) approval. Documentation of this IRB decision is available upon request. Limitations that naturally arose during the water diary process were as follows:

• Events that occupants would forget to fill in the diary could not be labeled after the disaggregation of the data.
• Start times listed in the diary would sometimes correspond to events that occurred 1–2 min before the reported time, implying that occupants would sometimes fill in the diary after the event.
• Specifically for faucet events, occupants mentioned occasionally leaving the faucet on to avoid reporting multiple events, resulting in long faucet durations that can represent atypical behavior in the model training process.
• The water diary was completed manually and was unreadable for some events.
• Some reported events did not match the information received from the meter.

In addition to these limitations, a power outage created a 2-day data gap in the smart water meter dataset, where the water diary was completed but measured water flow was missing.

The end-use disaggregation step separates concurrent water use events along with single events, that, aggregated on the axis of time, would give the original time series collected at the single-point residential water meter. While end-use disaggregation and end-use classification sometimes coalesce into one concept in literature, in this study we consider disaggregation as the first step of the end-use classification process (Nguyen et al 2018). Single events are defined as those that occur in isolation (e.g., dishwasher only), while combined or concurrent events have simultaneous occurrences of water usage (e.g., a toilet flush during a shower). A single i-th water use event E_i can be quantitatively characterized by a vector of features F_i , which include values of, e.g., start time, end time, average flow, and volume of that event. Separating and identifying overlapping, or concurrent, water use events is a significant challenge in residential water studies, and the accuracy of existing smart meter disaggregation models decreases substantially when these types of events are encountered (Cominola et al 2015). Concurrent events occur especially during longer duration events such as showers or outdoor irrigation. Thus, disaggregating concurrent events from one another by leveraging information on the characteristics of individual fixtures or by learning the patterns of individual end uses is essential for the purpose of creating a comprehensive water profile for the household.

In this analysis, we used the disaggregation model from Bethke et al (2021), developed based on the method created by Nguyen et al (2013) to separate concurrent events by calculating the vector gradients of the flow rate data to identify start and end times of overlapping events. Once we separated events with the above disaggregation approach, we manually labeled each appliance/fixture water event based on the water diary and examined the events further with the classification model described below. We repeated this process for every resampled resolution as well as the original 1 s data. At coarser resolutions, the performance of the disaggregation model deteriorated when detecting multiple short duration events happening simultaneously (e.g., hand washing), or short duration events happening on top of a long duration event. Therefore, in addition to naturally having fewer observations at coarser resolutions, the number of events that we were able to match with the diary also decreased (figure S21).

After disaggregating the original water use time series, we labeled each event by matching with the water diary. We then trained a random forest (RF) classifier to perform appliance/fixture end-uses classification, using the disaggregated water events resulting from the previous step of end-use disaggregation. The classification algorithm allocated each data point (i.e., a i-th water use event E_i ) in the dataset to one of the labeled classes, after training on tuples of events and associated features (E_i , F_i ).

RF models have been presented by Breiman (2001) as classical ensemble learning algorithms and have shown to be outstanding predictive models in classification tasks (Herrera et al 2010, James et al 2013). RFs are built using the same fundamental principles as decision trees and bagging (Bootstrap aggregation). Bagging introduces randomness into the tree building process by building many trees on random subsets of the training data with replacement; this process is also known as bootstrapping. Bagging then aggregates the predictions across all the trees, which reduces the variance of the overall procedure and improves predictive performance (Géron 2019). However, bagging trees could result in tree correlation that limits the effect of variance reduction. RFs help reduce variance by injecting more randomness into the training process (Hastie et al 2009). The RF algorithm is a bagging algorithm that draws random bootstrap samples from the training set. However, while bagging provides each tree with the full set of features, RFs have a random feature selection that makes trees more independent of each other, which often results in better variance-bias tradeoffs (table S1) (Hastie et al 2009, Probst et al 2019). In this study, the two features of average flow and duration were eventually selected to build the final models, based on the results of our feature importance analysis (figure S22). Therefore, the search for the split variable was limited to a random subset of the two chosen features. Feature importance was computed 2001 by evaluating which features contributed the most to the generalization power of the model (permutation-based feature importance; Breiman 2001).

To understand the mechanism used by RF models, it is necessary to understand the construction of classification decision trees. The goal of such a tree is to partition data into small and homogeneous groups. When travelling down the tree, data are split into possible responses called nodes that symbolize the branches of a tree. To perform each partitioning operation, a decision is based on an index (e.g., the Gini index), which allows RF models to partition the nodes of each tree into more homogenous groups that contain a larger proportion of one class in each subsequent node (Kuhn and Johnson 2013 ). The Gini index is calculated as in equation (1), where C is the total number of classes in the model and p_nk is the probability of the occurrence of class k at node n. In this study, six different classes were evaluated based on typical household end uses: faucets (f), toilets (t), showers (s), refrigerator (r), dishwasher (d), and washing machine (w). The sum of all probabilities at a certain node is equal to one (see equation (2)):

$$\begin{equation}G=\sum\limits _{k\varepsilon C}{p}_{nk}(1-{p}_{nk})\end{equation} \tag{ 1 }$$

$$\begin{equation}{p}_{\mathrm{f}}+{p}_{\mathrm{t}}+{p}_{\mathrm{s}}+{p}_{\mathrm{r}}+{p}_{\mathrm{d}}+{p}_{\mathrm{w}}=1.\end{equation} \tag{ 2 }$$

Other metrics similar to the Gini index can be used to build decision trees, including cross entropy and misclassification error. However, the Gini index is the most commonly used metric in the literature (James et al 2013). Moreover, according to Raileanu and Stoffel (2004), the frequency of disagreement of the Gini index and entropy is only 2% of all cases, yet entropy is generally slower to compute because it requires a logarithmic function. For the above reasons, we used the Gini index in this study.

Besides Gini, the RF algorithm involves several other hyperparameters that can be tuned to optimize model performance. While studies have shown that RF models are less sensitive towards tuning than other algorithms such as support-vector machines (Probst et al 2019), modest performance gains can still be valuable considering the limitations that naturally come with a small dataset. Using grid search, we gave ranges to RF hyperparameters to exhaustively try all possible combinations and select the best hyperparameter combination. Minimum sample at each leaf (2–5), minimum sample split (2, 5, 8, 12) number of sub-features (1, 2), maximum depth (3–10), and the number of trees (10, 20, 50, 100, 200) were initially given to the grid for hyperparameter tuning.

We consider two scenarios for calibration analysis of the classification model: the '1 s only calibration' (Scenario 1), and 'multi-resolution calibration' (Scenario 2).

'1 s only calibration' (Scenario 1): In this scenario, the RF model was trained only on the measured data at the 1 s resolution. Extended time series of 1 s resolution water use data are not usually available from utility records, but they can be collected in small-scale customized and experimental smart meter installations. With this scenario, we test whether investing efforts and resources in gathering a small model calibration dataset at sub-minute resolution is worth the potential gain of model disaggregation accuracy also at coarser resolutions. Our assumption behind this scenario is that the features of water use events can be more accurately learned from data collected at higher resolutions. In the 1 s trained RF model scenario, we split the labeled data into train (70% of the data) and test (30% of the data) datasets. The test set was used to assess the model performance on the 1 s trained data. Then, the entire resampled dataset from all other resolutions were separately used as test sets to compare the performance of the model on coarser resolutions.

'Multi-resolution calibration' (Scenario 2): In this scenario, we trained different RF models for each resolution (5 s, 10 s, 30 s, and 1 min) on their own dataset and compare their performances both with one another and with Scenario 1. In this scenario, we examine the value of retraining the RF model specifically for different temporal resolutions to quantify differences in performance between sampling resolution and, comparatively with Scenario 1, across different model training strategies. To retain the value of limited data and improve generalizability of the models, we implemented a k-fold cross-validation strategy (Hawkins et al 2003). We thus split the training set into k subsets, called folds, and then iteratively fit the model k times, each time training the data on k − 1 folds and validating on the remaining fold. In this study, we fit the model with k = 10. At the end of training, we averaged the performance across all validation folds as the final performance value for the model.

RF is a noise robust technique. However, when considering imbalanced problems, canonical machine learning algorithms generally tend to be biased towards the majority group. This behavior happens because such algorithms consider the number of objects in each group to be roughly similar (Krawczyk 2016, Ribeiro and Reynoso-Meza 2020). However, the minority class is often the most important when dealing with skewed distributions, and a performance metric should be chosen in a way to overcome such bias. While we do not directly balance the dataset used in this study because of its limited size, in this analysis we evaluate and compare the model performance using different formulations of the F1-score (FS). Specifically, we compare (i) micro-FS, which is a global metric attributing equal importance to each sample, thus giving emphasis on the most represented labels, (ii) macro-FS, which attributes equal importance to each class, and (iii) weighted FS, which computes the weighted average of the FS values obtained for individual classes. While using these metrics does not solve class imbalance, we examine different FS formulations to see whether our classifier gets biased towards well represented classes or not.

Micro-FS (usually referred to as simply FS) is a global performance metric that puts more emphasis on the most represented labels in the data set since it gives each sample the same importance. Labels that are underrepresented in the dataset may not be intended to influence the overall micro-FS heavily if the model is performing well on the other more common classes. Micro-FS (equation (3)) is defined as the harmonic mean of the precision (equation (4)) and recall (equation (5)):

$$\begin{equation}\text{Micro-FS}=2\times \frac{(\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})}{(\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})}\end{equation} \tag{ 3 }$$

$$\begin{equation} \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}\end{equation} \tag{ 4 }$$

$$\begin{equation} \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}\end{equation} \tag{ 5 }$$

where true positives (TP) are the number of correctly classified positive instances, false positives (FP) are the number of negative instances incorrectly classified as positive, and false negatives (FN) are the number of positive instances incorrectly classified as negative.

Macro-FS (short for macro-averaged F1 score) is used to assess the quality of classification in problems with multiple classes. The macro-FS gives the same importance to each class, with low values for models that only perform well on the common classes while performing poorly on the classes with less data. The macro-FS is defined as the mean of class-wise FS in equation (6):

$$\begin{equation}\text{Macro-FS}=\frac{1}{N}\sum\limits _{i=1}^{N}{\mathrm{F}\mathrm{S}}_{i}\end{equation} \tag{ 6 }$$

where i is the class index and N is the number of classes/labels.

The weighted-average FS (equation (7)) is calculated by taking the mean of all per-class F1 scores while considering the number of actual occurrences of each class in the dataset

$$\begin{equation}\text{Weighted-FS}=\frac{1}{H}\sum\limits _{i=1}^{N}\vert i\vert \times {\mathrm{F}\mathrm{S}}_{i}\end{equation} \tag{ 7 }$$

where i and N are as above, and H is the total number of aggregated elements across all classes.

The weighted-FS formulation modifies the macro-FS to account for class imbalance, while imbalance is not considered in micro-FS and macro-FS.

To make sure our study home could be a proper representative of a larger study scale, we initially visualized the time-of-day and day-of-week distribution of three major classes of events (shower, washing machine, and dishwasher) to find regular patterns of consumption similar to those displayed in larger datasets. Much of the occupants' water consumption occurs during typical weekday mornings and evenings. Figure 1(a) depicts shower end use distribution throughout the week and time of the day in our study home. The results show that showers have a more sporadic pattern of use on weekends while during weekdays most of them occur during regular morning and evening peak hours. These behavioral patterns align with the time-of-day and day-of-week distribution of showers reported in an analysis of water end use data gathered for 762 US households (Cominola et al 2020), shown in figure 1(b). The time-of-day and day-of-week distribution figures for the washing machine (figure S1) and dishwasher (figure S2) also show similar results. Washing machine events are observed mostly during weekends with a wide distribution throughout time of the day, while dishwashers are typically used throughout the week, either mornings or evenings. Comparison of the results show similar patterns between our study home and the larger study of US households used in Cominola et al (2020), demonstrating the potentially transferrable nature of our study home results. Similar widespread end-use data would help water planners and managers understand water consumption patterns, consumer behavior, and temporal variability. Decreasing consumption during peak time on a widespread scale could contribute to lowering overall peak demand for the local utility and reduce pressure on existing water infrastructure.

Zoom In Zoom Out Reset image size

The overall RF model performance across different resolutions in both calibration scenarios is presented in figure 2. Grey lines represent Scenario 1 (1 s only calibration) and blue lines represent Scenario 2 (multi-resolution calibration). The micro-FS, weighted-FS, and macro-FS are represented with dashed, solid, and dotted lines, respectively. We observe that Scenario 2 gives higher performance across different temporal resolutions regardless of the performance metric. For both 1 s and 5 s resolutions, the micro-FS and weighted-FS values are similar: 0.91 and 0.89 for the micro- and weighted-FSs, respectively, at the 1 s resolution, and 0.87 and 0.85 for the micro- and weighted-FSs, respectively, at the 5 s resolution. The macro-FS generally shows the lowest values for all resolutions for both scenarios. We observe a mild decrease in performance metrics with coarser temporal resolutions in Scenario 2, while performance metrics decrease notably for resolutions coarser than 5 s in Scenario 1, dropping as low as 0.2 for the 1 min resolution.

Zoom In Zoom Out Reset image size

Overall, our results indicate that the RF models learned end use event features better when trained at the same data sampling resolution that they are tasked to use to classify unseen events, provided that a training dataset with labelled events at that resolution is available. If classification models are trained for application on data measured at the same resolution (Scenario 2), those models can perform at an acceptable level of performance even at coarser resolutions, depending on the relative importance of different end use classes. This observation has important implications related to the tradeoffs between fine-resolution data collection and increased data analytics needs. For instance, if a utility wants an estimate of water consumption by the main indoor water uses in households (e.g., toilets and showers), the 1 min resolution model still provides an acceptable performance (weighted-FS equal to 0.73). This performance is lower than the FS of 0.89 obtained for the 1 s resolution model, but this loss in model accuracy is balanced by the benefit of gathering, storing, and analyzing fewer data observations at the coarser temporal resolution. Conversely, if detailed information on all end uses is required, only the 1 s and 5 s resolutions provide high performance predictions on all end use classes; for less represented end uses, performance is compromised at coarser resolutions.

Our detailed RF model testing results are presented in figure 3, where the predicted classes (right) are compared to the actual classes (left). Figure 3(a) represents the entire 1 s resolution set of events, while figure 3(b) zooms in on shorter duration events for clarity. The average flow rate (gal/min) and duration (s) were used as identifying features for our model. Of the total 654 events labeled, we used 196 events as a test set. The model predicts the test set with an accuracy of 92% and a weighted-FS of 0.89, which is noteworthy given the fact that the training dataset had limited observations in some classes such as dishwasher and washing machine. The model correctly predicts 179 events out of 196 total events of the test set.

Zoom In Zoom Out Reset image size

Yet, the high model performance in all classes might overrepresent the overall ability of our RF models to classify unseen end use events. Our results might imply that, due to the fine temporal resolution of the data, the model discerns the constant range of duration and average flow of those end uses with automatic water consumption cycles (e.g., washing machine, dishwasher) and detects them correctly. However, since our study represents a single household only, the model might be overfitting on data from automatic appliances due to the invariance of duration and flow in these specific automatic appliances, thus results on these specific end uses may not be generalizable.

It is important to note that, while individual toilet uses are typically homogeneous in terms of water consumption volume and duration, even considering dual-flush systems, the combination of toilet and bathroom faucet uses are difficult to detect and disaggregate because such uses are often almost simultaneous (e.g., use of toilet and consequent handwashing in a same minute). Although temporal resolutions finer than 1 min reduce disaggregation errors (Mazzoni et al 2021), we were not able to disaggregate all toilet events followed by faucets. Rather, we labeled the mentioned events as toilets since we attributed the subsequent faucet use due to the toilet use. As a result, toilets have a wider range of flow and duration, as shown in figure 3.

Figure 4 shows the classification results for Scenario 1 (1 s only calibration) applied to the resampled 5 s (figure 4(a)) and 1 min resolutions (figure 4(b)), respectively, selected as examples at the two extremes of the considered spectrum of data resolutions. Here, we report our analysis results in both US customary units (gal/min) and SI units (L min⁻¹). In comparing different temporal resolutions, coarser resolutions tend to compress data points on the vertical axis (i.e., decrease average event flow) and extend their range on the horizontal axis (i.e., increase event duration) due to temporal averaging. For example, toilet events that originally ranged from 1.7–3 gal/min (6.4–11.4 L min⁻¹) average flow in the measured 1 s resolution tend to shift to 1–2.5 gal/min (3.8–9.5 L min⁻¹) in the 5 s resolution and decrease further to 0.4–0.8 gal/min (1.5–3 L min⁻¹) in the 1 min resolution. The duration of events increases with coarser temporal resolution to an extent that the total volume of events is the same as to the volume in the original 1 s resolution measurements. The mentioned shifts in values of end-use features leads to decreased model performance with coarser temporal resolutions, up to a point where, as shown in figure 4(b), the model can no longer detect any toilet events. The model still correctly predicts showers and a few washing machine events at the 1 min resolution; however, the model application to the 1 min data predicts most other end uses as a faucet under Scenario 1. Similar Scenario 1 classification results for the 10 and 30 s resolutions are presented in the supporting information (figures S3 and S4) along with the zoomed in figures of the 5 s and 1 min resolutions for a detailed view (figures S5–S8).

Zoom In Zoom Out Reset image size

Figure 5 shows the confusion matrices of water end use classification across the events of our four-person study household for Scenario 1. Faucets (f) account for the most frequent end uses, followed by toilets (t). The matrices show the total number of events labeled for each resolution, the actual classes, and the predicted classes by the model. The results for the 5 s resolution show that of 382 total events that we were able to match with the water diary, 324 events were classified correctly (figure 5(a)). The main misclassifications were in predicting 14 actual toilet end uses as faucets and four actual faucet end uses as toilets. This misdetection mostly occurs for data that fall in the area with average flows of 1–1.5 gal/min (3.8–5.7 L min⁻¹) and durations of 25–50 s (see figure S5 in the supporting information). For the 1 min resolution (figure 5(b)), only 187 events had corresponding end uses in the water diary due to disaggregation errors where the model was not able to separate concurrent events because of loss of information that naturally accompanies coarser resolutions. Out of these 187 events, 92 were classified correctly. The classification model predicts 135 events as faucets. While only 73 of these events are actually faucets, they still account for 40% of the prediction accuracy, motivating consideration of F1-score metrics due to the imbalanced dataset.

Zoom In Zoom Out Reset image size

Figure 6 shows the confusion matrices of end use water consumption for Scenario 2, the multi-resolution calibration, for the 5 s (figure 6(a)) and 1 min (figure 6(b)) resolutions. Compared to Scenario 1 (figure 5(a)), the 5 s resolution performance in Scenario 2 has either slightly improved or stayed the same, with the exception of refrigerator events (r). The 1 s resolution trained model in Scenario 1 had a better performance in predicting 5 s resolution refrigerator events. The prediction of toilets improved notably from 43 to 51 out of 57 events. The main misclassifications were in predicting 5 actual toilet end uses as faucets and 6 actual faucet end uses as toilets. This misdetection mostly occurs for data that fall in the area with average flows of 1–1.5 gal/min (3.8–5.7 L min⁻¹) and durations of 25–50 s (see figure S5 in the supporting information). Overall, the 5 s resolution has a high performance under both scenarios, with performance metrics slightly less than those of the 1 s resolution (as shown in figure 2). In the 1 min resolution, our model correctly predicts 139 of 187 labeled events, having the highest prediction accuracy in washing machine (100%), faucet (90%), and shower (88%) events. These results imply that if any of the aforementioned end uses are of importance, the 1 min resolution can still be informative.

Zoom In Zoom Out Reset image size

With further investigation of the diagonals of the confusion matrices, we see how figure 6(b) has ameliorated in comparison to figure 5(b), increasing correct predictions from 93 to 139. The 1 min resolution model is still not able to discern refrigerator faucet events (r) from tap faucet events (f); however, this misclassification is not a critical issue since the refrigerator faucet is a faucet in nature. A noteworthy observation is that although the 1 min resolution model under Scenario 2 incorrectly classifies one shower and one actual faucet event as a washing machine (i.e., false positive, FP in equation (4)), it does not label any other actual washing machine event as other events (i.e., false negative, FN in equation (5)), which leads to a higher recall in this specific class (100%) than in the 1 s resolution model (86%) (refer to figure S12) and the 5 s resolution model (95%), with the tradeoff of lower precision (88% versus 95% in the 1 min and 5 s resolutions, respectively). Additional confusion matrices at other temporal resolutions are available in figures S9–S14 of the supporting information.

In general, misclassifications do not cause significant degradation in predicting total water consumption if they are infrequent and roughly symmetric across the diagonal (Srinivasan et al 2011). For example, if toilet events are misclassified as faucet events while the same (or nearly the same) number of faucet events are misclassified as toilet events, these misclassifications can cancel out in terms of the accurate total number of events for those classes.