Quantifying the accuracy of inter-beat intervals acquired from consumer-grade photoplethysmography wristbands using an electrocardiogram-aided information-based similarity approach

A total of 30 healthy right-handed college student volunteers (19 males and 11 females) with age ranging from 20 to 26 years (21.5 ± 2.1 years) from the Southeast University in China took part in this study. All participants were informed of the experimental protocol and matters needing attention, then signed the written informed consent prior to participating in the study. This study was approved by the Independent Ethics Committee for clinical research of Zhongda Hospital, affiliated to Southeast University (No. 2019ZDSYLL073-P01). None of the participants had any history of neurological or psychological disorders, and they all had normal vision or were corrected to normal vision (self-report). Three subjects were excluded due to device problems and data recording interruption.

The 30 healthy participants concurrently wore two different level wristbands on their non-dominant wrists and a gold-standard ECG measurement device following the manufacturer's instructions.

The two wristbands are both wireless sensors that can continuously record PPG signals and output IBI time series. The E4 wristband (Empatica Inc., Milan, Italy) is specifically designed for both research and commercial purposes, and overcomes the limitations of most commercial devices. The sampling frequency for the E4 PPG signal is 64 Hz. To validate the reliability of commonly used consumer-grade commercial PPG wristbands, the Honor band 5 (Huawei Technologies Co., Ltd, Shenzhen, China) was worn on the same wrist of each participant, and the wearing location of the two wristbands was randomized. In this study, the PPG signal sampling frequency for Honor band 5 was 100 Hz. Honor band 5 is a popular commercialized smart band and it is much cheaper than E4. The raw PPG data were directly exported from the E4 band via a client mobile application. The Honor band 5 is commercially used and restricted by proprietary confidentiality agreements; thus, the raw PPG data were exported from the device using a specific software development kit after signing an agreement with Huawei.

ECG recordings were collected by a US Food and Drug Administration-approved ambulatory electrocardiogram monitor (DynaDx Corporation, Mountain View, CA, USA) with a computer-based data acquisition system. The ECG recording equipment was a single-lead Holter device that can continuously record ECG for over 24 h. The ECG electrodes were placed on the subject's chest. The sampling frequency of the ECG signals was set to 512 Hz.

This study focused on the accuracy of IBI and HRV under different psychophysiological conditions. Outdoor conditions were not included because the accuracy of PPG during exercise is too poor to obtain reliable IBI time series when using consumer-grade wristbands. The data collection protocol is shown in table 1, and includes nine subsequent conditions: paced breathing at 0.17 Hz and 0.1 Hz, resting state, and three mental stress tasks followed by a resting state after each stress task.

Participants were asked to breathe following a respiratory pacer set at 0.17 Hz (6 s/breathing period) firstly, and then at 0.1 Hz (10 s/breathing period). Each session was conducted for 5 min. Paced breathing was included to maximize the respiratory sinus arrhythmia, which is associated with increased vagally mediated HRV measures (Vaschillo et al 2006).

2.4.1. Seated resting

2.4.2. Modified Montreal Imaging Stress Tasks (mental stress tasks)

The Montreal Imaging Stress Task (MIST) is a well-validated task for healthy adults and clinical populations (Castro et al 2015, Ming et al 2017, Noack et al 2019). Considering our current understanding of neural stress circuitry and the results of the most relevant MIST studies (Hermans et al 2014, Hariri 2015, Lago et al 2017, Velozo et al 2021), we hypothesize that the MIST will elicit increased self-reported stress ratings, HR, and decreased HF-HRV.

In this study, a modified MIST task was applied. Participants were asked to solve challenging arithmetic operations under extreme time pressure while feedback on their average performance relative to the performance of alleged other participants is given (figure 1). The operations are between three numbers (limited to two digits), in which the first two numbers are multiplied or divided, followed by the addition or subtraction of a third number. In order to maintain the stress state of the participants, the time limit was reduced by 10% for three consecutive correct answers, and to avoid participants giving up on difficult tasks, the time limit was increased by 10% for three consecutive wrong answers.

Zoom In Zoom Out Reset image size

The five-order band-pass Butterworth filter and zero-phase shift filter with a cut-off frequency of 0.3 Hz to 8 Hz were first used to eliminate the baseline drift, power frequency interference, and high-frequency noise. Then, an adaptive threshold was used to detect the peak point of each pulse cycle and to extract the IBI series. The Matlab function 'findpeaks' was applied (Christensen et al 2015, Lockmann et al 2016, Stobart et al 2018, Babola et al 2021, Bae et al 2021), where the first parameter 'minpeakheight = 0' (because the DC component was filtered during preprocessing) and the second parameter 'minpeakdistance = 0.4*fs' (fs is the sample frequency of the PPG signal, and 0.4 s was set as the minimum value of one cardiac cycle).

Figure 2 is an example of a PPG peak detection result. The PPG signal was recorded using Honor band 5 during task 1 (paced breathing, 0.17 Hz). The IBI can be obtained from the intervals between the detected pulse wave peaks, known as PP intervals (PPIs).

Zoom In Zoom Out Reset image size

The gold-standard IBI time series was obtained from the simultaneously recorded ECG signals. The ECG signals were preprocessed as follows: a third-order band-pass Butterworth filter and zero-phase shift filter with a cut-off frequency of 2 Hz to 30 Hz were first used to eliminate the baseline drift, power frequency interference, and high-frequency noise, which makes the QRS complex (including the Q wave, R wave, and S wave) more prominent, and avoids changing the position of R peaks.

At present, plenty of R-peak detection methods have been proposed in ECG signal analysis. However, when applying these detectors on ECG signals collected in daily life, especially via wearable single-lead ECG devices, the R-peak detection accuracies are usually unsatisfactory. As the gold standard, we must guarantee the accuracy of R-peak detection. Thus, a method for extracting high-quality RR intervals (RRI) proposed in our previous study (Cui et al 2020, Xue et al 2020) was applied, which combines five commonly used R-peak detection methods, namely Pan–Tompkins (Pan and Tompkins 1985, Hamilton and Tompkins 1986), jqrs (Johnson et al 2015), mteo (Solnik et al 2008), nqrs (Li and Yang 2013), and sixth power (Dohare et al 2014).

The gold-standard IBI time series can be obtained from the detected RRIs. Both RRIs and PPIs are IBI time series.

3.3.1. Accuracy quantification algorithm based on IBS approach

The accuracy of PPG-derived HR is commonly determined either by analyzing PPG pulse waveform characteristics or by establishing the correlation between the separately calculated HRV indices from PPG and ECG. Both methods rely on qualitative assessments.

In this study, the similarity index of dynamical changing patterns between IBI time series simultaneously recorded using PPG wristbands and the ECG device was calculated by means of symbolic coding and ranking statistics to quantify the accuracy of PPG-derived IBI time series.

The IBS approach was proposed to measure the distance or dissimilarity between two symbolic sequences based on measuring differences in the occurrence of repetitive patterns between two symbolic sequences (Yang et al 2003, Goldberger and Peng 2005, Wang et al 2016). The IBS algorithm is based on symbolic coding and ranking statistics, and thus it is not required to align the timelines of PPI and RRI precisely. In this study, the accuracy quantification algorithm for wristband-measured PPI time series was proposed as follows.

• (1)  
  Map the IBI time series (RRI and PPI) to binary symbolic sequences, where an increase in the IBI is represented by '1' and no change or a decrease in the IBI is represented by '0'.
• (2)  
  Map m + 1 successive IBI intervals to a binary sequence of length m, called an m-bit 'word'. Each m-bit word, therefore, represents a unique pattern of fluctuations in a given IBI time series.
• (3)  
  By shifting one data point at a time, the algorithm produces a collection of m-bit words over the whole time series (a total of 2^m possible words).

Different patterns of dynamics thus produce different distributions of these m-bit words. Figure 3 illustrates this mapping procedure using five-bit words (m = 5) from a part of the IBI time series. For m = 5, there are a total of 32 (= 2⁵) possible words. The first binary word (10010) shown in figure 3 is equivalent to the decimal number of 18 (1 × 24 + 1 × 21 = 18), so that (00100) and (01001) are termed 4 and 9, respectively.

• (4)  
  These m-bit words are then sorted according to their frequency of occurrence. The rank-frequency of any given m-bit word may differ between the two sequences mapped from two IBI time series.
• (5)  
  Plot the rank order of each m-bit word in the PPI symbolic sequence against its rank order in the RRI symbolic sequence. Figure 4 is a representative result of rank order comparison (the original time series were from subject #3 during resting state), where each data point on the graph represents a binary word (m = 5) with its rank on the RRI symbolic sequence plotted on the horizontal axis against that on the PPI symbolic sequence on the vertical axis. If two symbolic sequences are similar in their rank order, the data points will be located near the diagonal line (figure 4(a)), and vice versa (figure 4(b)).
• (6)  
  Let D_r (ψ1, ψ2) denote a distance value between 0 and 1 of two symbolic sequences, s_k denote an m-bit word, and L denote the number of unique m-bit words. Let R and p denote the word's rank and probability, respectively. Let F denote the weight of the word, where F is computed using Shannon's entropy and normalized with the normalization factor Z. The degree of distance between the two symbolic sequences can be defined as (Yang et al 2003, Goldberger and Peng 2005, Wang et al 2016):

  $$\begin{eqnarray}&&{D}_{r}\left({{\psi }}_{1},\,{{\psi }}_{2}\right)=\displaystyle \frac{1}{L}\displaystyle \sum _{K=1}^{L}\left|{R}_{1}\left({S}_{k}\right)-{R}_{2}\left({S}_{k}\right)\right|F\left({S}_{k}\right)\end{eqnarray} \tag{ 1 }$$

  $$\begin{eqnarray}&&\left({S}_{k}\right)=\left[-{p}_{1}\left({S}_{k}\right)\,\mathrm{log}\,{p}_{1}\left({S}_{k}\right)-{p}_{2}\left({S}_{k}\right)\,\mathrm{log}\,{p}_{2}\,\left({S}_{k}\right)\right]/Z\end{eqnarray} \tag{ 2 }$$

  $$\begin{eqnarray}&&Z=\displaystyle \sum _{K=1}^{L}\left[-{p}_{1}\left({S}_{k}\right)\,\mathrm{log}\,{p}_{1}\,\left({S}_{k}\right)-{p}_{2}\left({S}_{k}\right)\,\mathrm{log}\,{{\rm{p}}}_{2}\left({S}_{k}\right)\right]\end{eqnarray} \tag{ 3 }$$

  The sum is divided by the value L to keep D_r (ψ1, ψ2) in the range [0, 1]. A smaller distance value corresponds to a higher degree of similarity between PPI and RRI, and thus indicates high accuracy of PPI.
• (7)  
  Calculate the IBS distance value D1 between PPI time series and RRI time series. Then calculate the IBS distance between randomly shuffled RRI time series and the original RRI time series, repeat it 3–5 times and get the average distance value D2. The worst case is when D1 is close to D2.
• (8)  
  Add white noise of different power W_i to the original whole RRI time series (for example, let the SNR change from −15 dB to 75 dB, step = 1), and separately calculate the corresponding distance from the RRI time series. Repeat this process 3–5 times, and the average distance value is denoted as D_i , and then the relationship between the distance index D_i and SNR is obtained as shown in figure 5, where the blue line is a three-order Gaussian fitting curve, and the Spearman rank correlation was found between the SNR and IBS distance index D_i (r = −0.9251, P < 0.001).
• (9)  
  Record the added white noise power W1 when D_i is equal to or closest to D1, and thus estimate the white noise power contained in the PPI time series. For example, the IBS distance for the time series in figure 4(a) is D1 = 0.0871, and the corresponding SNR = 37.11 can be obtained using binary search. 'SNR = 37.11' means the ratio of signal-to-noise power is about 5000:1, which indicates that the PPI accuracy is high in this case.
• (10)  
  Reject 'low-accuracy' data segments. According to the data recording conditions, duration, analytic purpose, etc, define an SNR threshold to reject the data segments whose SNR is smaller than this threshold. For example, in the Results section, SNR < 25 (i.e. IBS > 0.21) was applied for a representative analysis.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

3.3.2. Tuning the parameter m

When the data length is short and m is too large, the changing patterns in the data only account for a small part of the 2^m patterns, which will reduce the accuracy of the distance index; when two long data segments have similar patterns and m is small, the distance index can only be roughly measured, and the accuracy assessment will be not reliable. Thus, in this study, the best value of m is chosen by comparing the IBS distance index of different m. According to table 1, the corresponding time durations of data collection tasks were different (i.e. 300 s, 500 s, and 360 s), so the data lengths were different. To calculate the IBS distance index corresponding to different m values at different data lengths, representatively, we compared the IBS distance index between the PPI and RRI during the tasks with different data lengths; e.g. 'Paced breathing (0.17 Hz)' (PB-300s), 'Seated resting state 1' (RS-500s), and 'Stress task 1' (ST-360s). Table 2 presents the results when m = 3, 4, 5 for each participant. In the case m = 3, there are five subjects whose IBS distance values are 0, which indicates that m is too small to contain enough changing patterns (2³ = 8), resulting in the IBS distance values being inaccurate.

Table 2. Representative results of the IBS distance index corresponding to different m at different data lengths (PB: paced breathing; RS: resting state; ST: stress task).

m	3			4			5
IBS subject #	PB-300s	RS-500s	ST-360s	PB-300s	RS-500s	ST-360s	PB-300s	RS-500s	ST-360s
1	0.071	0.250	0.187	0.014	0.114	0.057	0.044	0.098	0.048
2	0.111	0.225	0.113	0.091	0.131	0.142	0.102	0.153	0.171
3	0.153	0.273	0.240	0.154	0.216	0.252	0.080	0.289	0.243
4	0.000	0.020	0.006	0.058	0.078	0.057	0.085	0.110	0.115
5	0.215	0.199	0.048	0.099	0.107	0.024	0.034	0.123	0.055
6	0.066	0.150	0.051	0.022	0.070	0.114	0.054	0.106	0.103
7	0.032	0.343	0.298	0.118	0.318	0.276	0.121	0.287	0.282
8	0.154	0.370	0.022	0.167	0.397	0.073	0.189	0.354	0.070
9	0.000	0.221	0.412	0.021	0.166	0.295	0.065	0.195	0.283
10	0.159	0.215	0.176	0.140	0.097	0.051	0.161	0.085	0.088
12	0.093	0.211	0.273	0.136	0.290	0.140	0.188	0.211	0.138
13	0.153	0.205	0.069	0.170	0.258	0.161	0.142	0.287	0.080
14	0.080	0.075	0.237	0.046	0.130	0.106	0.046	0.085	0.105
15	0.185	0.188	0.349	0.071	0.230	0.330	0.063	0.186	0.383
16	0.144	0.206	0.379	0.176	0.326	0.396	0.156	0.240	0.333
17	0.000	0.176	0.199	0.087	0.111	0.144	0.087	0.102	0.158
18	0.351	0.239	0.083	0.228	0.318	0.108	0.235	0.259	0.127
19	0.194	0.102	0.290	0.136	0.236	0.344	0.154	0.205	0.278
21	0.000	0.229	0.248	0.078	0.109	0.243	0.156	0.126	0.201
23	0.032	0.175	0.262	0.046	0.128	0.207	0.072	0.135	0.181
24	0.332	0.340	0.365	0.205	0.401	0.358	0.277	0.382	0.368
25	0.200	0.200	0.099	0.121	0.130	0.052	0.109	0.075	0.100
26	0.088	0.149	0.138	0.044	0.244	0.114	0.082	0.254	0.099
27	0.000	0.304	0.345	0.133	0.321	0.302	0.068	0.253	0.249
28	0.151	0.030	0.038	0.094	0.073	0.041	0.096	0.136	0.046
29	0.128	0.000	0.216	0.100	0.038	0.259	0.099	0.066	0.294
30	0.034	0.057	0.155	0.080	0.145	0.204	0.066	0.153	0.143

Zoom In Zoom Out Reset image size

Taking the data segment 'PB-300s' from subject #4 (the IBS distance is 0 in table 2) as an example, the data points for both PPI and RRI are 292. Tables 3 and 4 display the word pattern and the occurrence frequency and ranking for each pattern separately for the cases m = 3 and m = 4. In the case m = 3, although the occurrence frequencies of eight patterns are different, their rankings are the same, resulting in an IBS distance value of 0. In the case m = 4, both the occurrence frequencies and their rankings are slightly different. Two patterns (0101 and 1011) did not appear for the RRI time series, and more patterns might be missing in the case m = 5. Considering the data recording durations (300 s, 360 s, and 500 s), we chose m = 4 in this study.

Table 3. Word pattern, occurrence frequency, and ranking (case m = 3).

	RRI time series		PPI time series
Word pattern	Occurrence frequency	Ranking	Occurrence frequency	Ranking
000	0.201	1	0.211	1
001	0.163	2	0.156	2
010	0.021	7	0.055	7
011	0.156	4	0.142	4
100	0.159	3	0.152	3
101	0.017	8	0.042	8
110	0.156	5	0.142	5
111	0.128	6	0.100	6

Table 4. Word pattern, occurrence frequency, and ranking (case m = 4).

	RRI time series		PPI time series
Word pattern	Occurrence frequency	Ranking	Occurrence frequency	Ranking
0000	0.080	7	0.094	5
0001	0.122	3	0.118	3
0010	0.007	14	0.024	13
0011	0.156	1	0.132	1
0100	0.021	11	0.031	11
0101	—	—	0.024	14
0110	0.063	8	0.076	6
0111	0.094	5	0.066	7
1000	0.118	4	0.118	4
1001	0.042	9	0.035	9
1010	0.014	13	0.031	12
1011	—	—	0.010	16
1100	0.139	2	0.122	2
1101	0.017	12	0.017	15
1110	0.094	6	0.066	8
1111	0.035	10	0.035	10

The IBS distance values were calculated separately between the gold-standard RRI time series and 1) shuffled RRI, 2) E4 band recorded PPI time series, and 3) Honor band 5 recorded PPI time series. The SNR of the PPI time series recorded during each task was estimated using the binary search method and the three-order Gaussian fitting curve, as shown in figure 5. The results are shown in table 5. According to figure 5 and table 5, the IBS values are negatively associated with the SNR; thus, a bigger IBS distance indicates a smaller SNR and low accuracy. The IBS distances of the PPI time series are much smaller than those of shuffled RRI that displayed the worst-case accuracy. In addition, for the PPI time series of the two wristbands, the IBS distances were in the order mental stress tasks > seated resting > paced breathing. During the mental stress tasks the participants made more hand movements, which could lead to more noise. During deep breathing, they were more focused on their breathing, and their PPI and RRI would vary with the breathing cycles; thus, the SNR could be bigger.

The correlation analysis between the HRV indices separately calculated using PPI and RRI is one of the most commonly used PPG accuracy assessment methods, and is thus widely applied to validate the reliability of IBI time series-based psychophysiological measures. Hence, an HRV index correlation analysis was performed in this study to evaluate the accuracy of PPI-calculated HRV indices.

The Pearson correlation was analyzed between the same HRV indices of the same task. The HRV indices applied in this study included meanHR (mean heart rate), SDNN (standard deviation of IBI series), RMSSD (root mean square of successive IBI differences), pNN50 (percentage of successive IBI that differ by more than 50 ms), SD1/SD2 (ratio of Poincaré plot standard deviation perpendicular to the line of identity to standard deviation along the line of identity), LFP (percentage of the low-frequency band (0.04–0.15 Hz) power), HFP (percentage of the high-frequency band (0.15–0.4 Hz) power), LF/HF (ratio of LFP to HFP).

The color map in figure 6 presents the Pearson correlation coefficients between PPI-derived and RRI-derived HRV indices before and after rejecting the low-SNR segments for each HRV index under nine data recording conditions. Representatively, the 'low-accuracy' segments are defined as SNR < 25 (IBS > 0.21) in this section, and other SNR thresholds are defined case by case. The HRV correlation coefficients in table 6 demonstrate that the PPI accuracy was not satisfactory compared with the gold-standard RRI time series for both the E4 band and Honor band 5, which may decrease the accuracy of PPI-based mental state classification. The results in figure 6 and table 6 indicate that the most direct method to improve the reliability is to reject PPI data segments with low SNR. After rejection, the averaged correlation coefficients for all HRV indices increased from 0.542 ± 0.235 to 0.922 ± 0.120 for E4 and from 0.596 ± 0.227 to 0.859 ± 0.145 for Honor 5. In this case, the accuracy of PPI-based mental state classification would be close to the RRI.

Zoom In Zoom Out Reset image size

As shown in figure 6 and table 6, after rejecting low-SNR segments, the accuracies of most HRV indices showed great improvements, with the exception that the Pearson correlation coefficients between the mean HR obtained using E4 and the gold-standard ECG device are almost the same (0.927 ± 0.075 vs 0.976 ± 0.019), indicating that E4 had more reliable mean HR values. However, the original (before rejection) Pearson correlation coefficients of other HRV indices measured using Honor 5 were higher than E4.

The HRV indices calculated via PPI time series were used in mental state classification. A total of 242 data segments (27 subjects * 9 conditions = 243, and the ninth condition of subject #9 was not completed) were included for each wristband, which were divided into four classes (four mental stress states): paced breathing–0.17 Hz, paced breathing–0.1 Hz, resting state, and stress state.

The HRV indices (i.e. SDNN, LFP, HFP, LF/HF, and SD1/SD2) were fed into an SVM model for mental state classification. In this study, a library for SVMs (LIBSVM) (Chang and Lin 2011) is implemented for the SVM classifier with the radial basis kernel function. Five-fold cross-validation was performed ten times to evaluate the model's predictive performance. The data segments were divided into five parts. One of the five parts was used as a testing set and the remaining four parts were used as a training set. To avoid information leakage, segments from the same subject were divided into either a training set or testing set. Before training, the features were normalized using the function scaleforSVM, which processes training and testing sets by mapping row minimum and maximum values to [−1, 1]. Then, the radial basis function kernel was selected, and the optimal parameters C and gamma were found using the function SVMcgForClass.

The averaged precision, sensitivity (recall), specificity, and overall accuracy were used to judge the performance of four-class mental stress state classification. The results are presented in table 7. To improve the reliability of classification, the PPI data segments with IBS distance > 0.21 (m = 4, SNR = 25) were rejected, which caused the precision, sensitivity, and specificity to increase from 76.18%, 75.44%, and 92.73% to 84.29%, 76.58%, and 93.37% for E4; and increase from 77.45%, 77.50%, and 93.00% to 79.99%, 79.75%, and 94.28% for Honor 5. The overall accuracy of the four-class mental stress state classification was increased from 77.3% to 81.9% for E4 and from 79.3% to 83.3% for Honor 5.

The IBI time series and its variability are playing a more and more significant role both in clinical and daily healthcare services. PPI-based psychophysiological applications have been implemented in consumer-grade smart wristbands; however, the reliability of this measurement has not been fully validated. Besides the mean HR, the underlying dynamical structure patterns of PPI time series are more influenced during different mental stress states. The most accurate assessment should be the comparison with standard ECG-calculated IBI signals; however, very few studies have assessed wristband-recorded IBI accuracy in terms of absolute agreement with gold-standard ECG recording systems, because the timelines of the two systems are difficult to align precisely. Hence, the quantitative assessment is difficult to conduct. To solve this problem, this study proposed a novel IBS distance indicator for quantitatively assessing the accuracy of PPI time series according to the information of underlying dynamical structures, and the SNRs of PPI time series measured using different level PPG wristbands were obtained. Compared with the gold-standard ECG-measured HRV, the accuracies of PPG-measured HRV indices for both E4 and Honor 5 were not satisfactory, i.e. the HRV correlation coefficients were no more than 0.6; however, the figure reached around 0.9 when rejecting low-SNR PPI segments, indicating that the PPG-measured HRV indices were strongly influenced by noise, and that the accuracy could be improved via appropriate de-noising methods. The PPI-HRV-based mental state classification accuracies were increased by about 4% by rejecting the low-accuracy data segments by a designated SNR threshold.

A four-class mental stress state induction experiment, where subjects were wearing two popular but different level PPG wristbands (E4 and Honor 5) and an ECG device, was conducted with the aim of validating the accuracy of PPI time series by comparing the HRV correlations (between PPI-HRV and RRI-HRV) and mental state classification accuracy before and after applying the proposed method to reject low SNR segments. The IBS distance and SNR (table 5), HRV correlations (figure 6 and tables 6), and four-class mental state classification (table 7) all proved that the accuracy of PPI time series could be validated and improved using the proposed method. The correlations between HRV indices were greatly improved (table 6), and the four-class mental state classification accuracy was increased by about 4%, indicating that the PPI noise ratio has a great influence on the accuracy of HRV indices. Pursuing the highest mental state classification accuracy is not the goal of this study, and the accuracy could definitely be increased when more sophisticated machine learning models are applied in future.

A few preliminary studies have assessed the accuracy of the E4 wristband. For example, one study focused on stress and emotion recognition (Ragot et al 2018). Two other studies focused on the PPG signals and suggested a satisfactory data quality (McCarthy et al 2016) and acceptable measurement error in mean HR and some HRV measures (Pietilä et al 2017). A few studies have assessed the accuracy of the E4 in measuring mean HR and HRV indices, and to evaluate how accuracy varies as a function of the condition of measurement (Menghini et al 2019, Schuurmans et al 2020, Stuyck et al 2022). E4 was also used in a multimodal assessment of adult attachment (Parra et al 2017) and in urban stress research (Chrisinger and King 2018).

Honor 5 is one of the most popular worldwide commercialized consumer-grade smart bands. To our knowledge, no published study has reported its data quality. Interestingly, based on the IBS distance and the corresponding SNR results in table 5, and the original correlation coefficients between PPI- and RRI-derived HRV indices in table 6, it seems that the original PPI accuracy of Honor 5 is a bit better than E4. One possible reason is that the sample frequency of PPG signals was 100 Hz for Honor 5 and 64 Hz for E4, and a higher sampling frequency could generate more accurate PPI time series (higher SNR), which can further influence the accuracy of HRV indices.

The mean/average HR is the most conventionally used function for the PPG wristband, and is the most stable and noise-resistant HRV index. The mean HR is commonly used as the key measure to validate the data quality of PPG wristbands. Although a very high correlation (> 0.9) was found between the mean HR of the E4 band and the ECG device, other HRV indices and even the mental state classification accuracy did not show the same conclusion. Instead, due to the higher sample frequency of Honor 5, it seemed that its HRV indices (except the mean HR and HFP, table 6) as well as the mental state classification accuracy (table 7) were more reliable than E4 in this study.

The poor data quality of PPI derived from consumer-grade PPG wristbands can be influenced by various factors. Researchers should carefully consider and account for the following factors to improve the accuracy and reliability of their findings. (1) Motion artifacts: wrist movements, such as hand gestures or physical activities, can introduce motion artifacts, which may lead to inconsistencies in the signal and impact the reliability of PPI monitoring. (2) Skin contact and fit: inadequate contact between the wristband sensors and the skin or improper fit of the device can result in signal distortions. (3) Skin conditions: dryness, sweat, or excessive hair on the wrist may contribute to noise in the IBI data. (4) Device calibration: inaccurate calibration of the wristband sensors may result in systematic errors in PPI measurements, affecting the overall quality of the data. (5) Battery and hardware issues: low battery levels or hardware malfunctions in the wristband device can impact the quality of signal acquisition. (6) Environmental interference: external factors such as electromagnetic interference or ambient light can affect the performance of optical sensors in wristbands. (7) Signal processing algorithms: the algorithms used for processing the raw sensor data play a significant role. Inaccuracies in signal processing methods may lead to errors in IBI calculations. (8) Device-specific limitations: different wristband models may have specific limitations in terms of sensor technology and data-processing capabilities. Understanding and addressing these limitations is important for mitigating signal quality issues.

The primary aim of this study was to objectively quantify inter-beat intervals acquired from consumer-grade PPG wristbands using an ECG-aided IBS approach. The information similarity was defined between PPI and RRI, and the PPG band and ECG device were concurrently worn on the same participant. The proposed approach may inadvertently capture certain gender-related patterns or differences. We acknowledge the significance of sex and gender considerations in research that involves human subjects. There might be broader societal or physiological variations that could impact the interpretation of our findings beyond the scope of our study. We hope that our findings contribute to the ongoing discussions in this field and encourage further investigations into the intricate impacts of gender on wearable health technologies.

In this study, we initially assessed the reliability of consumer-grade wristbands in home healthcare management using data from 30 healthy participants. Further research with a larger cohort with diverse populations and specific clinical conditions is necessary to validate the generalizability of our findings. For example, future work could expand to encompass populations affected by heart conditions to rigorously assess the method's performance under diverse cardiac pathologies.

The reliability of SNR estimation using the binary search method and the three-order Gaussian fitting curve depicted in figure 5 could be potentially influenced by the variation in sample space. Binary search is a commonly used method for finding values within a curve, but other numerical techniques could also be considered depending on the context, such as optimization algorithms, curve interpolation or regression, probabilistic estimation, and so on. The choice of method for finding values within a fitted Gaussian curve depends on various factors, including the computational complexity, required accuracy level, and the specific requirements of the application. We are open to exploring alternative methods or complementary approaches for SNR estimation that could potentially offer increased reliability in diverse sample space conditions. It might be worthwhile in the future to compare and benchmark different methods in terms of accuracy and computational efficiency to determine the most suitable approach for specific applications.

This study proposes a methodology that involves concurrently wearing ECG and PPG devices to assess the reliability of the PPI time series by examining the information similarity between PPI and RRI time series before practical applications. Specifically, before the release of a new PPG device or smart band function, rigorous testing and verification of its reliability is crucial. This methodology can find application in various scenarios, including the selection of wristband manufacturers and models before large-scale studies based on PPG wristbands, verification before the development of health management applications utilizing PPG wristbands, the selection of sampling frequency for PPG wristband hardware design, and pre-release testing of new features by wristband companies based on PPG technology.

The proposed IBS-based approach has an emphasis on the dynamical changing structures of PPI time series, and thus has no requirement for the timeline alignment of measured signals and ECG signals. More significantly, this method can be used for accuracy validation of any IBI time series measured by non-standard ECG devices, including PPG, BCG millimeter-wave radar, Wi-Fi, heart sounds, and so on. Verification using public PPG or BCG databases will be carried out in a follow-up paper.