140:6·L.Wang et al,. ATC Heartbeat Cycle -0.1 0.2 03 0.20.40.60.8 0 0.20.40.6 0.8 Time (s) Time (s) (a)Heartbeat motion stages (b)Heartbeat pattern of volunteer A (c)Heartbeat pattern of volunteer B Fig.3.Heartbeat movement cycle and pattern Time (s) Time(s) (a)Volunteer A (b)Volunteer B Fig.4.Five consecutive heartbeat cycles for volunteer A and B User Authentication(Section 6):Heartbeat authentication uses the SVM model for the heartbeat pattern of the given scenario.We first perform a per-heartbeat evaluation that gives the likelihood that the given heartbeat is from the authorized user.We then combine the likelihood of multiple consecutive heartbeats to improve the confidence in the decision.Our system dynamically determines the number of heartbeats that are required for the authentication process.For example,if the heartbeats have a consistently high likelihood of belonging to the authorized user,the authentication may only require as few as five heartbeats.If the system is not confident in the decision,it may instruct the user to press the phone on the chest for a longer time so that more heartbeat samples can be collected to improve the confidence. 3.2 Background of the SCG Signal The seismocardiogram(SCG)signals collected by accelerometers capture the heartbeat motion of the user. Heartbeat motion is a 3D self-driving heart deformation arising from the stimulation of the cardiac muscle [22].The human heart has two upper chambers(i.e.atria)and two bottom chambers(i.e.ventricles)[32].The continuous contraction and relaxation of atria and ventricles cause the heartbeat motion.As shown in Fig.3(a). one heartbeat motion cycle consists of seven stages:(1)atrial contraction(ATC),(2)mitral valve closing(MC).(3) aortic valve opening(AO),(4)point of maximal acceleration in the aorta (MA),(5)aortic valve closure (AC),(6) mitral valve opening(MO),(7)rapid filling of left ventricle (RF)[16,25]. The motion stages of the heartbeat cycle can be captured and identified using the accelerometer readings provided by mobile phones,see Figure 3(b).As the phone is pressed perpendicularly on the chest,we always use the readings of the y-axis of the accelerometer(pointing from the bottom to the top of the phone).Depending on the stage of the heartbeat cycle,the acceleration caused by the heart motion could be positive or negative. Therefore,each stage in the heartbeat cycle corresponds to one of the peaks or valleys in the SCG signal.Based on our measurements,the average amplitude of the AO peak is 0.2558 m/s2(SD=0.0384 m/s2).The background noise level of the accelerometer has a variance of 0.0104 m/s2.Therefore,commercial mobile phones provide enough Signal-to-Noise Ratio(SNR)for measuring the details in SCG signals. Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2,No.3,Article 140.Publication date:September 2018
140:6 • L. Wang et al. ATC MC AO MA AC MO RF Heartbeat Cycle (a) Heartbeat motion stages 0 0.2 0.4 0.6 0.8 Time (s) -0.5 0 0.5 Acceleration (m/s 2 ) ATC MC MA AO MO AC RF (b) Heartbeat pattern of volunteer A 0 0.2 0.4 0.6 0.8 Time (s) -0.3 -0.2 -0.1 0 0.1 0.2 Acceleration (m/s 2 ) ATC MC AO MA RF MO AC (c) Heartbeat pattern of volunteer B Fig. 3. Heartbeat movement cycle and pattern 01234 Time (s) -0.5 0 0.5 Acceleration (m/s 2 ) AO RF AO RF (a) Volunteer A 01234 Time (s) -0.2 0 0.2 AO RF AO RF Acceleration (m/s 2 ) (b) Volunteer B Fig. 4. Five consecutive heartbeat cycles for volunteer A and B User Authentication (Section 6): Heartbeat authentication uses the SVM model for the heartbeat pattern of the given scenario. We first perform a per-heartbeat evaluation that gives the likelihood that the given heartbeat is from the authorized user. We then combine the likelihood of multiple consecutive heartbeats to improve the confidence in the decision. Our system dynamically determines the number of heartbeats that are required for the authentication process. For example, if the heartbeats have a consistently high likelihood of belonging to the authorized user, the authentication may only require as few as five heartbeats. If the system is not confident in the decision, it may instruct the user to press the phone on the chest for a longer time so that more heartbeat samples can be collected to improve the confidence. 3.2 Background of the SCG Signal The seismocardiogram (SCG) signals collected by accelerometers capture the heartbeat motion of the user. Heartbeat motion is a 3D self-driving heart deformation arising from the stimulation of the cardiac muscle [22]. The human heart has two upper chambers (i.e. atria) and two bottom chambers (i.e. ventricles) [32]. The continuous contraction and relaxation of atria and ventricles cause the heartbeat motion. As shown in Fig.3(a), one heartbeat motion cycle consists of seven stages: (1) atrial contraction (ATC), (2) mitral valve closing (MC), (3) aortic valve opening (AO), (4) point of maximal acceleration in the aorta (MA), (5) aortic valve closure (AC), (6) mitral valve opening (MO), (7) rapid filling of left ventricle (RF) [16, 25]. The motion stages of the heartbeat cycle can be captured and identified using the accelerometer readings provided by mobile phones, see Figure 3(b). As the phone is pressed perpendicularly on the chest, we always use the readings of the y-axis of the accelerometer (pointing from the bottom to the top of the phone). Depending on the stage of the heartbeat cycle, the acceleration caused by the heart motion could be positive or negative. Therefore, each stage in the heartbeat cycle corresponds to one of the peaks or valleys in the SCG signal. Based on our measurements, the average amplitude of the AO peak is 0.2558 m/s 2 (SD= 0.0384 m/s 2 ). The background noise level of the accelerometer has a variance of 0.0104 m/s 2 . Therefore, commercial mobile phones provide enough Signal-to-Noise Ratio (SNR) for measuring the details in SCG signals. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
Unlock with Your Heart:Heartbeat-based Authentication on Commercial Mobile Phones.140:7 0.012 2.5 0.05 -Volunteer A -Volunteer A -Volunteer A 0.01 Volunteer B Volunteer E 0.04 Volunteer B Volunteer C Volunteer C Volunteer C 密0.008 Volunteer D Volunteer D Volunteer D 0.006 Volunt rE Volunteer E Voluntee g0.04 0.02 0.002 在0.01 .400 -200 0 200 400 3 -50 0 50 Deviation(ms) Ratio (AO/RF) Deviation(ms) (a)Deviation of heartbeat interval from the (b)Ratio of AO amplitude to RF Amplitude (c)Deviation of AO-RF interval from the mean mean value value Fig.5.Variations in the SCG signal 3.3 Characteristics of the SCG Signal By looking at the SCG waveforms,we have the following observations that lead to the possibility of using the SCG signal for authentication: First,the SCG signals of different people go through the same seven stages,but have different signal patterns in terms of amplitudes of the corresponding peaks and intervals between peaks.Figure 3(b)and Figure 3(c)show two SCG samples of one heartbeat cycle from two volunteers.While both volunteers have similar heart rates(73 BPM and 71 BPM,respectively),the two SCG patterns have distinctive features.For example,the amplitudes of the AO peaks for the two volunteers are quite different.Such difference in heartbeat motion comes from the differences in the size,position and shape of the heart [38].Therefore,the heartbeat motion patterns contain unique biometric features of the given user [22]. Second,the SCG signals of the same user are consistent over time.Figure 4 shows five consecutive heartbeat patterns of two volunteers.While there are small variations in the signals,we observe that the heartbeat patterns from the same person are consistent for consecutive heartbeat cycles.Furthermore,with heartbeat patterns collected across three months and with different clothes,we find that heartbeat patterns of the same user are quite stable.Therefore,the SCG signal can potentially serve as a consistent identity for the user. 4 HEARTBEAT SEGMENTATION AND ALIGNMENT In this section,we describe the heartbeat segmentation and alignment process,in which the continuous heartbeat signals are divided into individual heartbeat cycles.High precision signal alignment is vital to heartbeat- based authentication systems.This is because a misaligned heartbeat signal will lead to incorrect positioning of the different heartbeat stages.Consequently,such incorrect positioning will lead to errors in user authentication. However,due to the variances in both the amplitude and timing of the SCG signals,it is challenging to precisely align the heartbeat signals. 4.1 Variations in the SCG Signal While human heartbeats are repetitive motions,ECG-based experiments show that heartbeats are not perfectly periodical [2,3,27,45].Therefore,the SCG signals also have variations in both the amplitude and timing of the peaks corresponding to different heart motion stages. First,human heartbeat rates are not stable.There are intrinsic Heartbeat Rate Variability(HRV)in SCG signals [42].Figure 5(a)shows the Probability Density Function(PDF)of the deviation in time intervals between two normal heartbeats for five volunteers sitting on the chair.The ground truth values are obtained by manually selecting the auto-correlation peaks in the SCG signals.We observe that the standard deviation of heartbeat interval is 46ms,which is consistent with results from ECG signals [42].Thus,the duration of heartbeat cycle Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2.No.3,Article 140.Publication date:September 2018
Unlock with Your Heart: Heartbeat-based Authentication on Commercial Mobile Phones • 140:7 -400 -200 0 200 400 Deviation (ms) 0 0.002 0.004 0.006 0.008 0.01 0.012 Probability Density Volunteer A Volunteer B Volunteer C Volunteer D Volunteer E (a) Deviation of heartbeat interval from the mean value 0123 Ratio (AO/RF) 0 0.5 1 1.5 2 2.5 Probability Density Volunteer A Volunteer B Volunteer C Volunteer D Volunteer E (b) Ratio of AO amplitude to RF Amplitude -50 0 50 Deviation (ms) 0 0.01 0.02 0.03 0.04 0.05 Probability Density Volunteer A Volunteer B Volunteer C Volunteer D Volunteer E (c) Deviation of AO-RF interval from the mean value Fig. 5. Variations in the SCG signal 3.3 Characteristics of the SCG Signal By looking at the SCG waveforms, we have the following observations that lead to the possibility of using the SCG signal for authentication: First, the SCG signals of different people go through the same seven stages, but have different signal patterns in terms of amplitudes of the corresponding peaks and intervals between peaks. Figure 3(b) and Figure 3(c) show two SCG samples of one heartbeat cycle from two volunteers. While both volunteers have similar heart rates (73 BPM and 71 BPM, respectively), the two SCG patterns have distinctive features. For example, the amplitudes of the AO peaks for the two volunteers are quite different. Such difference in heartbeat motion comes from the differences in the size, position and shape of the heart [38]. Therefore, the heartbeat motion patterns contain unique biometric features of the given user [22]. Second, the SCG signals of the same user are consistent over time. Figure 4 shows five consecutive heartbeat patterns of two volunteers. While there are small variations in the signals, we observe that the heartbeat patterns from the same person are consistent for consecutive heartbeat cycles. Furthermore, with heartbeat patterns collected across three months and with different clothes, we find that heartbeat patterns of the same user are quite stable. Therefore, the SCG signal can potentially serve as a consistent identity for the user. 4 HEARTBEAT SEGMENTATION AND ALIGNMENT In this section, we describe the heartbeat segmentation and alignment process, in which the continuous heartbeat signals are divided into individual heartbeat cycles. High precision signal alignment is vital to heartbeatbased authentication systems. This is because a misaligned heartbeat signal will lead to incorrect positioning of the different heartbeat stages. Consequently, such incorrect positioning will lead to errors in user authentication. However, due to the variances in both the amplitude and timing of the SCG signals, it is challenging to precisely align the heartbeat signals. 4.1 Variations in the SCG Signal While human heartbeats are repetitive motions, ECG-based experiments show that heartbeats are not perfectly periodical [2, 3, 27, 45]. Therefore, the SCG signals also have variations in both the amplitude and timing of the peaks corresponding to different heart motion stages. First, human heartbeat rates are not stable. There are intrinsic Heartbeat Rate Variability (HRV) in SCG signals [42]. Figure 5(a) shows the Probability Density Function (PDF) of the deviation in time intervals between two normal heartbeats for five volunteers sitting on the chair. The ground truth values are obtained by manually selecting the auto-correlation peaks in the SCG signals. We observe that the standard deviation of heartbeat interval is 46ms, which is consistent with results from ECG signals [42]. Thus, the duration of heartbeat cycle Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
140:8·L.Wang et al.. Generate Coarse Template Coarse Alignment SCG Sequence Linear Locate Prune Cross Detect Heart Rate Interpolation All Peaks Noisy Peaks Correlation Correlation Peaks Fig.6.Heart Rate Estimation Scheme could be changing by as much as 1/20 of the cycle length since a normal heartbeat lasts for about one second at a heart rate of 60 BPM. Second,the peak amplitude in the SCG signal varies significantly.Figure 5(b)shows the PDF for the ratio of the amplitude of the AO peak to the RF peak in the same heartbeat cycle.These two peaks are the most prominent features in the SCG signal.Different persons have different AO to RF ratios,as shown in Figure 4.The standard deviations of AO to RF ratio is larger than 0.25 for all volunteers.This implies that the AO to RF ratio for the same person also varies significantly,e.g.in consecutive heartbeat cycles,either the AO or the RF peak could be the highest peak in the cycle,see Figure 4(b).Therefore,it is challenging to identify the AO and RF peaks using a small number of heartbeat cycles.Existing systems use hints from other measurements,such as the photoplethysmogram(PPG)[59],to help identify the AO peak.However,our system only has the SCG signals as the reference to perform the segmentation. Fortunately,we observe that the time interval between the AO stage and RF stage is relatively stable.Figure 5(c)shows the PDF of the deviation in the time interval between the AO and the RF peak.The standard deviation of the AO-RF interval is 9.48 ms,which is much smaller than that of the heartbeat interval.This implies that the ratio of the AO-RF interval to the heartbeat interval also changes significantly,as the AO-RF interval is stable and the heartbeat interval is unstable.We further verified that the AO-RF intervals are stable under different states. We collect SCG signals when users finish exercising,recline on the sofa and lie on the bed.While the heart rates are significantly higher in the exercising state,the standard deviation of the AO-RF interval is still small(i.e., 11.2 ms).The standard deviations of AO-RF interval for the reclining and lying states are 8.25 ms and 5.86 ms, respectively. Based on the above observations,we choose to use the interval between the ATC stage and the RF stage as the reference for heartbeat segmentation and alignment.We choose the ATC-RF interval due to two reasons. First,the ATC-RF interval contains the two highest peaks in the SCG signal,i.e.,AO and RF,that can be easily identified.Second,the time interval between AO and RF has smaller variations than other parts of the heartbeat cycle.We design a two-step process to divide and align the heartbeat using the signals in the reference interval as follows. 4.2 Heart Rate Estimation Given a new SCG sequence,the first step is to use a heart rate estimation algorithm,as shown in Figure 6,to measure the heart rates.To estimate the heart rates,we first use a linear interpolation algorithm to normalize the accelerometer readings to a standard sampling rate (e.g.,100 Hz).This step ensures that our system can work on mobile phones that have different sampling rates for the accelerometer. The second step of heart rate estimation is to derive a coarse-template of the reference ATC-RF interval from the SCG signals.To identify the ATC-RF interval,we first locate all the peaks(local maximum points)in an SCG sequence with a two-second duration.We assume that the heart rates of the user are between 50 BPM and 120 BPM.Therefore,there is at least one full heartbeat cycle in the two-second SCG signal.We sort the local maximum points by their amplitudes as the labels shown in Figure 7(a).We then perform a pruning algorithm to remove noisy peaks.Starting from the highest peaks,we add the peaks into a candidate set one-by-one in the descending order of their amplitudes.If the current peak is within a time interval of r to one of the candidate peaks in the set, Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2,No.3,Article 140.Publication date:September 2018
140:8 • L. Wang et al. SCG Sequence Generate Coarse Template Linear Interpolation Prune Noisy Peaks Locate All Peaks Coarse Alignment Detect Correlation Peaks Cross Correlation Heart Rate Fig. 6. Heart Rate Estimation Scheme could be changing by as much as 1/20 of the cycle length since a normal heartbeat lasts for about one second at a heart rate of 60 BPM. Second, the peak amplitude in the SCG signal varies significantly. Figure 5(b) shows the PDF for the ratio of the amplitude of the AO peak to the RF peak in the same heartbeat cycle. These two peaks are the most prominent features in the SCG signal. Different persons have different AO to RF ratios, as shown in Figure 4. The standard deviations of AO to RF ratio is larger than 0.25 for all volunteers. This implies that the AO to RF ratio for the same person also varies significantly, e.g., in consecutive heartbeat cycles, either the AO or the RF peak could be the highest peak in the cycle, see Figure 4(b). Therefore, it is challenging to identify the AO and RF peaks using a small number of heartbeat cycles. Existing systems use hints from other measurements, such as the photoplethysmogram (PPG) [59], to help identify the AO peak. However, our system only has the SCG signals as the reference to perform the segmentation. Fortunately, we observe that the time interval between the AO stage and RF stage is relatively stable. Figure 5(c) shows the PDF of the deviation in the time interval between the AO and the RF peak. The standard deviation of the AO-RF interval is 9.48 ms, which is much smaller than that of the heartbeat interval. This implies that the ratio of the AO-RF interval to the heartbeat interval also changes significantly, as the AO-RF interval is stable and the heartbeat interval is unstable. We further verified that the AO-RF intervals are stable under different states. We collect SCG signals when users finish exercising, recline on the sofa and lie on the bed. While the heart rates are significantly higher in the exercising state, the standard deviation of the AO-RF interval is still small (i.e., 11.2 ms). The standard deviations of AO-RF interval for the reclining and lying states are 8.25 ms and 5.86 ms, respectively. Based on the above observations, we choose to use the interval between the ATC stage and the RF stage as the reference for heartbeat segmentation and alignment. We choose the ATC-RF interval due to two reasons. First, the ATC-RF interval contains the two highest peaks in the SCG signal, i.e., AO and RF, that can be easily identified. Second, the time interval between AO and RF has smaller variations than other parts of the heartbeat cycle. We design a two-step process to divide and align the heartbeat using the signals in the reference interval as follows. 4.2 Heart Rate Estimation Given a new SCG sequence, the first step is to use a heart rate estimation algorithm, as shown in Figure 6, to measure the heart rates. To estimate the heart rates, we first use a linear interpolation algorithm to normalize the accelerometer readings to a standard sampling rate (e.g., 100 Hz). This step ensures that our system can work on mobile phones that have different sampling rates for the accelerometer. The second step of heart rate estimation is to derive a coarse-template of the reference ATC-RF interval from the SCG signals. To identify the ATC-RF interval, we first locate all the peaks (local maximum points) in an SCG sequence with a two-second duration. We assume that the heart rates of the user are between 50 BPM and 120 BPM. Therefore, there is at least one full heartbeat cycle in the two-second SCG signal. We sort the local maximum points by their amplitudes as the labels shown in Figure 7(a). We then perform a pruning algorithm to remove noisy peaks. Starting from the highest peaks, we add the peaks into a candidate set one-by-one in the descending order of their amplitudes. If the current peak is within a time interval of τ to one of the candidate peaks in the set, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018