Gait Recognition Using WiFi Signals Wei Wang Alex X.Liu Muhammad Shahzads State Key Laboratory for Novel Software Technology,Nanjing University,China Department of Computer Science and Engineering,Michigan State University,USA s Department of Computer Science,North Carolina State University,USA Email:ww@cs.nju.edu.cn,alexliu @cse.msu.edu,mshahza@ncsu.edu Abstract In this paper,we propose WifiU,which uses commercial WiFi devices to capture fine-grained gait patterns to recognize hu- mans.The intuition is that due to the differences in gaits of different people,the WiFi signal reflected by a walking hu- man generates unique variations in the Channel State Inform- ation(CSD)on the WiFi receiver.To profile human movement using CSI,we use signal processing techniques to generate spectrograms from CSI measurements so that the resulting 7.7m spectrograms are similar to those generated by specifically (a)Application scenario (b)Data collection environment designed Doppler radars.To extract features from spectro- grams that best characterize the walking pattern,we perform Figure 1.WifiU system overview autocorrelation on the torso reflection to remove imperfection in spectrograms.We evaluated WifiU on a dataset with 2,800 be a laptop.Figure 1 shows an overview of our WifiU system. gait instances collected from 50 human subjects walking in a In WifiU,the specific information that the receiver measures room with an area of 50 square meters.Experimental results is the Channel State Information(CSD)of each received WiFi show that WifU achieves top-1,top-2,and top-3 recognition frame.With a human walking around.as a human is mostly accuracies of 79.28%,89.52%,and 93.05%,respectively. made of water,the WiFi signal reflected by the human body generates unique,although small,variations in the CSI meas- ACM Classification Keywords urements on the receiver due to the well-known multi-path H.1.2 User/Machine Systems;I.5.Pattern Recognition effect of wireless signals.These variations in CSI allow us to use signal processing techniques to obtain gait information Author Keywords such as walking speed,gait cycle time,footstep length,and Gait Recognition;Device-free Sensing. movement speeds of legs and torso.As it is well known that humans have quite unique gait [8,15].the gait patterns that INTRODUCTION the WiFi receiver obtains can be used to recognize the walk- ing human subject.Fundamentally,WifiU recognizes humans Motivation and Proposed Approach based on who they are because WifiU extracts unique bio- With the success of WiFi industry,commercially available metrics information from WiFi signals and uses it to perform WiFi devices can not only achieve high speed and low cost, human recognition. but can also measure small changes in WiFi signals and thus sense the changes in their surrounding environments caused WifiU enables many potential applications that require user by moving objects such as humans [30,31].In this paper, identification.For example,a smart building can recognize we propose WifiU,which uses Commercial Off-The-Shelf the user using WifiU,while the user is walking along the cor- (COTS)WiFi devices to capture fine-grained gait patterns so ridor.Based on the user identity,it can automatically open the that we can recognize humans.WifiU consists of two WiFi door when he/she approaches his/her office.Similarly,smart devices,one for continuously sending signals,which can be a home applications can use WifiU to adjust background music router,and one for continuously receiving signals,which can or ambient temperature based on who is at home. Compared with traditional gait recognition systems,which Permission to make digital or hard copies of all or part of this work for personal or use cameras [17,floor sensors [18],or wearable sensors [8 to capture gait information,WifiU is easier to deploy and has tion on the first page.Copyrights for components of this work owned by others than better coverages.From the deployment perspective,WifiU ACM must be honored.Abstracting with credit is permitted.To copy otherwise,or re publish,to post on servers or to redistribute to lists,requires prior specific permission does not require any special hardware(such as floor sensors) and/or a fee.Request permissions from permissions@acm.org. and the human subject does not require to wear any hardware UbiComp '16,September 12-16.2016.Heidelberg.Germany (such as accelerometers).WiFi devices are ubiquitous and ©2016AC3M.ISBN978-1-4503-4461-6/1609.s15.00 most homes/offices are covered by WiFi signals.The hard- D0L:http:/dx.doi.org/10.1145/2971648.2971670
Gait Recognition Using WiFi Signals Wei Wang† Alex X. Liu†‡ Muhammad Shahzad§ †State Key Laboratory for Novel Software Technology, Nanjing University, China ‡Department of Computer Science and Engineering, Michigan State University, USA §Department of Computer Science, North Carolina State University, USA Email: ww@cs.nju.edu.cn, alexliu@cse.msu.edu, mshahza@ncsu.edu Abstract In this paper, we propose WifiU, which uses commercial WiFi devices to capture fine-grained gait patterns to recognize humans. The intuition is that due to the differences in gaits of different people, the WiFi signal reflected by a walking human generates unique variations in the Channel State Information (CSI) on the WiFi receiver. To profile human movement using CSI, we use signal processing techniques to generate spectrograms from CSI measurements so that the resulting spectrograms are similar to those generated by specifically designed Doppler radars. To extract features from spectrograms that best characterize the walking pattern, we perform autocorrelation on the torso reflection to remove imperfection in spectrograms. We evaluated WifiU on a dataset with 2,800 gait instances collected from 50 human subjects walking in a room with an area of 50 square meters. Experimental results show that WifiU achieves top-1, top-2, and top-3 recognition accuracies of 79.28%, 89.52%, and 93.05%, respectively. ACM Classification Keywords H.1.2 User/Machine Systems; I.5. Pattern Recognition Author Keywords Gait Recognition; Device-free Sensing. INTRODUCTION Motivation and Proposed Approach With the success of WiFi industry, commercially available WiFi devices can not only achieve high speed and low cost, but can also measure small changes in WiFi signals and thus sense the changes in their surrounding environments caused by moving objects such as humans [30, 31]. In this paper, we propose WifiU, which uses Commercial Off-The-Shelf (COTS) WiFi devices to capture fine-grained gait patterns so that we can recognize humans. WifiU consists of two WiFi devices, one for continuously sending signals, which can be a router, and one for continuously receiving signals, which can Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. UbiComp ’16, September 12-16, 2016, Heidelberg, Germany © 2016 ACM. ISBN 978-1-4503-4461-6/16/09. . . $15.00 DOI: http://dx.doi.org/10.1145/2971648.2971670 Signal analysis WiFi router WiFi signal reflection Laptop WiFi signal (a) Application scenario Table Table Walking route (5.5m) 7.7 m 6.5 m 1.6 m sender receiver (b) Data collection environment Figure 1. WifiU system overview be a laptop. Figure 1 shows an overview of our WifiU system. In WifiU, the specific information that the receiver measures is the Channel State Information (CSI) of each received WiFi frame. With a human walking around, as a human is mostly made of water, the WiFi signal reflected by the human body generates unique, although small, variations in the CSI measurements on the receiver due to the well-known multi-path effect of wireless signals. These variations in CSI allow us to use signal processing techniques to obtain gait information such as walking speed, gait cycle time, footstep length, and movement speeds of legs and torso. As it is well known that humans have quite unique gait [8, 15], the gait patterns that the WiFi receiver obtains can be used to recognize the walking human subject. Fundamentally, WifiU recognizes humans based on who they are because WifiU extracts unique biometrics information from WiFi signals and uses it to perform human recognition. WifiU enables many potential applications that require user identification. For example, a smart building can recognize the user using WifiU, while the user is walking along the corridor. Based on the user identity, it can automatically open the door when he/she approaches his/her office. Similarly, smart home applications can use WifiU to adjust background music or ambient temperature based on who is at home. Compared with traditional gait recognition systems, which use cameras [17], floor sensors [18], or wearable sensors [8] to capture gait information, WifiU is easier to deploy and has better coverages. From the deployment perspective, WifiU does not require any special hardware (such as floor sensors) and the human subject does not require to wear any hardware (such as accelerometers). WiFi devices are ubiquitous and most homes/offices are covered by WiFi signals. The hard-
ware that we experimented with in this work,namely Net- the scenario where a burglar attempts to know who is at home Gear JR6100 WiFi router and ThinkPad X200 laptop (with by eavesdropping the WiFi signal emitted by the WiFi router Intel 5300 WiFi NIC),are COTS devices and we do not re- in the victim's house.As WiFi signals can penetrate through quire any hardware modification to these devices.Further- obstacles such as furniture,wooden doors,and walls [3].the more,unlike cameras,WifiU does not require lighting and burglar only needs to passively measure the CSI of the signal works in dark just as well as in light outside the house without decoding the content of the WiFi packets.Therefore,it is hard for the victim to prevent privacy Technical Challenges and Solutions breaches.While protecting privacy breach is not the focus of The first challenge is to profile gait patterns using CSI dy- this paper,we hope that this work will bring this privacy issue namics.Extracting gait information from CSI signals is dif- to the research community and inspire future work. ficult because the signal reflections of different body parts are mixed together in the CSI waveform.As different hu- RELATED WORK man body parts move at different speeds while walking,the Gait Based Human Recognition radio signal reflections from different body parts have differ- Prior gait based human recognition schemes mostly use in- ent frequencies.To separate the radio signal reflections from formation collected from the following three types of sensors: different body parts,we convert the CSI waveforms (of two cameras.floor sensors.and wearable sensors [81.Camera dimensions:time and amplitude)to spectrograms in the time- based schemes first generate a sequence of human silhouettes frequency domain (of three dimensions:time,frequency,and from the video frames by separating the moving objects from amplitude).We apply spectrogram enhancement techniques the background,and then use the sequence of human silhou- to reduce signal noises.The resulting spectrograms give us ettes to perform recognition [17].An example camera based detailed human gait information similar to those generated scheme achieves an accuracy of 91%on the USF HumanID by Doppler radars [22.271. Database with 122 subjects [291.Floor sensor based schemes The second challenge is to extract features from spectrograms use force sensors under the floor to detect human footsteps that best characterize the walking pattern of a human.There [18.An example floor sensor based scheme achieves an ac- are various metrics (such as walking speed and gait cycle curacy of 90%~97.5%for identification a single target per- time)that can be used to characterize human walking pat- son [281.Wearable sensor based schemes use the acceler- terns.However,obtaining these measurements from a spec- ometers equipped in smart phones and wearable devices to trogram is challenging because we need to precisely measure collect gait information and extract features for human re- the time of gait cycles.For example,to measure gait cycle cognition [6,19.An example wearable sensor based scheme time,i.e.,the time between two consecutive instances for the achieves an accuracy of 90%over 6 human subjects 24. right heel hits the ground [33],we need to determine the time that the right heel hitting the ground,which is non-trivial.To Activity and Gait Recognition using Radar Signals As a human body can reflect wireless signals,human move- accurately measure gait cycle time,we perform autocorrela- ment can be analyzed using the Doppler shift of the re- tion on the contours of the torso reflection to remove small imperfection in spectrograms.We also generate a spectro- flected signal or the time-of-flight measurements obtained gram signature to distinguish persons with similar walking through Frequency-Modulated Continuous-Wave (FMCW) radars.The micro-Doppler signature,which describes the ve- speeds and footstep lengths. locity of different body parts,can be used for human activity Summary of Experimental Results recognition [16,21,22].Using the Boulic walking model,it is We implemented WifiU on COTS laptop and WiFi router. possible to estimate parameters from the micro-Doppler sig- We conducted experiments on our gait database that contains nature and use these parameters to rebuild the animation of more than 2,800 gait instances collected from 50 human sub- the walking process [27].Tahmoush et al.built a radar based jects walking in a typical laboratory with an area of 50 square gait recognition system that uses the stride rate as features meters as shown in Figure 1(b).We anonymized all collected and achieve recognition accuracy of 80%over 8 human sub- data to protect privacy.Over the 50 subjects,WifiU achieves jects [25].Adib et al.used FMCW Radar to capture a human recognition accuracies of 79.28%,89.52%,and 93.05%for figure behind the wall and to identify the user with an accur- top-1,top-2,and top-3 candidates,respectively acy of 88%over 15 human subjects [1].Compared to radar signals,WiFi signals are much easier and cheaper to obtain, Limitations and Privacy Implications but have much narrower bandwidth(20 MHz compared to The current implementation of WifiU has two limitations. 1.79 GHz of the FMCW radar used in WiTrack [21)and thus First,the recognition system is suitable only for confined have much lower time-resolutions.This makes human gait spaces,such as a corridor or a narrow entrance,because the recognition using WiFi signals much harder than using radar users must walk on a predefined path in a predefined direc- signals.Consequently,the techniques in this paper are funda- tion.Second,WifiU can extract the gait pattern only when mentally different from those used in FMCW radars there is a single user walking on the predefined path.Our future work is to explore the possibility of using multiple Activity Recognition using WiFi Signals devices to separate gait signals from multiple users. CSI measurements have been used for applications such as human activity recognition [31,32],fall detection [12],and From the perspective of privacy,hackers can potentially use even object localization [23].E-eyes utilized CSI histogram WifiU to identify"victims"without being detected.Consider to achieve in-place and walking activity recognition with an
ware that we experimented with in this work, namely NetGear JR6100 WiFi router and ThinkPad X200 laptop (with Intel 5300 WiFi NIC), are COTS devices and we do not require any hardware modification to these devices. Furthermore, unlike cameras, WifiU does not require lighting and works in dark just as well as in light. Technical Challenges and Solutions The first challenge is to profile gait patterns using CSI dynamics. Extracting gait information from CSI signals is dif- ficult because the signal reflections of different body parts are mixed together in the CSI waveform. As different human body parts move at different speeds while walking, the radio signal reflections from different body parts have different frequencies. To separate the radio signal reflections from different body parts, we convert the CSI waveforms (of two dimensions: time and amplitude) to spectrograms in the timefrequency domain (of three dimensions: time, frequency, and amplitude). We apply spectrogram enhancement techniques to reduce signal noises. The resulting spectrograms give us detailed human gait information similar to those generated by Doppler radars [22, 27]. The second challenge is to extract features from spectrograms that best characterize the walking pattern of a human. There are various metrics (such as walking speed and gait cycle time) that can be used to characterize human walking patterns. However, obtaining these measurements from a spectrogram is challenging because we need to precisely measure the time of gait cycles. For example, to measure gait cycle time, i.e., the time between two consecutive instances for the right heel hits the ground [33], we need to determine the time that the right heel hitting the ground, which is non-trivial. To accurately measure gait cycle time, we perform autocorrelation on the contours of the torso reflection to remove small imperfection in spectrograms. We also generate a spectrogram signature to distinguish persons with similar walking speeds and footstep lengths. Summary of Experimental Results We implemented WifiU on COTS laptop and WiFi router. We conducted experiments on our gait database that contains more than 2,800 gait instances collected from 50 human subjects walking in a typical laboratory with an area of 50 square meters as shown in Figure 1(b). We anonymized all collected data to protect privacy. Over the 50 subjects, WifiU achieves recognition accuracies of 79.28%, 89.52%, and 93.05% for top-1, top-2, and top-3 candidates, respectively. Limitations and Privacy Implications The current implementation of WifiU has two limitations. First, the recognition system is suitable only for confined spaces, such as a corridor or a narrow entrance, because the users must walk on a predefined path in a predefined direction. Second, WifiU can extract the gait pattern only when there is a single user walking on the predefined path. Our future work is to explore the possibility of using multiple devices to separate gait signals from multiple users. From the perspective of privacy, hackers can potentially use WifiU to identify “victims” without being detected. Consider the scenario where a burglar attempts to know who is at home by eavesdropping the WiFi signal emitted by the WiFi router in the victim’s house. As WiFi signals can penetrate through obstacles such as furniture, wooden doors, and walls [3], the burglar only needs to passively measure the CSI of the signal outside the house without decoding the content of the WiFi packets. Therefore, it is hard for the victim to prevent privacy breaches. While protecting privacy breach is not the focus of this paper, we hope that this work will bring this privacy issue to the research community and inspire future work. RELATED WORK Gait Based Human Recognition Prior gait based human recognition schemes mostly use information collected from the following three types of sensors: cameras, floor sensors, and wearable sensors [8]. Camera based schemes first generate a sequence of human silhouettes from the video frames by separating the moving objects from the background, and then use the sequence of human silhouettes to perform recognition [17]. An example camera based scheme achieves an accuracy of 91% on the USF HumanID Database with 122 subjects [29]. Floor sensor based schemes use force sensors under the floor to detect human footsteps [18]. An example floor sensor based scheme achieves an accuracy of 90%∼97.5% for identification a single target person [28]. Wearable sensor based schemes use the accelerometers equipped in smart phones and wearable devices to collect gait information and extract features for human recognition [6, 19]. An example wearable sensor based scheme achieves an accuracy of 90% over 6 human subjects [24]. Activity and Gait Recognition using Radar Signals As a human body can reflect wireless signals, human movement can be analyzed using the Doppler shift of the re- flected signal or the time-of-flight measurements obtained through Frequency-Modulated Continuous-Wave (FMCW) radars. The micro-Doppler signature, which describes the velocity of different body parts, can be used for human activity recognition [16,21,22]. Using the Boulic walking model, it is possible to estimate parameters from the micro-Doppler signature and use these parameters to rebuild the animation of the walking process [27]. Tahmoush et al. built a radar based gait recognition system that uses the stride rate as features and achieve recognition accuracy of 80% over 8 human subjects [25]. Adib et al. used FMCW Radar to capture a human figure behind the wall and to identify the user with an accuracy of 88% over 15 human subjects [1]. Compared to radar signals, WiFi signals are much easier and cheaper to obtain, but have much narrower bandwidth (20 MHz compared to 1.79 GHz of the FMCW radar used in WiTrack [2]) and thus have much lower time-resolutions. This makes human gait recognition using WiFi signals much harder than using radar signals. Consequently, the techniques in this paper are fundamentally different from those used in FMCW radars. Activity Recognition using WiFi Signals CSI measurements have been used for applications such as human activity recognition [31, 32], fall detection [12], and even object localization [23]. E-eyes utilized CSI histogram to achieve in-place and walking activity recognition with an
accuracy of 92%[32].WiFall used anomaly detection al- there is no moving object around,the magnitude of CSI re- gorithm on CSI values to detect falling [12].PinLoc gathered mains relatively constant.However,a moving human can location dependent CSI value distributions to localize a tar- significantly distort the CSI magnitude because human bodies get with accuracy of several centimeters [23].MRW used are good reflectors of wireless signals.When a human moves. CSI or Received Signal Strength (RSS)values to monitor the wireless signal reflected by his body will go through a dif- the presence of human movement behind a concrete wall [4] ferent path with different length,as illustrated in Figure 1(a). CARM used CSI-speed model to extract human movement speeds from CSI signal [31].However,these speed features Governed by the principle of superposition of waves,signal are still not accurate enough for gait pattern recognition.In reflected by human may add constructively or destructively comparison,we perform time-frequency analysis on CSI val- with WiFi signals traveled through other paths,e.g.,the Line- ues to obtain spectrograms,and then to measure human walk- Of-Sight (LOS)path.Whether these WiFi signals are added ing metrics,such as gait cycle time,torso and leg speed,and constructively or destructively is determined by the relative footstep length from the spectrograms.Note that there are phase differences between these signals [26].The phase of schemes that use special hardware(such as USRP [20]and the signal changes by 2 when the length change of the sig- directional antennas [13,30])to get fine-grained measure- nal path is equal to the signal wavelength.Whenever the hu- ments from WiFi signals:in comparison,we only use COTS man moves by half of the wavelength,the path length of the WiFi devices. human reflected signal will change by the amount of the sig- nal wavelength.Consequently,we will observe a full cycle of In recent work parallel with ours,Zeng et al.built a system magnitude changes in CSI values according to the principle called WiWho that also uses WiFi CSI signals to recognize of superposition of waves.The factor of 2 in the path length human gaits [34].In WiWho,the human subject was asked to change is due to the round trip path travelled by the reflected walk on a path with a distance of I meter parallel to the Line- WiFi signal.The signal wavelength A for 5 GHz WiFi signal Of-Sight (LOS)path between the WiFi sender and receiver. band is 5.15~5.79 cm.This implies that we can detect small The recognition accuracy of WiWho is 92%to 80%for 2 to 6 movements of a few centimeters by observing the magnitude human subjects.Compared to WiWho,WifiU uses advanced changes of CSI values.For a walking human who moves at signal processing algorithms to extract CSI variations so that the speed of 1 meter per second,we observe 34~38 cycles of it can recognize gaits at a distance of more than 6 meters to magnitude fluctuations in CSI values,given the wavelength of the LOS path.Moreover,WifiU extracts more elaborated gait 5.15~5.79 cm,as each cycle represents the person moves by features from CSI signals the distance of half-wavelength.This provides a very detailed WIFI SIGNAL PROCESSING measurement of the human walking speed. In this section,we first collect CSI measurements from WiFi However,the CSI measurements obtained from commercial signals using COTS WiFi devices.Second,we use the Prin- WiFi cards contain noises from various sources such as in- cipal Component Analysis (PCA)technique to extract the terference coming from nearby devices,transmission power principal components from the correlated CSI measurements adaptation at the sender,and imperfect clock synchronization so that the uncorrelated noises in different subcarriers are re- [101.Figure 2(a)shows the magnitude of a raw CSI stream duced.Third,we use Short Time Fourier Transform(STFT) (Stream A)captured while a human is walking around.Al- to convert PCA components into spectrograms. Fourth though we can observe the fluctuations in CSI values caused we apply frequency domain denoising algorithms(such as by the moving human subject,these fluctuations are irregu- noise floor subtraction,spectrogram superimposition,and 2- lar due to environmental noises.Thus,we must denoise CSI dimensional filtering)to further enhance the spectrogram. measurements before we extract human gait information. CSI Data Collection WifiU collects CSI measurements on the receiving end of a Existing CSI denoising schemes,such as low-pass filters [30] WiFi link between two WiFi devices.For each pair of a send- do not work well for our purpose because CSI streams con- ing antenna and a receiving antenna,we obtain CSI values tain high-level impulse and burst noises.Figure 2(a)shows an from 30 OFDM subcarriers used by 802.11n [14].Thus,we example CSI stream with impulse noises at the time of 10.55 get 2 x 3 x 30=180 CSI values for each received 802.11n seconds as pointed by an arrow.Figure 2(b)shows the low- frame when the sender has 2 antennas and the receiver has pass filtering result of CSI Stream A after passing through 3 antennas.The sequence of CSI values for each subcar- a Butterworth filter with a cutoff frequency at 150 Hz.We rier for a given pair of sending/receiving antenna is called can still observe small residual fluctuations,as pointed by a CSI stream.As our system sends 2.500 WiFi frames per an arrow,after low-pass filtering,due to the wide bandwidth second,we collected 2,500 CSI values for each of the 180 of impulse noises.Figure 2(b)plots another CSI Stream B. CSI streams in one second.We removed the impact of Car- which is measured on the same sender/receiver antenna pair rier Frequency Offset(CFO)by using only the amplitude of but with a subcarrier frequency about 10 MHz higher than the CSI values while ignoring the CSI phase,as described in that of CSI Stream A.Zooming into the waveform segments our earlier work [31. separated by the four vertical reference lines,we observe that the "valleys"for Stream B always appear earlier than that of Denoising CSI Measurements Stream A.This indicates that variations in CSI streams have The CSI values describe how the phase and magnitude of the different phases.Different CSI streams often have differ- wireless signal change when the signal travels from the send- ent phases because they differ in their subcarrier frequency ing antenna to the receiving antenna over a subcarrier.When
accuracy of 92% [32]. WiFall used anomaly detection algorithm on CSI values to detect falling [12]. PinLoc gathered location dependent CSI value distributions to localize a target with accuracy of several centimeters [23]. MRW used CSI or Received Signal Strength (RSS) values to monitor the presence of human movement behind a concrete wall [4]. CARM used CSI-speed model to extract human movement speeds from CSI signal [31]. However, these speed features are still not accurate enough for gait pattern recognition. In comparison, we perform time-frequency analysis on CSI values to obtain spectrograms, and then to measure human walking metrics, such as gait cycle time, torso and leg speed, and footstep length from the spectrograms. Note that there are schemes that use special hardware (such as USRP [20] and directional antennas [13, 30]) to get fine-grained measurements from WiFi signals; in comparison, we only use COTS WiFi devices. In recent work parallel with ours, Zeng et al. built a system called WiWho that also uses WiFi CSI signals to recognize human gaits [34]. In WiWho, the human subject was asked to walk on a path with a distance of 1 meter parallel to the LineOf-Sight (LOS) path between the WiFi sender and receiver. The recognition accuracy of WiWho is 92% to 80% for 2 to 6 human subjects. Compared to WiWho, WifiU uses advanced signal processing algorithms to extract CSI variations so that it can recognize gaits at a distance of more than 6 meters to the LOS path. Moreover, WifiU extracts more elaborated gait features from CSI signals. WIFI SIGNAL PROCESSING In this section, we first collect CSI measurements from WiFi signals using COTS WiFi devices. Second, we use the Principal Component Analysis (PCA) technique to extract the principal components from the correlated CSI measurements so that the uncorrelated noises in different subcarriers are reduced. Third, we use Short Time Fourier Transform (STFT) to convert PCA components into spectrograms. Fourth, we apply frequency domain denoising algorithms (such as noise floor subtraction, spectrogram superimposition, and 2- dimensional filtering) to further enhance the spectrogram. CSI Data Collection WifiU collects CSI measurements on the receiving end of a WiFi link between two WiFi devices. For each pair of a sending antenna and a receiving antenna, we obtain CSI values from 30 OFDM subcarriers used by 802.11n [14]. Thus, we get 2 × 3 × 30 = 180 CSI values for each received 802.11n frame when the sender has 2 antennas and the receiver has 3 antennas. The sequence of CSI values for each subcarrier for a given pair of sending/receiving antenna is called a CSI stream. As our system sends 2,500 WiFi frames per second, we collected 2,500 CSI values for each of the 180 CSI streams in one second. We removed the impact of Carrier Frequency Offset (CFO) by using only the amplitude of the CSI values while ignoring the CSI phase, as described in our earlier work [31]. Denoising CSI Measurements The CSI values describe how the phase and magnitude of the wireless signal change when the signal travels from the sending antenna to the receiving antenna over a subcarrier. When there is no moving object around, the magnitude of CSI remains relatively constant. However, a moving human can significantly distort the CSI magnitude because human bodies are good reflectors of wireless signals. When a human moves, the wireless signal reflected by his body will go through a different path with different length, as illustrated in Figure 1(a). Governed by the principle of superposition of waves, signal reflected by human may add constructively or destructively with WiFi signals traveled through other paths, e.g., the LineOf-Sight (LOS) path. Whether these WiFi signals are added constructively or destructively is determined by the relative phase differences between these signals [26]. The phase of the signal changes by 2π when the length change of the signal path is equal to the signal wavelength. Whenever the human moves by half of the wavelength, the path length of the human reflected signal will change by the amount of the signal wavelength. Consequently, we will observe a full cycle of magnitude changes in CSI values according to the principle of superposition of waves. The factor of 2 in the path length change is due to the round trip path travelled by the reflected WiFi signal. The signal wavelength λ for 5 GHz WiFi signal band is 5.15∼5.79 cm. This implies that we can detect small movements of a few centimeters by observing the magnitude changes of CSI values. For a walking human who moves at the speed of 1 meter per second, we observe 34∼38 cycles of magnitude fluctuations in CSI values, given the wavelength of 5.15∼5.79 cm, as each cycle represents the person moves by the distance of half-wavelength. This provides a very detailed measurement of the human walking speed. However, the CSI measurements obtained from commercial WiFi cards contain noises from various sources such as interference coming from nearby devices, transmission power adaptation at the sender, and imperfect clock synchronization [10]. Figure 2(a) shows the magnitude of a raw CSI stream (Stream A) captured while a human is walking around. Although we can observe the fluctuations in CSI values caused by the moving human subject, these fluctuations are irregular due to environmental noises. Thus, we must denoise CSI measurements before we extract human gait information. Existing CSI denoising schemes, such as low-pass filters [30], do not work well for our purpose because CSI streams contain high-level impulse and burst noises. Figure 2(a) shows an example CSI stream with impulse noises at the time of 10.55 seconds as pointed by an arrow. Figure 2(b) shows the lowpass filtering result of CSI Stream A after passing through a Butterworth filter with a cutoff frequency at 150 Hz. We can still observe small residual fluctuations, as pointed by an arrow, after low-pass filtering, due to the wide bandwidth of impulse noises. Figure 2(b) plots another CSI Stream B, which is measured on the same sender/receiver antenna pair but with a subcarrier frequency about 10 MHz higher than that of CSI Stream A. Zooming into the waveform segments separated by the four vertical reference lines, we observe that the “valleys” for Stream B always appear earlier than that of Stream A. This indicates that variations in CSI streams have different phases. Different CSI streams often have different phases because they differ in their subcarrier frequency
Stream A 30 -Stream B 10.1 10.2 10.5 10.6 10.7 20 10.1 10.2 10.5 10.6 107 10.1 10.9 10.3 10.5 10.6 (a)Raw CSI waveforms(Stream A) (b)CSI waveforms after low-pass filtering (c)CSI waveforms after PCA denoising Figure 2.CSI waveforms for a person walking along a straight path as in Figure 1(b) and thus their radio wavelengths also differ.The phases of moving speeds,we convert the PCA denoised CSI waveforms these CSI streams can differ by a and their waveforms may into the time-frequency domain.We use the Short-Time Four- even cancel each other.Therefore,directly adding these CSI ier Transform(STFT)technique to transform the waveforms streams is also not a good choice for noise removal. to spectrograms so that CSI waveforms can be analyzed in the We use Principal Component Analysis(PCA)based denois- time-frequency domain.The spectrogram uses a sliding win- dow to cut a small piece of samples from the waveform and ing algorithm to remove noises in the CSI signal [311.Phase differences in CSI streams can be characterized by their cross then performs Fast Fourier transform(FFT)on the waveform. correlations.PCA can automatically discover the correlation The spectrogram has three dimensions:time,frequency,and between CSI streams and recombine CSI streams to extract FFT amplitude.The window size for FFT determines the components that represent the variation caused by human tradeoffs between frequency and time resolution of STFT. activity.To remove the noise,we extract the first twenty PCA With a larger window size,the STFT has higher frequency resolution but lower time resolutions.CSI measurements for components from the 180 CSI streams and discard the rest, human walking have frequency of 30~40 Hz and changes in which are mostly noisy components.The optimal number of tens of milliseconds.Thus,we choose to use an FFT size PCA components is determined by experiments shown in the experimental results section.The PCA denoised CSI wave- of 1024 samples and sliding window step size of 32 samples forms are much smoother than those generated by low-pass because it gives suitable frequency resolution of 2.44 Hz and filtering and preserve the details in CSI amplitude changes. time resolution of 12.8 ms to trace the changes in walking signals.Figure 3 shows an example spectrogram in a heat Figure 2(c)shows the second PCA component derived from the raw CSI streams in Figure 2(a).We observe that there map,where hotter color represents higher FFT amplitude.To are 17 peaks for the 0.5 second between 10~10.5 seconds build the heat map for the spectrogram,we use the first 80 in Figure 2(c).This gives 34 cycles/sec fluctuations in CSI. FFT magnitude of each chunk to represent the energy in the matching the walk speed of 1m/s of the human subject. frequency range of 0~146 Hz Spectrogram Enhancement Spectrogram Generation After a spectrogram is generated,we apply spectrogram en- It is still difficult to extract human gait information from a hancement techniques to further reduce the noise in the spec- PCA denoised CSI waveform because the signal reflections trogram. First,we get the energy level for each chunk by of different body parts are mixed together in the waveform. summing the magnitude of the first 80 FFT points.We Different human body parts move at different speeds during ignore chunks with energy level lower than an empirical different phases in walking.A gait cycle of human walking threshold so that silent periods are removed.We normalize has two phases:the stance phase and the swing phase [33]. the FFT magnitudes for each remaining chunk by dividing The stance phase starts at the time that the right heel touches them with the energy level of the chunk.Second,we ap- the ground.During the stance phase,the left leg moves for- ply frequency domain denoising method by subtracting the ward.At the end of the stance phase.the right toe leaves the noise floor from the spectrogram [7].The noise floor level is ground.In the swing phase,the human body is supported by estimated through a long-term mean over the spectrograms, the left leg and the right leg is swinging forward.Thus,the e.g.,average magnitude over 4 seconds.If the resulting mag- right leg moves at a higher speed than the torso and the left leg nitude becomes negative after subtraction,we set it to zero. is almost static during this phase [27].As different body parts Third,we superimpose the spectrograms generated by the move at different speeds,the radio signal reflections of differ- first twenty PCA results by adding up the magnitude of cor- ent body parts have different frequencies.CSI frequency is responding time-frequency blocks.Fourth,we apply a 2- determined by human moving speeds as f=2v/A,where v dimensional Gaussian low-pass filter,with a size of 5 and is the moving speed,f is the frequency of the CSI waveform, o of 0.8,on the resulting spectrogram.Finally,we obtain and A is the radio signal wavelength.For example,a moving a high quality spectrogram that gives detailed information for speed of 1 m/s can be converted to the frequency of 34.54 Hz the walking process of a human subject. when the radio wavelength A is 5.79cm. FEATURE EXTRACTION As the radio signal reflections of different body parts have dif- In this section,we present how gait features can be extracted ferent frequencies,the CSI fluctuations of different frequen- from the CSI spectrogram.The gait features include walking cies are separable in the frequency domain.Thus,to separate speed,gait cycle time,footstep length,movement speeds of the radio signal reflections of different body parts of different torso and legs,and spectrogram signatures
10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 35 40 45 50 55 Time (seconds) CSI amplitude Impulse noises (a) Raw CSI waveforms (Stream A) 10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 25 30 35 40 45 50 55 Time (seconds) CSI amplitude Stream A Stream B Residual noises (b) CSI waveforms after low-pass filtering 10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 −30 −20 −10 0 10 20 30 Time (seconds) PCA amplitude (c) CSI waveforms after PCA denoising Figure 2. CSI waveforms for a person walking along a straight path as in Figure 1(b) and thus their radio wavelengths also differ. The phases of these CSI streams can differ by π and their waveforms may even cancel each other. Therefore, directly adding these CSI streams is also not a good choice for noise removal. We use Principal Component Analysis (PCA) based denoising algorithm to remove noises in the CSI signal [31]. Phase differences in CSI streams can be characterized by their cross correlations. PCA can automatically discover the correlation between CSI streams and recombine CSI streams to extract components that represent the variation caused by human activity. To remove the noise, we extract the first twenty PCA components from the 180 CSI streams and discard the rest, which are mostly noisy components. The optimal number of PCA components is determined by experiments shown in the experimental results section. The PCA denoised CSI waveforms are much smoother than those generated by low-pass filtering and preserve the details in CSI amplitude changes. Figure 2(c) shows the second PCA component derived from the raw CSI streams in Figure 2(a). We observe that there are 17 peaks for the 0.5 second between 10∼10.5 seconds in Figure 2(c). This gives 34 cycles/sec fluctuations in CSI, matching the walk speed of 1m/s of the human subject. Spectrogram Generation It is still difficult to extract human gait information from a PCA denoised CSI waveform because the signal reflections of different body parts are mixed together in the waveform. Different human body parts move at different speeds during different phases in walking. A gait cycle of human walking has two phases: the stance phase and the swing phase [33]. The stance phase starts at the time that the right heel touches the ground. During the stance phase, the left leg moves forward. At the end of the stance phase, the right toe leaves the ground. In the swing phase, the human body is supported by the left leg and the right leg is swinging forward. Thus, the right leg moves at a higher speed than the torso and the left leg is almost static during this phase [27]. As different body parts move at different speeds, the radio signal reflections of different body parts have different frequencies. CSI frequency is determined by human moving speeds as f = 2v/λ, where v is the moving speed, f is the frequency of the CSI waveform, and λ is the radio signal wavelength. For example, a moving speed of 1 m/s can be converted to the frequency of 34.54 Hz when the radio wavelength λ is 5.79cm. As the radio signal reflections of different body parts have different frequencies, the CSI fluctuations of different frequencies are separable in the frequency domain. Thus, to separate the radio signal reflections of different body parts of different moving speeds, we convert the PCA denoised CSI waveforms into the time-frequency domain. We use the Short-Time Fourier Transform (STFT) technique to transform the waveforms to spectrograms so that CSI waveforms can be analyzed in the time-frequency domain. The spectrogram uses a sliding window to cut a small piece of samples from the waveform and then performs Fast Fourier transform (FFT) on the waveform. The spectrogram has three dimensions: time, frequency, and FFT amplitude. The window size for FFT determines the tradeoffs between frequency and time resolution of STFT. With a larger window size, the STFT has higher frequency resolution but lower time resolutions. CSI measurements for human walking have frequency of 30∼40 Hz and changes in tens of milliseconds. Thus, we choose to use an FFT size of 1024 samples and sliding window step size of 32 samples because it gives suitable frequency resolution of 2.44 Hz and time resolution of 12.8 ms to trace the changes in walking signals. Figure 3 shows an example spectrogram in a heat map, where hotter color represents higher FFT amplitude. To build the heat map for the spectrogram, we use the first 80 FFT magnitude of each chunk to represent the energy in the frequency range of 0∼146 Hz. Spectrogram Enhancement After a spectrogram is generated, we apply spectrogram enhancement techniques to further reduce the noise in the spectrogram. First, we get the energy level for each chunk by summing the magnitude of the first 80 FFT points. We ignore chunks with energy level lower than an empirical threshold so that silent periods are removed. We normalize the FFT magnitudes for each remaining chunk by dividing them with the energy level of the chunk. Second, we apply frequency domain denoising method by subtracting the noise floor from the spectrogram [7]. The noise floor level is estimated through a long-term mean over the spectrograms, e.g., average magnitude over 4 seconds. If the resulting magnitude becomes negative after subtraction, we set it to zero. Third, we superimpose the spectrograms generated by the first twenty PCA results by adding up the magnitude of corresponding time-frequency blocks. Fourth, we apply a 2- dimensional Gaussian low-pass filter, with a size of 5 and σ of 0.8, on the resulting spectrogram. Finally, we obtain a high quality spectrogram that gives detailed information for the walking process of a human subject. FEATURE EXTRACTION In this section, we present how gait features can be extracted from the CSI spectrogram. The gait features include walking speed, gait cycle time, footstep length, movement speeds of torso and legs, and spectrogram signatures
ment is detected when the variance of the PCA result substan- 80 tially deviates from the average noise level.Once the move- ment is detected,we use the torso speed to find the period that 60 the person walks with a steady speed. Estimating Gait Cycle Time We now estimate gait cycle time,which is defined as the time duration between two consecutive events that the right heel Time (seconds) touches the ground.Although we can roughly count steps Figure 3.CSI Spectrogram with human walking. from Figure 3,we need a robust estimation scheme that works Understanding CSI Spectrogram for various walking styles.In WifiU,we use the upper con- tour for the torso reflection to estimate the gait cycle time. We observe that CSI spectrograms give us similar informa- The torso contour frequency fic(t)is defined as: tion as the expensive Doppler radars [16,22,27].Thus,we can extract human gait patterns from CSI spectrograms using fic(t)=max Ff,)一>Y (2) insights from Doppler radars.In particular,we can observe ∑aFf,t) the moving speed changes of different body parts,e.g.,torso and legs,from the spectrogram.Figure 3 shows the spec- where F(f,t)is the magnitude of FFT at time t and frequency trogram obtained in a scenario where a human walks on a f.The summation of m F(f,t)gives the total energy in straight line for a distance of 5.5 meters.Since the torso has the frequency range of 0~fimar.Thus,fic(t)is the highest a larger reflection area than other body parts,the signal en- frequency that has an energy ratio larger than y for the FFT ergy reflected from the torso is the strongest component in the result at time t.We set the threshold y as 5%in our sys- spectrogram.We observe a high energy ("hot"colored)band, tem.After smoothing the resulting fic(t)with a low-pass which corresponds to torso reflection,in the spectrogram in filter,we get the torso contour speed vre(t)as the thick line Figure 3.From the frequency change in the torso reflection in Figure 4(a)using the equation vre(t)=fic(t)/2.Com- in Figure 3,we learn that the subject starts walking at the paring the curve in Figure 4(a)with the spectrogram in Fig- time of 0.5 seconds,speeds up,and then walks with a steady ure 3,we observe that our method reliably tracks the torso speed between the time of 1.5~5 seconds,finally he slows speed variations during the walking periods.The reason for down and stops at the time of 6 seconds.During the walking choosing the contour rather than the frequency that has max- process,the torso reflection has a frequency range of 27~46 imum energy is due to its robustness to noises.With a small Hz,which corresponds to 0.70~1.18 m/s moving speed.We amount of noise,as in the 3~4 seconds period in Figure 3. observe that the torso moving speed fluctuates regularly for the frequency with maximum energy,i.e.,the"hottest"fre- about twice per second.This type of regular fluctuations in quency for a given time,may change significantly and makes torso speed is also observed in Doppler radar results [27] the measurement unreliable and discontinuous in time. The"blue flames"in the frequency region of 40~60 Hz are We use the autocorrelation of the torso contour curve to ro- the reflections of swinging legs [25].The magnitude of leg bustly estimate gait cycle time,which is calculated as: reflections are weak due to the much smaller reflection area R()=>(vic(t)-H)(vic(t-t)-H) (3) of legs.Both the regular speed changes in leg and body reflec- tion indicate that the gait cycle,which contains two footsteps, where u is the average speed of the torso contour.The auto- has a duration of about 1 second.With these basic under- correlation is taken on the period of steady walking,which standings of the spectrogram,we are ready to extract useful is defined as the period where the torso speed(calculated in information about the walking pattern. Section 4.4)is no less than 80%of the maximum torso speed. By taking the autocorrelation,we can find the period of the Detecting Start of Walking curve by comparing the curve to a version of itself that is displaced by t in time.This gives a better estimation than We detect the start of walking based on the fact that mov- directly searching for peaks on the torso contour curve. ing humans generate higher variances in CSI measurements than background noises.When there are no human move- Compared to the original torso contour curve in Figure 4(a),it ments,the variances in CSI measurements are mostly caused is easier to find the peaks in the autocorrelation result in Fig- by noises.Since the noise level changes slowly over time,we ure 4(b).Each peak in autocorrelation means that the contour use a dynamic thresholding algorithm to track the noise level. curve remains similar to the original version when displaced We first calculate the variance var(t)for a segment with 500 by a time of t.As the gait cycle contains two footsteps,the samples at time t in the second PCA component derived in torso speed varies twice in each gait cycle.So,the torso con- Section 3.We do not use the first PCA component because tour curve is similar when displaced by L/2,where L is the it may contain a large amount of noise [31].We update the gait cycle time.In Figure 4(b),the first peak in the autocor- noise level estimation N(t)using an Exponential Moving Av- relation appears at the point=0.57 second;thus,we obtain erage algorithm during the silent period: an estimation for the gait cycle time L=2t as 1.14 seconds. N(t)=(1-an)N(t-1)+an x var(t) (1) Estimating Torso and Leg Speeds where the coefficient an is set to 0.1.We set the detection We estimate the torso and leg speed using the percentile threshold as three times the noise level N(t)so that a move- method developed for Doppler radars [27].The percentile
Figure 3. CSI Spectrogram with human walking. Understanding CSI Spectrogram We observe that CSI spectrograms give us similar information as the expensive Doppler radars [16, 22, 27]. Thus, we can extract human gait patterns from CSI spectrograms using insights from Doppler radars. In particular, we can observe the moving speed changes of different body parts, e.g., torso and legs, from the spectrogram. Figure 3 shows the spectrogram obtained in a scenario where a human walks on a straight line for a distance of 5.5 meters. Since the torso has a larger reflection area than other body parts, the signal energy reflected from the torso is the strongest component in the spectrogram. We observe a high energy (“hot” colored) band, which corresponds to torso reflection, in the spectrogram in Figure 3. From the frequency change in the torso reflection in Figure 3, we learn that the subject starts walking at the time of 0.5 seconds, speeds up, and then walks with a steady speed between the time of 1.5∼5 seconds, finally he slows down and stops at the time of 6 seconds. During the walking process, the torso reflection has a frequency range of 27∼46 Hz, which corresponds to 0.70∼1.18 m/s moving speed. We observe that the torso moving speed fluctuates regularly for about twice per second. This type of regular fluctuations in torso speed is also observed in Doppler radar results [27]. The “blue flames” in the frequency region of 40∼60 Hz are the reflections of swinging legs [25]. The magnitude of leg reflections are weak due to the much smaller reflection area of legs. Both the regular speed changes in leg and body reflection indicate that the gait cycle, which contains two footsteps, has a duration of about 1 second. With these basic understandings of the spectrogram, we are ready to extract useful information about the walking pattern. Detecting Start of Walking We detect the start of walking based on the fact that moving humans generate higher variances in CSI measurements than background noises. When there are no human movements, the variances in CSI measurements are mostly caused by noises. Since the noise level changes slowly over time, we use a dynamic thresholding algorithm to track the noise level. We first calculate the variance var(t) for a segment with 500 samples at time t in the second PCA component derived in Section 3. We do not use the first PCA component because it may contain a large amount of noise [31]. We update the noise level estimation N(t) using an Exponential Moving Average algorithm during the silent period: N(t) = (1−αn)N(t −1) +αn ×var(t) (1) where the coefficient αn is set to 0.1. We set the detection threshold as three times the noise level N(t) so that a movement is detected when the variance of the PCA result substantially deviates from the average noise level. Once the movement is detected, we use the torso speed to find the period that the person walks with a steady speed. Estimating Gait Cycle Time We now estimate gait cycle time, which is defined as the time duration between two consecutive events that the right heel touches the ground. Although we can roughly count steps from Figure 3, we need a robust estimation scheme that works for various walking styles. In WifiU, we use the upper contour for the torso reflection to estimate the gait cycle time. The torso contour frequency ftc(t) is defined as: ftc(t) = max( f F(f,t) ∑ fmax 0 F(f,t) > γ ) (2) where F(f,t) is the magnitude of FFT at time t and frequency f . The summation of ∑ fmax 0 F(f,t) gives the total energy in the frequency range of 0 ∼ fmax. Thus, ftc(t) is the highest frequency that has an energy ratio larger than γ for the FFT result at time t. We set the threshold γ as 5% in our system. After smoothing the resulting ftc(t) with a low-pass filter, we get the torso contour speed vtc(t) as the thick line in Figure 4(a) using the equation vtc(t) = ftc(t)λ/2. Comparing the curve in Figure 4(a) with the spectrogram in Figure 3, we observe that our method reliably tracks the torso speed variations during the walking periods. The reason for choosing the contour rather than the frequency that has maximum energy is due to its robustness to noises. With a small amount of noise, as in the 3∼4 seconds period in Figure 3, the frequency with maximum energy, i.e., the “hottest” frequency for a given time, may change significantly and makes the measurement unreliable and discontinuous in time. We use the autocorrelation of the torso contour curve to robustly estimate gait cycle time, which is calculated as: R(τ) = ∑t (vtc(t)− µ) (vtc(t −τ)− µ) (3) where µ is the average speed of the torso contour. The autocorrelation is taken on the period of steady walking, which is defined as the period where the torso speed (calculated in Section 4.4) is no less than 80% of the maximum torso speed. By taking the autocorrelation, we can find the period of the curve by comparing the curve to a version of itself that is displaced by τ in time. This gives a better estimation than directly searching for peaks on the torso contour curve. Compared to the original torso contour curve in Figure 4(a), it is easier to find the peaks in the autocorrelation result in Figure 4(b). Each peak in autocorrelation means that the contour curve remains similar to the original version when displaced by a time of τ. As the gait cycle contains two footsteps, the torso speed varies twice in each gait cycle. So, the torso contour curve is similar when displaced by L/2, where L is the gait cycle time. In Figure 4(b), the first peak in the autocorrelation appears at the point τ = 0.57 second; thus, we obtain an estimation for the gait cycle time L = 2τ as 1.14 seconds. Estimating Torso and Leg Speeds We estimate the torso and leg speed using the percentile method developed for Doppler radars [27]. The percentile