Multi-Touch in the Air - Concurrent Micromovement Recognition Using RF Signals

This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination IEEE/ACM TRANSACTIONS ON NETWORKING Multi-Touch in the Air:Concurrent Micromovement Recognition Using RF Signals Lei Xie,Member,IEEE,Chuyu Wang,Student Member,IEEE,Alex X.Liu,Senior Member,IEEE, Jiangiang Sun,and Sanglu Lu,Member,IEEE Abstract-The human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards/touchpads to the brand-new approaches of performing interactions in the air.In this paper,we propose RF-glove,a sys- tem that recognizes concurrent multiple finger micromovement using RF signals,so as to realize the vision of "multi-touch in 1)Zoom In 2)Zoom OUT 3)Rotate Left4)Rotate Right (ZI) (RL) (RR) the air."It uses a commercial-off-the-shelf(COTS)RFID reader with three antennas and five COTS tags attached to the five fingers of a glove,one tag per finger.During the process of a user performing finger micromovements,we let the RFID reader continuously interrogate these tags and obtain the backscattered RF signals from each tag.For each antenna-tag pair,the reader 5)Flick 6)Swipe Left 7)Swipe Right 8)Punch (SL) (SR) obtains a sequence of RF phase values called a phase profile P門 from the tag's responses over time.To tradeoff between accuracy Fig.1.Example finger micromovements. and robustness in terms of matching resolution,we propose a two phase approach,including coarse-grained filtering and fine- grained matching.To tackle the variation of template phase in a more natural approach,such that the user can directly profiles at different positions,we propose a phase-model-based manipulate the virtual or real objects in the air.This paper con- solution to reconstruct the template phase profiles based on cerns multi-touch in the air,i.e.,the recognition of concurrent the exact locations.Experiment results show that we achieve micromovements using Radio Frequency(RF)signals in RFID an average accuracy of 92.1%under various moving speeds, systems [4]-9.In particular,we consider the concurrent orientation deviations,and so on. micromovements of multiple fingers such as zoom in/out, Index Terms-Passive RFID,RF Signal,micromovement rotate left/right,flick,swipe left/right and punch,as illustrated recognition,prototype design. in Fig.1.This is useful for many applications that requires human-computer interaction through fine-grained concurrent I.INTRODUCTION finger micromovements,such as motion sensing games.For example,a user can manipulate a virtual object with his finger A.Motivation micromovements,such as rotating or stretching the object. TOWADAYS,the human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards and touchpads to the brand-new B.Summary and Limitations of Prior Art approaches of performing interactions in the air.The users can Existing motion sensing technologies use either cameras perform the interactions with the computer using their arms, or sensors.Microsoft Kinect [1]and Leap Motion [3]con- legs and even fingers [1]-[3].In this way,the applications trollers use cameras to capture human motions based on vision of virtual reality and augmented reality can be supported technologies.The key limitation of camera based schemes is that they are more or less affected by the viewing angle Manuscript received December 13.2016:revised July 18.2017:accepted and light condition.Nintendo Wii [2]video game systems November 5,2017;approved by IEEE/ACM TRANSACTIONS ON NETWORK- ING Editor X.Zhou.This work was supported in part by the National Natural use wearable sensors based on infrared technologies.The key Science Foundation of China under Grant 61472185,Grant 61472184,Grant limitation of sensor based schemes is that the sensors are 61373129.Grant 61321491,and Grant 61502224.in part by the Jiangsu Nat- ural Science Foundation under Grant BK20151390,in part by the Fundamental often too big to be conveniently wear.Recently RF-IDraw Research Funds for the Central Universities under Grant 020214380035,in is proposed to use a 2-dimensional array of RFID antennas to part by the National Science Foundation under Grant CNS-1421407,in part by track the movement trajectory of a finger,which is attached the Jiangsu Innovation and Entrepreneurship (Shuangchuang)Program,and with an RFID tag [10].It constructs an efficient beam for in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization.(Corresponding authors:Alex X.Liu:Sanglu Lu.) detecting the finger moving direction by intersecting the beams The authors are with the State Key Laboratory for Novel Software of multiple antennas.However,RF-IDraw is designed to track Technology,Nanjing University.Nanjing 210023,China (e-mail: a fairly large range movement of one finger,e.g.,in the size of Ixie@nju.edu.cn: wangcyu217@dislab.nju.edu.cn; alexliu@nju.edu.cn: sunjq@dislab.nju.edu.cn:sanglu@nju.edu.cn). 20~30cm.It does not work well for tracking the concurrent Digital Object Identifier 10.1109/TNET.2017.2772781 movements of multiple fingers because its median accuracy 1063-66922017 IEEE.Personal use is permitted,but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE/ACM TRANSACTIONS ON NETWORKING 1 Multi-Touch in the Air: Concurrent Micromovement Recognition Using RF Signals Lei Xie , Member, IEEE, Chuyu Wang, Student Member, IEEE, Alex X. Liu, Senior Member, IEEE, Jianqiang Sun, and Sanglu Lu, Member, IEEE Abstract— The human–computer interactions have moved from the conventional approaches of entering inputs into the keyboards/touchpads to the brand-new approaches of performing interactions in the air. In this paper, we propose RF-glove, a sys￾tem that recognizes concurrent multiple finger micromovement using RF signals, so as to realize the vision of "multi-touch in the air." It uses a commercial-off-the-shelf (COTS) RFID reader with three antennas and five COTS tags attached to the five fingers of a glove, one tag per finger. During the process of a user performing finger micromovements, we let the RFID reader continuously interrogate these tags and obtain the backscattered RF signals from each tag. For each antenna–tag pair, the reader obtains a sequence of RF phase values called a phase profile from the tag’s responses over time. To tradeoff between accuracy and robustness in terms of matching resolution, we propose a two phase approach, including coarse-grained filtering and fine￾grained matching. To tackle the variation of template phase profiles at different positions, we propose a phase-model-based solution to reconstruct the template phase profiles based on the exact locations. Experiment results show that we achieve an average accuracy of 92.1% under various moving speeds, orientation deviations, and so on. Index Terms— Passive RFID, RF Signal, micromovement recognition, prototype design. I. INTRODUCTION A. Motivation NOWADAYS, the human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards and touchpads to the brand-new approaches of performing interactions in the air. The users can perform the interactions with the computer using their arms, legs and even fingers [1]–[3]. In this way, the applications of virtual reality and augmented reality can be supported Manuscript received December 13, 2016; revised July 18, 2017; accepted November 5, 2017; approved by IEEE/ACM TRANSACTIONS ON NETWORK￾ING Editor X. Zhou. This work was supported in part by the National Natural Science Foundation of China under Grant 61472185, Grant 61472184, Grant 61373129, Grant 61321491, and Grant 61502224, in part by the Jiangsu Nat￾ural Science Foundation under Grant BK20151390, in part by the Fundamental Research Funds for the Central Universities under Grant 020214380035, in part by the National Science Foundation under Grant CNS-1421407, in part by the Jiangsu Innovation and Entrepreneurship (Shuangchuang) Program, and in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization. (Corresponding authors: Alex X. Liu; Sanglu Lu.) The authors are with the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China (e-mail: lxie@nju.edu.cn; wangcyu217@dislab.nju.edu.cn; alexliu@nju.edu.cn; sunjq@dislab.nju.edu.cn; sanglu@nju.edu.cn). Digital Object Identifier 10.1109/TNET.2017.2772781 Fig. 1. Example finger micromovements. in a more natural approach, such that the user can directly manipulate the virtual or real objects in the air. This paper con￾cerns multi-touch in the air, i.e., the recognition of concurrent micromovements using Radio Frequency (RF) signals in RFID systems [4]–[9]. In particular, we consider the concurrent micromovements of multiple fingers such as zoom in/out, rotate left/right, flick, swipe left/right and punch, as illustrated in Fig. 1. This is useful for many applications that requires human-computer interaction through fine-grained concurrent finger micromovements, such as motion sensing games. For example, a user can manipulate a virtual object with his finger micromovements, such as rotating or stretching the object. B. Summary and Limitations of Prior Art Existing motion sensing technologies use either cameras or sensors. Microsoft Kinect [1] and Leap Motion [3] con￾trollers use cameras to capture human motions based on vision technologies. The key limitation of camera based schemes is that they are more or less affected by the viewing angle and light condition. Nintendo Wii [2] video game systems use wearable sensors based on infrared technologies. The key limitation of sensor based schemes is that the sensors are often too big to be conveniently wear. Recently RF-IDraw is proposed to use a 2-dimensional array of RFID antennas to track the movement trajectory of a finger, which is attached with an RFID tag [10]. It constructs an efficient beam for detecting the finger moving direction by intersecting the beams of multiple antennas. However, RF-IDraw is designed to track a fairly large range movement of one finger, e.g., in the size of 20∼30cm. It does not work well for tracking the concurrent movements of multiple fingers because its median accuracy 1063-6692 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information

This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination IEEE/ACM TRANSACTIONS ON NETWORKING is 3.7cm,which means that the accuracy of tracking two REID fingers could be 7.4cm,but finger movements are typically 2cm to 5cm.Furthermore,the deployment cost of RF-IDraw RFID is relatively expensive as it requires an antenna array of eight Antenna antennas and two RFID readers.Similarly,RFID localization eration schemes do not work well for recognizing concurrent multi- ane finger micromovements because the location accuracy is not Hands with multiple enough.For example,the state-of-the-art localization schemes RFID tags PinIt achieves an accuracy of 16cm at 90 percentile [11]. and Tagoram achieves an accuracy with a median error dis- tance of 6.35cm [12].In summary,the above RFID-based localization schemes,including RF-IDraw,are not suitable for micromovement recognition,as they mainly focus on the absolute tag positioning rather than the relative movement pattern of multiple tags.As a matter of fact,to achieve more accurate performance in the micromovement recognition,we should focus on the phase variation pattern caused by the micromovement of multiple fingers,instead of capturing the location variation of multiple fingers,since the former metric captures the micromovement in much more fine granularity than the latter. (b) Fig.2.System Overview.(a)Antenna deployment.(b)Tag deployment. C.Proposed Approach In this paper,we propose RF-Glove,a concurrent multi- finger micromovement recognition system based on RF sig- In other words,each different type of multi-finger micromove- nals.RF-Glove uses a commercial off-the-shelf(COTS)RFID ments can be characterized by different RF phase variations reader with 3 antennas and five EPCglobal C1G2 standard Thus,by capturing the distinguishing RF phase variation passive tags attached to the five fingers of a glove,one patterns,we can recognize different multi-finger micromove- tag per finger.The three antennas form two antenna pairs, ments.Our RF-micromovement model fundamentally explains which are placed in a mutually orthogonal manner on a flat why multi-finger micromovements can be recognized based on plane.Fig.2 shows the overview of our system with the 3 phase variations from RF signals. antennas deployed on the office ceiling.In performing multi- finger micromovements,we let the RFID reader continuously D.Technical Challenges and Solutions interrogate these tags and obtain the backscattered RF signals There are several technical challenges we need to address in from each tag.For each antenna-tag pair,the reader obtains a this paper.The first challenge is to properly tradeoff between sequence of RF phase values called a phase profile.For each accuracy and robustness in terms of matching resolution. type of multi-finger micromovement,we obtain a set of 3x 5 Given a testing set of phase profiles and a few template phase profiles.Given the phase profile set of a testing multi- sets of phase profiles for the micromovement,we need to finger micromovement,we compare the corresponding set of find the template set that the testing set matches the best. phase profiles with the templates of each type of multi-finger If the matching resolution is too high,then the matching micromovement to find the most similar template. robustness is too low due to the inherent unstableness in multi- To understand how RF signals vary with multi-finger micro- finger micromovements.If the matching resolution is too low, movements,in this paper,we propose a 3D positioning model then the matching accuracy is too low due to the inherent and a RF-micromovement model,respectively,to depict the common characteristics among different types of multi-finger relationship between the multi-finger movement and the RF- micromovements.To address this challenge,we propose a signals.Specifically,to recognize the large range movement, two-phase approach to this matching problem.In this first such as the swipe and punch,and locate the position of phase,we perform a coarse-grained filtering to identify some the hand during the small-range micromovement,such as the template sets that the testing set should not be matched to. zoom in/out and flick,we propose a 3D positioning model by referring to the moving status of the fingers and the to continuously locate the tags'positions for further micro- variation trend of the phase profile.In the second phase, movement recognition.To depict the relationship between we perform a fine-grained matching to match the testing set the phase variation and the multi-finger micromovement, to one of the remaining template sets,by referring to the we propose a RF-micromovement model that quantifies the details of phase profiles with time warping.Thus,we can use relationship between RF signals and micromovements.Our different matching resolutions to tradeoff between accuracy RF-micromovement model shows that RF phase variations and robustness. and multi-finger micromovement present a linear relationship, The second challenge is to tackle the variation of tem- when it is performed in the central beam of the antenna.plate phase profiles at different positions.It is observed that

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 2 IEEE/ACM TRANSACTIONS ON NETWORKING is 3.7cm, which means that the accuracy of tracking two fingers could be 7.4cm, but finger movements are typically 2cm to 5cm. Furthermore, the deployment cost of RF-IDraw is relatively expensive as it requires an antenna array of eight antennas and two RFID readers. Similarly, RFID localization schemes do not work well for recognizing concurrent multi- finger micromovements because the location accuracy is not enough. For example, the state-of-the-art localization schemes PinIt achieves an accuracy of 16cm at 90 percentile [11], and Tagoram achieves an accuracy with a median error dis￾tance of 6.35cm [12]. In summary, the above RFID-based localization schemes, including RF-IDraw, are not suitable for micromovement recognition, as they mainly focus on the absolute tag positioning rather than the relative movement pattern of multiple tags. As a matter of fact, to achieve more accurate performance in the micromovement recognition, we should focus on the phase variation pattern caused by the micromovement of multiple fingers, instead of capturing the location variation of multiple fingers, since the former metric captures the micromovement in much more fine granularity than the latter. C. Proposed Approach In this paper, we propose RF-Glove, a concurrent multi- finger micromovement recognition system based on RF sig￾nals. RF-Glove uses a commercial off-the-shelf (COTS) RFID reader with 3 antennas and five EPCglobal C1G2 standard passive tags attached to the five fingers of a glove, one tag per finger. The three antennas form two antenna pairs, which are placed in a mutually orthogonal manner on a flat plane. Fig. 2 shows the overview of our system with the 3 antennas deployed on the office ceiling. In performing multi- finger micromovements, we let the RFID reader continuously interrogate these tags and obtain the backscattered RF signals from each tag. For each antenna-tag pair, the reader obtains a sequence of RF phase values called a phase profile. For each type of multi-finger micromovement, we obtain a set of 3 × 5 phase profiles. Given the phase profile set of a testing multi- finger micromovement, we compare the corresponding set of phase profiles with the templates of each type of multi-finger micromovement to find the most similar template. To understand how RF signals vary with multi-finger micro￾movements, in this paper, we propose a 3D positioning model and a RF-micromovement model, respectively, to depict the relationship between the multi-finger movement and the RF￾signals. Specifically, to recognize the large range movement, such as the swipe and punch, and locate the position of the hand during the small-range micromovement, such as the zoom in/out and flick, we propose a 3D positioning model to continuously locate the tags’ positions for further micro￾movement recognition. To depict the relationship between the phase variation and the multi-finger micromovement, we propose a RF-micromovement model that quantifies the relationship between RF signals and micromovements. Our RF-micromovement model shows that RF phase variations and multi-finger micromovement present a linear relationship, when it is performed in the central beam of the antenna. Fig. 2. System Overview. (a) Antenna deployment. (b) Tag deployment. In other words, each different type of multi-finger micromove￾ments can be characterized by different RF phase variations. Thus, by capturing the distinguishing RF phase variation patterns, we can recognize different multi-finger micromove￾ments. Our RF-micromovement model fundamentally explains why multi-finger micromovements can be recognized based on phase variations from RF signals. D. Technical Challenges and Solutions There are several technical challenges we need to address in this paper. The first challenge is to properly tradeoff between accuracy and robustness in terms of matching resolution. Given a testing set of phase profiles and a few template sets of phase profiles for the micromovement, we need to find the template set that the testing set matches the best. If the matching resolution is too high, then the matching robustness is too low due to the inherent unstableness in multi- finger micromovements. If the matching resolution is too low, then the matching accuracy is too low due to the inherent common characteristics among different types of multi-finger micromovements. To address this challenge, we propose a two-phase approach to this matching problem. In this first phase, we perform a coarse-grained filtering to identify some template sets that the testing set should not be matched to, by referring to the moving status of the fingers and the variation trend of the phase profile. In the second phase, we perform a fine-grained matching to match the testing set to one of the remaining template sets, by referring to the details of phase profiles with time warping. Thus, we can use different matching resolutions to tradeoff between accuracy and robustness. The second challenge is to tackle the variation of tem￾plate phase profiles at different positions. It is observed that

This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination XIE et al:MULTI-TOUCH IN THE AIR:CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS when the human subject performs the micromovement at the are attached to a controlling ball to detect the motions of ball positions out of the central beams of the antennas,the phase rotation from users.Compared with our RF-Glove system, profiles for the same micromovement might be different to a the tags in Tagball follow the same movement trace where certain extent at different positions.Hence,it is inaccurate to the tags in RF-Glove may follow different movement traces. directly match the testing set of phase profiles to the original RF-IDraw [10]uses a 2-dimensional array of RFID antennas template set of phase profiles.To address this challenge,we to track the movement trajectory of one finger attached with propose a solution to reconstruct the template phase profiles an RFID tag,so that it can reconstruct the trajectory shape based on the exact locations.We first propose a 3D positioning of the specified finger.However,RF-IDraw is designed to method based on the AoA method to figure out the locations track a fairly large range movement of one finger,e.g.,in of multiple fingers.Based on the fingers'location,we propose the size of 20~30cm.It does not work well for tracking the a model to depict the relationship between the phase variation concurrent movements of multiple fingers because its median and the specified movement.We further derive the correspond- accuracy is 3.7cm,which means that the accuracy of tracking ing template phase profiles based on the exact locations. two fingers could be 7.4cm,but finger movements are typically We make four key contributions in this paper.First,we 2cm to 5cm.Furthermore,the deployment cost of RF-IDraw propose RF-Glove,an RF signal based concurrent micromove- is relatively expensive as it requires an antenna array of ment recognition system.Second,we propose a 3D position- eight antennas and two RFID readers.Different from the ing model and a RF-micromovement model,respectively,to positioning-based techniques from RF-IDraw,in this paper,to depict the relationship between the multi-finger movement and achieve more accurate performance in micromovement recog- the RF-signals.Third,we propose a phase profiling based nition,we directly investigate the phase variation pattern from approach to RF signal based multi-finger micromovement the concurrent micromovement of multiple fingers,instead of recognition.Last,we implemented RF-Glove using COTS capturing the location variation of multiple fingers,since the RFID systems and evaluated its performance in realistic set- former metric captures the micromovement in much more fine tings.Experiment results show that we achieve an average granularity than the latter. accuracy of 92.1%under various moving speeds,orientation deviations.etc. III.MODELING RF SIGNAL VARIATIONS AND MULTI-FINGER MICROMOVEMENTS II.RELATED WORK Like the functionalities of the general purpose touch pad, the scheme of"multi-touch in the air"should also have both RFID-Based Localization:Prior work on RFID-based local- the positioning and gesture-recognition functionalities.The ization primarily rely on RSSI (Received Signal Strength) positioning functionality aims to locate the tagged fingers information [13],[14]to acquire the absolute location of an in a coarse-grained manner.In this way,we are able to object.State-of-the-art systems use phase value to estimate the easily recognize the large-range movement of the tagged absolute location of an object with higher accuracy [11],[12], fingers caused by the arm movement.The gesture-recognition [15]-[21].By deploying multiple antennas and measuring the functionality aims to further recognize the micromovement of phase difference between the received signals at different multiple tagged fingers in a fine-grained manner. antennas,these systems can effectively reduce the localization Therefore,to understand how RF-signals vary with large- error to a few centimeters.Further,PinIt exploits multi-path range movement,we propose a 3D positioning model that effect to accurately locate RFIDs by using synthetic aperture quantifies the relationship between the RF signal and the radar created via antenna motion to extract multi-path profiles position of tagged fingers in the 3-dimensional space.To for accurate localization [11].Tagoram exploits tag mobility to understand how RF signals vary with multi-finger micro- build a virtual antenna array,and uses differential augmented movements,we propose an RF micromovement model that hologram to facilitate the instant tracking of a mobile RFID quantifies the relationship between RF signals and multi-finger tag [12].While the above work mainly focuses on absolute micromovements.It shows that each different type of multi- object localization.Spatial-Temporal Phase Profiling (STPP) finger micromovements can be characterized by different RF is proposed for the relative localization of RFID tags [22]. phase variation patterns.Thus,by capturing the distinguishing Liu et al.[20]propose a pose sensing system called Tag- RF phase variation patterns,we can recognize different multi- Compass that uses a single tag to determine the orientation finger micromovements. as well as the position of the associated object.A completely We use the commercial RFID reader ImpinJ R420 and Laird different method based on the polarization properties of the S9028 antenna to receive RF signals.Laird S9028 antenna pro- RF waves is exploited to achieve fine-grained pose sensing. vides a consistent and continuous reading zone with circular RFID-Based Motion Tracking:Prior activity sensing sys- polarization.As shown in Figure 2(a),we deploy three anten- tems propose various approaches to recognize gestures for nas on the room ceiling,say A,B and C.The antenna pair activity sensing.These systems can be primarily classified into AB and AC are deployed in a mutually orthogonal fashion vision-based,infrared-based,electric field-based and wearable along the X-axis and Y-axis,respectively.By leveraging the approaches [23]-[25].RFID systems have recently been used Angle of Arrival (AoA)positioning method,the AoA from for trajectory tracking [10],[26].[27]and motion tracking the antenna pair AB can differentiate the movement along the [28]-[32].Lin et al.[29]proposed a 3D human-computer X-axis,while the AoA from the antenna pair AC can differ- interaction system called Tagball,where multiple passive tags entiate the movement along the Y-axis.We use Alien 9640

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. XIE et al.: MULTI-TOUCH IN THE AIR: CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 3 when the human subject performs the micromovement at the positions out of the central beams of the antennas, the phase profiles for the same micromovement might be different to a certain extent at different positions. Hence, it is inaccurate to directly match the testing set of phase profiles to the original template set of phase profiles. To address this challenge, we propose a solution to reconstruct the template phase profiles based on the exact locations. We first propose a 3D positioning method based on the AoA method to figure out the locations of multiple fingers. Based on the fingers’ location, we propose a model to depict the relationship between the phase variation and the specified movement. We further derive the correspond￾ing template phase profiles based on the exact locations. We make four key contributions in this paper. First, we propose RF-Glove, an RF signal based concurrent micromove￾ment recognition system. Second, we propose a 3D position￾ing model and a RF-micromovement model, respectively, to depict the relationship between the multi-finger movement and the RF-signals. Third, we propose a phase profiling based approach to RF signal based multi-finger micromovement recognition. Last, we implemented RF-Glove using COTS RFID systems and evaluated its performance in realistic set￾tings. Experiment results show that we achieve an average accuracy of 92.1% under various moving speeds, orientation deviations, etc. II. RELATED WORK RFID-Based Localization: Prior work on RFID-based local￾ization primarily rely on RSSI (Received Signal Strength) information [13], [14] to acquire the absolute location of an object. State-of-the-art systems use phase value to estimate the absolute location of an object with higher accuracy [11], [12], [15]–[21]. By deploying multiple antennas and measuring the phase difference between the received signals at different antennas, these systems can effectively reduce the localization error to a few centimeters. Further, PinIt exploits multi-path effect to accurately locate RFIDs by using synthetic aperture radar created via antenna motion to extract multi-path profiles for accurate localization [11]. Tagoram exploits tag mobility to build a virtual antenna array, and uses differential augmented hologram to facilitate the instant tracking of a mobile RFID tag [12]. While the above work mainly focuses on absolute object localization, Spatial-Temporal Phase Profiling (STPP) is proposed for the relative localization of RFID tags [22]. Liu et al. [20] propose a pose sensing system called Tag￾Compass that uses a single tag to determine the orientation as well as the position of the associated object. A completely different method based on the polarization properties of the RF waves is exploited to achieve fine-grained pose sensing. RFID-Based Motion Tracking: Prior activity sensing sys￾tems propose various approaches to recognize gestures for activity sensing. These systems can be primarily classified into vision-based, infrared-based, electric field-based and wearable approaches [23]–[25]. RFID systems have recently been used for trajectory tracking [10], [26], [27] and motion tracking [28]–[32]. Lin et al. [29] proposed a 3D human-computer interaction system called Tagball, where multiple passive tags are attached to a controlling ball to detect the motions of ball rotation from users. Compared with our RF-Glove system, the tags in Tagball follow the same movement trace where the tags in RF-Glove may follow different movement traces. RF-IDraw [10] uses a 2-dimensional array of RFID antennas to track the movement trajectory of one finger attached with an RFID tag, so that it can reconstruct the trajectory shape of the specified finger. However, RF-IDraw is designed to track a fairly large range movement of one finger, e.g., in the size of 20∼30cm. It does not work well for tracking the concurrent movements of multiple fingers because its median accuracy is 3.7cm, which means that the accuracy of tracking two fingers could be 7.4cm, but finger movements are typically 2cm to 5cm. Furthermore, the deployment cost of RF-IDraw is relatively expensive as it requires an antenna array of eight antennas and two RFID readers. Different from the positioning-based techniques from RF-IDraw, in this paper, to achieve more accurate performance in micromovement recog￾nition, we directly investigate the phase variation pattern from the concurrent micromovement of multiple fingers, instead of capturing the location variation of multiple fingers, since the former metric captures the micromovement in much more fine granularity than the latter. III. MODELING RF SIGNAL VARIATIONS AND MULTI-FINGER MICROMOVEMENTS Like the functionalities of the general purpose touch pad, the scheme of “multi-touch in the air” should also have both the positioning and gesture-recognition functionalities. The positioning functionality aims to locate the tagged fingers in a coarse-grained manner. In this way, we are able to easily recognize the large-range movement of the tagged fingers caused by the arm movement. The gesture-recognition functionality aims to further recognize the micromovement of multiple tagged fingers in a fine-grained manner. Therefore, to understand how RF-signals vary with large￾range movement, we propose a 3D positioning model that quantifies the relationship between the RF signal and the position of tagged fingers in the 3-dimensional space. To understand how RF signals vary with multi-finger micro￾movements, we propose an RF micromovement model that quantifies the relationship between RF signals and multi-finger micromovements. It shows that each different type of multi- finger micromovements can be characterized by different RF phase variation patterns. Thus, by capturing the distinguishing RF phase variation patterns, we can recognize different multi- finger micromovements. We use the commercial RFID reader ImpinJ R420 and Laird S9028 antenna to receive RF signals. Laird S9028 antenna pro￾vides a consistent and continuous reading zone with circular polarization. As shown in Figure 2(a), we deploy three anten￾nas on the room ceiling, say A, B and C. The antenna pair AB and AC are deployed in a mutually orthogonal fashion along the X-axis and Y -axis, respectively. By leveraging the Angle of Arrival (AoA) positioning method, the AoA from the antenna pair AB can differentiate the movement along the X-axis, while the AoA from the antenna pair AC can differ￾entiate the movement along the Y -axis. We use Alien 9640

This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination IEEE/ACM TRANSACTIONS ON NETWORKING general-purpose tags,which are EPC C1G2 standards compli- ant.We attach five RFID tags to the five fingers of a glove,one tag per finger,as shown in Figure 2(b).In performing multi- P(X,0,) finger micromovements,we let the RFID reader continuously 08i0,0,0) interrogate these tags and obtain RF signals from each tag via three antennas. Operation Plane 口0' A.3D Positioning Model p-0,0,h (x,y,-h) When the human subject performs the multi-touch gesture in the air with the tagged fingers,he/she usually performs the Fig.3.The position of the tag P on the operation plane. following two kinds of movement:1)Large-range movement: the human subject performs the movement with fairly large range in the 3-dimensional space,such as swipe left/right and punch.The moving range is usually greater than half of the wave length (i.e.,about 17cm)so that the phase changes of the RFID tags exceed a complete period.2)Small-range micromovement:the human subject performs the movement with very small range in the 3-dimensional space,such as zoom in/out,rotate left/right,and flick.The moving range is less than half of the wave length(i.e.,about 17cm)so that the 10 phase changes of the RFID tags does not exceed a complete X axis period. Fig.4.The hyperbola:the intersection between the conical surface and the Therefore,for the large-range movement,since the phase operation plane. change exceeds a complete period,it is neither accurate nor necessary to recognize the movement by phase changes. Instead,we can leverage the 3D positioning method to effec- as (,y,-h).Thus POll =vx2 +2+h2.Since the angle tively recognize the large-range movement.For the small- ∠POP'=a,then range micromovement,since the position change of the tagged ‖PO=IPOll cosa. (1) fingers is rather small (it is usually less than 5cm).thus we rely on the phase changes to recognize the small-range micromove- Hence,as P'Ol =Eq.(1)is equivalent to ment.Nevertheless,the phase changes of the micromovement l=Vx2+y2+h2.cosa. (2) also depends on the exact position of the tagged fingers.For example,the phase changes of the same type of micromove- Therefore, ment may vary to a certain extent when it is performed at sin2a·x2-cos2a·y2=h2.cos2a. different positions of the 3-dimensional space.In summary, (3) 3D positioning is essential in recognizing both the large-range It implies that the feasible solution of P on the operational and small-range movement. plane is a hyperbola.We further illustrate the above conclu- Suppose we can build a 3D coordinate system according to sion with an example as shown in Fig.4.Since the angle the operation plane,as shown in Fig.3.The antenna pair is POP'=a,and OP'is collinear with the X-axis,so the deployed along the X-axis,while the origin O is set to the possible trace of P in the 3-dimensional space can be denoted center of the antenna pair.The X-axis and Y-axis are mutually as a conical surface originated from the point O.When the orthogonal and parallel to the operation plane,and the Z-axis conical surface intersects with the operation plane,the possible is orthogonal to the operation plane.Assume that a specified trace of P forms a hyperbola on the operation plane. tag is denoted as a point P=(,y,z)on the operation plane, Suppose the human subject is performing the micromove- the projection of the point P on the X-axis is P.As in ment on the same operation plane,more or less.That is conventional operations of multi-touch in the air,the tagged to say,the distance h between the operation plane and the fingers of the human subjects are separated with a fairly large antenna plane keeps almost constant.As we deploy two mutu- distance (e.g.,150cm~200cm)to the three antennas,whereas ally orthogonal antenna pairs along the X-axis and Y-axis, the antennas are separated with a limited distance (e.g.,20cm respectively,then,according to the two antenna pairs,the ~30cm)to each other,thus we can leverage the Angle of feasible solutions of the tag's position can be estimated as two Arrival(AoA)method to figure out the direction of the tag in hyperbolas intersecting on the operation plane.Thus,we can the 3D space.Then,according to the AoA method,we can estimate the position of the tag by computing the intersections estimate the angle between OP and OP/as a.Assume that of the two hyperbolas.Fig.5 shows an example of positioning the projection of the origin O on the operation plane is O', the tag by computing the intersections between two hyperbolas then the distance between the antenna plane and the operation on the operation plane.Here,the antennas are separated with a plane is h=OO'l=-z.Therefore,the coordinate of horizontal/vertical distance of 20cm,and the distance between O'is (0,0,-h),the coordinate of P can be also denoted the operation plane and the antenna plane is set to 100cm

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 4 IEEE/ACM TRANSACTIONS ON NETWORKING general-purpose tags, which are EPC C1G2 standards compli￾ant. We attach five RFID tags to the five fingers of a glove, one tag per finger, as shown in Figure 2(b). In performing multi- finger micromovements, we let the RFID reader continuously interrogate these tags and obtain RF signals from each tag via three antennas. A. 3D Positioning Model When the human subject performs the multi-touch gesture in the air with the tagged fingers, he/she usually performs the following two kinds of movement: 1) Large-range movement: the human subject performs the movement with fairly large range in the 3-dimensional space, such as swipe left/right and punch. The moving range is usually greater than half of the wave length (i.e., about 17cm) so that the phase changes of the RFID tags exceed a complete period. 2) Small-range micromovement: the human subject performs the movement with very small range in the 3-dimensional space, such as zoom in/out, rotate left/right, and flick. The moving range is less than half of the wave length (i.e., about 17cm) so that the phase changes of the RFID tags does not exceed a complete period. Therefore, for the large-range movement, since the phase change exceeds a complete period, it is neither accurate nor necessary to recognize the movement by phase changes. Instead, we can leverage the 3D positioning method to effec￾tively recognize the large-range movement. For the small￾range micromovement, since the position change of the tagged fingers is rather small (it is usually less than 5cm), thus we rely on the phase changes to recognize the small-range micromove￾ment. Nevertheless, the phase changes of the micromovement also depends on the exact position of the tagged fingers. For example, the phase changes of the same type of micromove￾ment may vary to a certain extent when it is performed at different positions of the 3-dimensional space. In summary, 3D positioning is essential in recognizing both the large-range and small-range movement. Suppose we can build a 3D coordinate system according to the operation plane, as shown in Fig.3. The antenna pair is deployed along the X-axis, while the origin O is set to the center of the antenna pair. The X-axis and Y -axis are mutually orthogonal and parallel to the operation plane, and the Z-axis is orthogonal to the operation plane. Assume that a specified tag is denoted as a point P = (x, y, z) on the operation plane, the projection of the point P on the X-axis is P . As in conventional operations of multi-touch in the air, the tagged fingers of the human subjects are separated with a fairly large distance (e.g., 150cm∼200cm) to the three antennas, whereas the antennas are separated with a limited distance (e.g., 20cm ∼30cm) to each other, thus we can leverage the Angle of Arrival (AoA) method to figure out the direction of the tag in the 3D space. Then, according to the AoA method, we can estimate the angle between OP and OP as α. Assume that the projection of the origin O on the operation plane is O , then the distance between the antenna plane and the operation plane is h = OO = −z. Therefore, the coordinate of O is (0, 0, −h), the coordinate of P can be also denoted Fig. 3. The position of the tag P on the operation plane. Fig. 4. The hyperbola: the intersection between the conical surface and the operation plane. as (x, y, −h). Thus P O = x2 + y2 + h2. Since the angle ∠POP = α, then P O = P O · cos α. (1) Hence, as P O = |x|, Eq.(1) is equivalent to |x| = x2 + y2 + h2 · cos α. (2) Therefore, sin2 α · x2 − cos2 α · y2 = h2 · cos2 α. (3) It implies that the feasible solution of P on the operational plane is a hyperbola. We further illustrate the above conclu￾sion with an example as shown in Fig.4. Since the angle ∠POP = α, and OP is collinear with the X-axis, so the possible trace of P in the 3-dimensional space can be denoted as a conical surface originated from the point O. When the conical surface intersects with the operation plane, the possible trace of P forms a hyperbola on the operation plane. Suppose the human subject is performing the micromove￾ment on the same operation plane, more or less. That is to say, the distance h between the operation plane and the antenna plane keeps almost constant. As we deploy two mutu￾ally orthogonal antenna pairs along the X-axis and Y -axis, respectively, then, according to the two antenna pairs, the feasible solutions of the tag’s position can be estimated as two hyperbolas intersecting on the operation plane. Thus, we can estimate the position of the tag by computing the intersections of the two hyperbolas. Fig.5 shows an example of positioning the tag by computing the intersections between two hyperbolas on the operation plane. Here, the antennas are separated with a horizontal/vertical distance of 20cm, and the distance between the operation plane and the antenna plane is set to 100cm.

This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination. XIE et al:MULTI-TOUCH IN THE AIR:CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 60 Yaxis Contour lines of 一Hyperbola 1 `、phase values -Hyperbola 2 Target position Antenna position 、、 X axis (xsys,zs)\ ds Ny(xeye,Ze） de 0 1 /Antenna 例 △d Z axis 20 Fig.7.The relationship between△dand△z. Fig.5.Tag positioning:the intersection between two hyperbolas on the variation of phase value A from adjacent phase values as operation plane. follows,where term u=0r+0R+TAc is canceled out: △0= ·×2△d)mod2π (5) Here,Ad is the variation of distance between the tag and the antenna.If the movement Ad is smaller than half a wavelength ≥，ie,about 16.4cm,then, Operation Plane △8= 2×△d ×2π」 (6) X V 入 Thus,we can derive that the movement of the tag towards the antenna is△d=六△， Fig.6.Estimate the parameter h by moving the hand linearly with distance d. For our system deployment,we deploy three antennas,say A,B and C.on the office ceiling,as shown in Fig.2(a).To To figure out the hyperbolas of the specified tag,it is depict the micromovement of the tagged fingers,it is essential essential to estimate the distance h.To estimate the para- to build a 3-dimensional coordinate system.Without loss of meter h,we can let the human subject perform a specified generality,we build a coordinate system by setting the center movement,e.g.,performing the push/pull movement with a of antenna A as the origin,as shown in Fig.7.Then,for specified distance d from time t to t'.Then,according to the antenna A,let Ad be the change of the distance between the distance d,we can enumerate all feasible values of h,and tag and the antenna,.and let△x,△y,and△e be the change of compute the intersection point P(t)and P(t')between the positions for the tag along the three dimensions X.Y,and Z, hyperbolas,respectively,at the start time and end time of the respectively.Suppose a tag moves from position (s,4s,2)to movement.We finally determine the estimate of h when the (re:e,ze),then△d=V√径+y+z径-√g+y+z径，and corresponding moving distance is most close to d.Fig.6 shows Az=ze-2s.In our system,we require the tagged fingers to an example of estimating the parameter h by moving the hand be separated with the antennas with a fairly large distance of linearly with a specified distance d. more than 150cm.Suppose we let the glove attached with five tags operate within the central beam area of the antenna.Then, B.RF-Micromovement Model we rely on the following theorem to depict the relationship between△dand△z. The phase value of an RF signal describes the degree that Theorem 1:Considering a 3-dimensional space with the received signal offsets from the sent signal,ranging from antenna A as the origin,suppose a tag moves from position 0 to 360 degrees.Let d be the distance between the RFID (s,ys,2s)to (e,ye,ze),then,considering antenna A,we antenna and the tag,the signal traverses a round-trip with a define the change of the distance between the tag and the distance of 2d in each backscatter communication.Thus,the antenna△d=vc+y+2径-√rg+y好+2径，and the phase value 0 output by an RFID reader can be expressed as: change of positions for the tag along the dimension X,Y and ×2d+4）mod2π (4) Zare△x=re-ra,△y=ye-ys,and△z=ze-zs: respectively.If each tag moves within its central beam,i.e., where A is the wave length.Besides the RF phase rotation 2s s,2s>ys.ze e,and ze ye,then.Ad is over distance,the reader's transmitter,the tag's reflection approximately equal to Az. characteristic,and the reader's receiver will also introduce Proof:According to the definition of△dand△z, some additional phase rotation,denoted as er,OR and rAc respectively.We use u=0T+0R+rAc to denote this △d 哈+呢+2是-+班+图 (7) diversity term in Equation(1). △z 2e-2s As we focus on the variation of tag positions rather than x2+呢+是-x?++2 (8) absolute positions,for each antenna-tag pair,we compute the (2e-)·(√哈+经+径+√号+好+)

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. XIE et al.: MULTI-TOUCH IN THE AIR: CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 5 Fig. 5. Tag positioning: the intersection between two hyperbolas on the operation plane. Fig. 6. Estimate the parameter h by moving the hand linearly with distance d. To figure out the hyperbolas of the specified tag, it is essential to estimate the distance h. To estimate the para￾meter h, we can let the human subject perform a specified movement, e.g., performing the push/pull movement with a specified distance d from time t to t . Then, according to the distance d, we can enumerate all feasible values of h, and compute the intersection point P(t) and P(t ) between the hyperbolas, respectively, at the start time and end time of the movement. We finally determine the estimate of h when the corresponding moving distance is most close to d. Fig.6 shows an example of estimating the parameter h by moving the hand linearly with a specified distance d. B. RF-Micromovement Model The phase value of an RF signal describes the degree that the received signal offsets from the sent signal, ranging from 0 to 360 degrees. Let d be the distance between the RFID antenna and the tag, the signal traverses a round-trip with a distance of 2d in each backscatter communication. Thus, the phase value θ output by an RFID reader can be expressed as: θ = (2π λ × 2d + μ) mod 2π, (4) where λ is the wave length. Besides the RF phase rotation over distance, the reader’s transmitter, the tag’s reflection characteristic, and the reader’s receiver will also introduce some additional phase rotation, denoted as θT , θR and θT AG respectively. We use μ = θT + θR + θT AG to denote this diversity term in Equation (1). As we focus on the variation of tag positions rather than absolute positions, for each antenna-tag pair, we compute the Fig. 7. The relationship between Δd and Δz. variation of phase value Δθ from adjacent phase values as follows, where term μ = θT + θR + θT AG is canceled out: Δθ = (2π λ × 2Δd) mod 2π. (5) Here, Δd is the variation of distance between the tag and the antenna. If the movement Δd is smaller than half a wavelength λ 2 , i.e., about 16.4cm, then, Δθ = 2 × Δd λ × 2π. (6) Thus, we can derive that the movement of the tag towards the antenna is Δd = λ 4πΔθ. For our system deployment, we deploy three antennas, say A, B and C, on the office ceiling, as shown in Fig.2(a). To depict the micromovement of the tagged fingers, it is essential to build a 3-dimensional coordinate system. Without loss of generality, we build a coordinate system by setting the center of antenna A as the origin, as shown in Fig.7. Then, for antenna A, let Δd be the change of the distance between the tag and the antenna, and let Δx, Δy, and Δz be the change of positions for the tag along the three dimensions X, Y , and Z, respectively. Suppose a tag moves from position (xs, ys, zs) to (xe, ye, ze), then Δd = x2 e + y2 e + z2 e− x2 s + y2 s + z2 s , and Δz = ze − zs. In our system, we require the tagged fingers to be separated with the antennas with a fairly large distance of more than 150cm. Suppose we let the glove attached with five tags operate within the central beam area of the antenna. Then, we rely on the following theorem to depict the relationship between Δd and Δz. Theorem 1: Considering a 3-dimensional space with antenna A as the origin, suppose a tag moves from position (xs, ys, zs) to (xe, ye, ze), then, considering antenna A, we define the change of the distance between the tag and the antenna Δd = x2 e + y2 e + z2 e − x2 s + y2 s + z2 s , and the change of positions for the tag along the dimension X, Y and Z are Δx = xe − xs, Δy = ye − ys, and Δz = ze − zs, respectively. If each tag moves within its central beam, i.e., zs  xs, zs  ys, ze  xe, and ze  ye, then, Δd is approximately equal to Δz. Proof: According to the definition of Δd and Δz, Δd Δz = x2 e + y2 e + z2 e − x2 s + y2 s + z2 s ze − zs (7) = x2 e + y2 e + z2 e − x2 s + y2 s + z2 s (ze − zs) · ( x2 e + y2 e + z2 e + x2 s + y2 s + z2 s ) (8)