计算机科学与技术（参考文献）RF-Dial - Rigid Motion Tracking and Touch Gesture Detection for Interaction via RFID Tags

IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 RF-Dial:Rigid Motion Tracking and Touch Gesture Detection for Interaction via RFId Tags Yanling Bu,Student Member,IEEE,Lei Xie,Member,IEEE,Yinyin Gong,Chuyu Wang,Member,IEEE, Lei Yang,Member,IEEE,Jia Liu,Member,IEEE,and Sanglu Lu,Member,IEEE Abstract-With the rising of demands for novel human-computer interaction approaches in the 2D plane,a number of intelligent devices come into being.For example,Microsoft Surface Dial supports simple clicks and rotations for the interaction with computer. However,these approaches are dedicated devices,and they might require batteries or have limited functions.In this paper,we propose RF-Dial to realize a light-weight,battery-free and functional 2D human-computer interaction solution via commercial off-the-shelf (COTS)passive RFID tags.What RF-Dial shines is that it can easily turn an ordinary object,e.g.,a board eraser,into an intelligent interaction device.By deploying a tag array on the side face of the object together with a dipole tag on the top face,RF-Dial cannot only track the rigid motion of the object but also detect the touch gesture of a user on the surface of the object,including translation, rotation,click,press and hold,and swipe.To do the motion tracking,RF-Dial builds a phase-based model that captures the translation and the rotation of the tagged object simultaneously,by jointly exploiting the information of phase variations and the topology of the tag array.To detect the touch gesture,RF-Dial builds an RSSI-based model that uses the impact of the touching finger on the tag antenna's impedance to estimate the touch position in real time,which is robust to environmental factors like position or orientation.We implemented a prototype of RF-Dial with commodity RFID devices.Extensive experiments show that RF-Dial achieves an accurate rigid motion tracking,with a small error of 0.6cm for the translation tracking,and a small error of 1.9 degrees for the rotation estimation. Besides,RF-Dial can also detect the touch gesture accurately,as the 90 percent of touch position errors are less than 2.09mm. Index Terms-RFID,human-computer interaction,tag array,translation,rotation,coupling effect,touch gesture. ◆ INTRODUCTION TN modern times,the widely used approaches for the novel HCI designs [2-18].RFID can even work in the non- human-computer interaction (HCI)are operated in the line-of-sight situation due to its backscatter communication. 2D plane,like the touch screen and the mouse.By moving Therefore,we hope to use RFID to answer such a question: or stroking these interaction devices,users can access the "Is it possible to design a battery-free and light-weight solution objects in the computer and manipulate them conveniently.to the 2D human-computer interaction,thereby even an ordinary With the rise of the computer aided art design and other object can be easily turned into an intelligent interaction device?" novel applications,a number of intelligent devices have In this paper,we propose RF-Dial to realize a novel come into being as the response to the demand for brand-2D human-computer interaction solution via COTS passive new 2D interaction solutions.For example,Microsoft Sur- RFID tags.We attach a tag array to the side face together face Dial [1]emerged in 2016,supporting simple clicks and with one tag on the top face of an object,denote them as rotations for the natural and friendly interaction. movement tags and the touch tag,respectively.As shown The latest HCI approaches are mainly based on the in Fig.1,we deploy two RFID antennas orthogonally to computer vision or sensors.For computer vision-based realize our vision.Specifically,we continuously track the approaches,they use cameras to monitor the movement rigid motion of the tagged object with movement tags,in- of limbs or fingers.However,they are mainly limited by cluding the translation and the rotation simultaneously,and privacy concerns,the light condition and the viewing an- detect the touch gesture with the touch tag,including the gle.For sensor-based approaches,they use commercial- click,the press and hold,and the swipe.In this way,an off-the-shelf (COTS)sensors like inertial sensors to track ordinary object such as a candy box can be turned into an the movement of devices.Their main constraints are the intelligent interaction device.For example,we can realize limited battery life and the high hardware cost.Thankfully, an functional drawing application with RF-Dial.It tracks RFID provides the battery-free sensing technology to enable the translation of movement tags to draw lines,and adjusts the line color automatically by the rotation just during the drawing process;The touch tag functions as buttons and Yanling Bu,Lei Xie,Yinyin Gong,Chuyu Wang,Jia Liu,and Sanglu Lu are with the State Key Laboratory for Novel Softiare Technology,Nanjing sliders,receiving the user's commands,i.e.,adjusting the University,China. line color and width,as shown in the case study (Sec- E-mail: yanling@smail.nju.edu.cn, Ixie@nju.edu.cn, yy-tion 9.4).Technically,based on RF-signals from movement gong@dislab.nju.edu.cn, chuyu@nju.edu.cn, jialiu@nju.edu.cn, tags,we build a rigid transformation model to reflect the sanglu@nju.edu.cn. Lei Yang is with the Department of Computing,The Hong Kong Poly- relationship between the motion of the tagged object and technic University,Hong Kong,China. the corresponding phase variations of each movement tag in E-mail:young @tagsys.org. the tag array.As the movement tags form a tag array with .Lei Xie is the corresponding author. the fixed topology,we can derive the translation and the

IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 1 RF-Dial: Rigid Motion Tracking and Touch Gesture Detection for Interaction via RFID Tags Yanling Bu, Student Member, IEEE, Lei Xie, Member, IEEE, Yinyin Gong, Chuyu Wang, Member, IEEE, Lei Yang, Member, IEEE, Jia Liu, Member, IEEE, and Sanglu Lu, Member, IEEE Abstract—With the rising of demands for novel human-computer interaction approaches in the 2D plane, a number of intelligent devices come into being. For example, Microsoft Surface Dial supports simple clicks and rotations for the interaction with computer. However, these approaches are dedicated devices, and they might require batteries or have limited functions. In this paper, we propose RF-Dial to realize a light-weight, battery-free and functional 2D human-computer interaction solution via commercial off-the-shelf (COTS) passive RFID tags. What RF-Dial shines is that it can easily turn an ordinary object, e.g., a board eraser, into an intelligent interaction device. By deploying a tag array on the side face of the object together with a dipole tag on the top face, RF-Dial cannot only track the rigid motion of the object but also detect the touch gesture of a user on the surface of the object, including translation, rotation, click, press and hold, and swipe. To do the motion tracking, RF-Dial builds a phase-based model that captures the translation and the rotation of the tagged object simultaneously, by jointly exploiting the information of phase variations and the topology of the tag array. To detect the touch gesture, RF-Dial builds an RSSI-based model that uses the impact of the touching finger on the tag antenna’s impedance to estimate the touch position in real time, which is robust to environmental factors like position or orientation. We implemented a prototype of RF-Dial with commodity RFID devices. Extensive experiments show that RF-Dial achieves an accurate rigid motion tracking, with a small error of 0.6cm for the translation tracking, and a small error of 1.9 degrees for the rotation estimation. Besides, RF-Dial can also detect the touch gesture accurately, as the 90 percent of touch position errors are less than 2.09mm. Index Terms—RFID, human-computer interaction, tag array, translation, rotation, coupling effect, touch gesture. ✦ 1 INTRODUCTION I N modern times, the widely used approaches for the human-computer interaction (HCI) are operated in the 2D plane, like the touch screen and the mouse. By moving or stroking these interaction devices, users can access the objects in the computer and manipulate them conveniently. With the rise of the computer aided art design and other novel applications, a number of intelligent devices have come into being as the response to the demand for brand￾new 2D interaction solutions. For example, Microsoft Sur￾face Dial [1] emerged in 2016, supporting simple clicks and rotations for the natural and friendly interaction. The latest HCI approaches are mainly based on the computer vision or sensors. For computer vision-based approaches, they use cameras to monitor the movement of limbs or fingers. However, they are mainly limited by privacy concerns, the light condition and the viewing an￾gle. For sensor-based approaches, they use commercial￾off-the-shelf (COTS) sensors like inertial sensors to track the movement of devices. Their main constraints are the limited battery life and the high hardware cost. Thankfully, RFID provides the battery-free sensing technology to enable • Yanling Bu, Lei Xie, Yinyin Gong, Chuyu Wang, Jia Liu, and Sanglu Lu are with the State Key Laboratory for Novel Software Technology, Nanjing University, China. E-mail: yanling@smail.nju.edu.cn, lxie@nju.edu.cn, yy￾gong@dislab.nju.edu.cn, chuyu@nju.edu.cn, jialiu@nju.edu.cn, sanglu@nju.edu.cn. • Lei Yang is with the Department of Computing, The Hong Kong Poly￾technic University, Hong Kong, China. E-mail: young@tagsys.org. • Lei Xie is the corresponding author. novel HCI designs [2–18]. RFID can even work in the non￾line-of-sight situation due to its backscatter communication. Therefore, we hope to use RFID to answer such a question: “Is it possible to design a battery-free and light-weight solution to the 2D human-computer interaction, thereby even an ordinary object can be easily turned into an intelligent interaction device?”. In this paper, we propose RF-Dial to realize a novel 2D human-computer interaction solution via COTS passive RFID tags. We attach a tag array to the side face together with one tag on the top face of an object, denote them as movement tags and the touch tag, respectively. As shown in Fig. 1, we deploy two RFID antennas orthogonally to realize our vision. Specifically, we continuously track the rigid motion of the tagged object with movement tags, in￾cluding the translation and the rotation simultaneously, and detect the touch gesture with the touch tag, including the click, the press and hold, and the swipe. In this way, an ordinary object such as a candy box can be turned into an intelligent interaction device. For example, we can realize an functional drawing application with RF-Dial. It tracks the translation of movement tags to draw lines, and adjusts the line color automatically by the rotation just during the drawing process; The touch tag functions as buttons and sliders, receiving the user’s commands, i.e., adjusting the line color and width, as shown in the case study (Sec￾tion 9.4). Technically, based on RF-signals from movement tags, we build a rigid transformation model to reflect the relationship between the motion of the tagged object and the corresponding phase variations of each movement tag in the tag array. As the movement tags form a tag array with the fixed topology, we can derive the translation and the

IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 Tagged Daily absolute touch position,we observe that the RSSI deviation during the swipe is position-independent and orientation- insensitive.We explore the signal variation when touching different positions on the tag,and build an RSSI-based MeTg Touch Ges model to verify the robustness of the RSSI deviation,which Track Rigid Touch Tag: Modon is mainly related to impedance change due to touch.Hence, Deteer Toueh based on the RSSI deviation,we can identify the absolute Gesture candidate touch positions.Note that,as the used linear Fig.1.An example application scenario of RF-Dial tag has the dipole antenna,the RSSI variation during the swipe across the tag forms a symmetric -wave pattern.To rotation of the tagged object for each snapshot based on the eliminate such ambiguity of touch positions,we use half of rigid transformation model.Note that,the phase contours the tag as buttons and the other half as the slider.Referring of RF-signals vary at different positions in the scanning to the RSSI variation,it is easy to determine which gesture area,the relationship between the tag movement and the is performed,thereby we can identify which part of the tag phase variation is different,regarding to which we split is touched and further derive the unique touch position.By the effective scanning area into linear region and non-linear tracking the consecutive touch positions of the swipe,we region.Meanwhile,based on RF-signals from the touch tag, are able to estimate the swipe direction and distance. we build an RSSI-based model to depict the relationship be- Overall,we make the following three main contribu- tween the touch position on the tag and the corresponding tions.First,we propose a novel interaction scheme via RFID RSSI variation of received RF-signals.According to the RSSI- technology,supporting the rigid motion tracking and the based model,the RSSI deviation is mainly related to the touch gesture detection.An ordinary object can be turned impedance change when touching the tag,so it is position- into an intelligent HCI device via attaching a tag array independent and orientation-insensitive.Consequently,we on the side face together with one linear tag on the top can rely on only one general RSSI deviation template to ac- face,denoted as movement tags and the touch tag,respec- curately and robustly determine the touch position,without tively.Second,we build a phase-based model to reflect the the known start touch position or the fixed tag deployment, relationship between the motion of tagged object and the among the whole monitoring area. corresponding phase variations of movement tags in the There are three key challenges to realize RF-Dial.1) array.We also build an RSSI-based model to depict the rela- How to estimate the rigid motion of the tagged object based tionship between the touch position and the corresponding on RE-signals of tags,including the translation and rotation RSSI deviation of the touch tag.Third,we implemented a simultaneously,is a key problem.To tackle this challenge,we prototype system of RF-Dial and evaluated its performance build a rigid transformation model to reflect the relationship in the real environment.Extensive experiments show that between the motion of the tagged object and the correspond- RF-Dial achieves an accurate rigid motion tracking,with a ing phase variations of each movement tag in the tag array. small error of 0.6cm for translation tracking,and a small As the topology of the movement tag array is fixed,we are error of 1.9 degrees for rotation estimation.Besides,RF-Dial able to decompose the rigid motion of the tagged object can also detect the touch gesture accurately,as the 90 percent referring to the phase variations of at least two movement of touch position errors are less than 2.09mm. tags,and then derive the translation and the rotation of the tagged object for each snapshot during the motion.2)How 2 RELATED WORK to address the variation of phase contours at different positions RFID-based Localization:A straightforward solution for in the effective scanning area is a key problem.Our empirical RFID-based human-computer interaction is to utilize RFID study shows that the phase contours are close to concentric localization schemes to directly locate tagged objects [2- circles with the antenna at the center.Hence,even for the 6,20-26].State-of-the-art systems mainly use phase values same rigid motion of the tagged object,the antenna could for the accurate localization [2-4,6,22,23].PinIt [2]uses collect different phase variations at different positions.To multi-path profiles of tags to accurately locate tags with the tackle this challenge,regarding to the relationship between synthetic aperture radar created via the antenna motion the tag movement and the phase variation,we split the Rather than the absolute localization,STPP [6]is the first whole scanning area into linear region and non-linear region. work to tackle 2D relative localization,which uses the Specifically,the tag movement in the linear region is linear spatial-temporal dynamics in the phase profiles to identify to the phase variation,thus we can extract the tag movement the relative positions of tags.More than only using the based on the phase variations detected from the two orthog- phase information,RFind [20]leverages the complete phys- onal antennas.While in the non-linear region,we locate the ical properties of RF-signals to realize the ultra-wideband tag first,then extract the tag movement based on the phase localization.RFind is capable of emulating over 220MHz of contours at the tag's position.3)How to obtain the absolute bandwidth without changing the tag and remains compliant touch position of the tag when the tag moves to any position with current regulations.However,most approaches figure with different orientations within the monitoring area is a key out the absolute positions of tags in a separate manner, problem.Existing work like [19]leverages the phase variation whereas RF-Dial aims to track the movement of the tag to detect the touch gesture,however,the phase is sensitive array in a comprehensive manner.By referring to the fixed to the position and orientation of the tag,so it can only topology of tag array,RF-Dial can accurately track the rigid track the touch position with the known start touch point transformation of tagged object,including the translation of a fixed tag.To tackle the challenge of determining the and rotation simultaneously

IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 2 RFID Antenna Pair Track Rigid Motion · Rotation · Translation Rigid Motion Touch Tag: Detect Touch Gesture Movement Tags: Tagged Daily Objects Touch Gesture · Click · Press & Hold · Swipe Fig. 1. An example application scenario of RF-Dial rotation of the tagged object for each snapshot based on the rigid transformation model. Note that, the phase contours of RF-signals vary at different positions in the scanning area, the relationship between the tag movement and the phase variation is different, regarding to which we split the effective scanning area into linear region and non-linear region. Meanwhile, based on RF-signals from the touch tag, we build an RSSI-based model to depict the relationship be￾tween the touch position on the tag and the corresponding RSSI variation of received RF-signals. According to the RSSI￾based model, the RSSI deviation is mainly related to the impedance change when touching the tag, so it is position￾independent and orientation-insensitive. Consequently, we can rely on only one general RSSI deviation template to ac￾curately and robustly determine the touch position, without the known start touch position or the fixed tag deployment, among the whole monitoring area. There are three key challenges to realize RF-Dial. 1) How to estimate the rigid motion of the tagged object based on RF-signals of tags, including the translation and rotation simultaneously, is a key problem. To tackle this challenge, we build a rigid transformation model to reflect the relationship between the motion of the tagged object and the correspond￾ing phase variations of each movement tag in the tag array. As the topology of the movement tag array is fixed, we are able to decompose the rigid motion of the tagged object referring to the phase variations of at least two movement tags, and then derive the translation and the rotation of the tagged object for each snapshot during the motion. 2) How to address the variation of phase contours at different positions in the effective scanning area is a key problem. Our empirical study shows that the phase contours are close to concentric circles with the antenna at the center. Hence, even for the same rigid motion of the tagged object, the antenna could collect different phase variations at different positions. To tackle this challenge, regarding to the relationship between the tag movement and the phase variation, we split the whole scanning area into linear region and non-linear region. Specifically, the tag movement in the linear region is linear to the phase variation, thus we can extract the tag movement based on the phase variations detected from the two orthog￾onal antennas. While in the non-linear region, we locate the tag first, then extract the tag movement based on the phase contours at the tag’s position. 3) How to obtain the absolute touch position of the tag when the tag moves to any position with different orientations within the monitoring area is a key problem. Existing work like [19] leverages the phase variation to detect the touch gesture, however, the phase is sensitive to the position and orientation of the tag, so it can only track the touch position with the known start touch point of a fixed tag. To tackle the challenge of determining the absolute touch position, we observe that the RSSI deviation during the swipe is position-independent and orientation￾insensitive. We explore the signal variation when touching different positions on the tag, and build an RSSI-based model to verify the robustness of the RSSI deviation, which is mainly related to impedance change due to touch. Hence, based on the RSSI deviation, we can identify the absolute candidate touch positions. Note that, as the used linear tag has the dipole antenna, the RSSI variation during the swipe across the tag forms a symmetric Ω-wave pattern. To eliminate such ambiguity of touch positions, we use half of the tag as buttons and the other half as the slider. Referring to the RSSI variation, it is easy to determine which gesture is performed, thereby we can identify which part of the tag is touched and further derive the unique touch position. By tracking the consecutive touch positions of the swipe, we are able to estimate the swipe direction and distance. Overall, we make the following three main contribu￾tions. First, we propose a novel interaction scheme via RFID technology, supporting the rigid motion tracking and the touch gesture detection. An ordinary object can be turned into an intelligent HCI device via attaching a tag array on the side face together with one linear tag on the top face, denoted as movement tags and the touch tag, respec￾tively. Second, we build a phase-based model to reflect the relationship between the motion of tagged object and the corresponding phase variations of movement tags in the array. We also build an RSSI-based model to depict the rela￾tionship between the touch position and the corresponding RSSI deviation of the touch tag. Third, we implemented a prototype system of RF-Dial and evaluated its performance in the real environment. Extensive experiments show that RF-Dial achieves an accurate rigid motion tracking, with a small error of 0.6cm for translation tracking, and a small error of 1.9 degrees for rotation estimation. Besides, RF-Dial can also detect the touch gesture accurately, as the 90 percent of touch position errors are less than 2.09mm. 2 RELATED WORK RFID-based Localization: A straightforward solution for RFID-based human-computer interaction is to utilize RFID localization schemes to directly locate tagged objects [2– 6, 20–26]. State-of-the-art systems mainly use phase values for the accurate localization [2–4, 6, 22, 23]. PinIt [2] uses multi-path profiles of tags to accurately locate tags with the synthetic aperture radar created via the antenna motion. Rather than the absolute localization, STPP [6] is the first work to tackle 2D relative localization, which uses the spatial-temporal dynamics in the phase profiles to identify the relative positions of tags. More than only using the phase information, RFind [20] leverages the complete phys￾ical properties of RF-signals to realize the ultra-wideband localization. RFind is capable of emulating over 220MHz of bandwidth without changing the tag and remains compliant with current regulations. However, most approaches figure out the absolute positions of tags in a separate manner, whereas RF-Dial aims to track the movement of the tag array in a comprehensive manner. By referring to the fixed topology of tag array, RF-Dial can accurately track the rigid transformation of tagged object, including the translation and rotation simultaneously

IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 RFID-based Motion Tracking:Prior RFID-based motion RF-signals Rigid Motion Tracking tracking systems propose various approaches for the trajec- tory tracking [7-12,27]and orientation tracking [13,14,28]. Data Preprocessing Motion or Motion Representative work like RF-IDraw [7]and PolarDraw [8] Data Segmentation touch? use a single tag to reconstruct the handwriting,regarding Template Generation Initial State Estimation the tag as a mass point for the motion tracking.Tag- Re-sampling Compass [14]uses a single tag to determine the orientation Feature Extraction Movement Tracking and position of the tagged object based on polarization Tag Movement Derivation properties of RF waves.Further,recent work uses the tag Fitting吗 Rigid Transformation array to track the trajectory or orientation of the mov- Touch Gesture Detection Tracking ing object.Specifically,Pantomime [9]enables the accurate Datu Preprocessing trajectory tracking of the tagged object with a tag array, Movement Calibration using a multiple tag single antenna system.Tagyro [13] Gesture Detection Outlier Detection realizes the 3D orientation tracking with an array of RFID tags,by converting the real-time phase offsets between tags Touch Estimation Outlier Elimination into the orientation angle.However,these approaches track either the trajectory or the orientation of the moving object, Interaction Commands without detecting the translation and rotation of the tagged Click Press and hold Swipe Translation S Rotation R object simultaneously.Tagball 16 is the closest work to RF-Dial,which studies the motion behavior,including the Fig.2.System framework translation and rotation,of a ball attached with a tag array. TABLE 1 However,Tagball solves the problem by the absolute local- Signal variation of different operations ization on multiple tags.Specifically,it first estimates the Operations Considerable Signal Variation absolute positions of multiple tags via the phase values,and Movement Tag Touch Tag None No No then figures out the translation and rotation of the tagged Movement Yes Yes object based on the estimated positions of tags.Hence,the Touch No Yes localization errors are further introduced to the estimation of the translation and rotation.Therefore,it requires plenty insensitive.With only one general RSSI deviation template, of tags,i.e.,12 tags in total,to provide enough data to the we can accomplish the light-weight and fine-grained touch Extended Kalman Filter-based tracking model to guarantee detection accurately among the whole operating area. the tracking accuracy.In comparison,RF-Dial tracks the translation and rotation of tagged objects simultaneously by 3 SYSTEM OVERVIEW directly referring to phase variations from at least two tags, thus it achieves more accuracy in the motion tracking. RF-Dial is designed to provide two functions:one is to track RFID-based Touch Sensing:Besides the traditional the rigid motion of the tagged object,including the transla- studies on localization or tracking,recently researchers also tion and the rotation,the other is to detect the touch gesture have studied the tag's physical change when the conductor on one linear tag,such as the click,the press and hold,and touches the tag,i.e.,the liquid and human beings,and tried the swipe.The basic idea is to use a tag array attached on to utilize such characteristics for applications like liquid the side face of an object to track its movement,and use detection [29]and touch interaction [19,30,31].PaperID [30] another single tag attached on the top face of the object to provides the capability to use COTS tags to sense the finger detect the touch gesture,as shown in Fig.1.Suppose the touch,swipe touch,and other gestures by the support tags in a array that track the motion are movement fags,the vector machine(SVM).But this sensing capability is very single tag that detects the touch gesture is the touch tag.To coarse,i.e.,it only can detect whether the touch happens reduce the mutual coupling between tags,we separate the but cannot determine where the exact touch position is on touch tag and the movement tags with enough distances, the tag.Meanwhile,the machine learning algorithm requires i.e.,more than 6cm of height difference.Fig.2 illustrates the high training overhead.RIO [19]observes that when a the system framework.After receiving the raw RF-signals human finger touches the tag,the tag's impedance changes, from tags,we first determine which kind of the operation, which causes the phase change among received RF-signals i.e.,touch or movement.The intuition is that for the touch correspondingly.Based on the phase variation template,it gesture,the signal of the touch tag changes significantly tracks the finger position during a swipe using the segmen- but the signals of movement tags keep steady,while for the tal dynamic time warping (SDTW)method.However,RIO motion,the signal of either the touch tag or the movement can only work with the known start finger position,other- tags changes along with the continuous motion,as shown in wise it cannot use the SDTW to track the finger position. TABLE 1.Therefore,by extracting the features of the signal Also,the phase is sensitive to the distance from the tag and variation of tags,such as the phase or RSSI difference in antenna,so the template is required to be updated when time intervals,we can determine which operation the user the tag is moved to different positions.Whereas,RF-Dial is performs using the random forest algorithm.Details are designed to track the absolute finger position without the shown in Section 7.5. known start position.Our solution is based on the RSSI de- As shown in Fig.2,there are two main function modules viation resulted from the impedance change when the touch to realize our goal of the rigid motion tracking and the touch happens,which is position-independent and orientation- gesture detection separately.In the following,we first build

IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 3 RFID-based Motion Tracking: Prior RFID-based motion tracking systems propose various approaches for the trajec￾tory tracking [7–12, 27] and orientation tracking [13, 14, 28]. Representative work like RF-IDraw [7] and PolarDraw [8] use a single tag to reconstruct the handwriting, regarding the tag as a mass point for the motion tracking. Tag￾Compass [14] uses a single tag to determine the orientation and position of the tagged object based on polarization properties of RF waves. Further, recent work uses the tag array to track the trajectory or orientation of the mov￾ing object. Specifically, Pantomime [9] enables the accurate trajectory tracking of the tagged object with a tag array, using a multiple tag single antenna system. Tagyro [13] realizes the 3D orientation tracking with an array of RFID tags, by converting the real-time phase offsets between tags into the orientation angle. However, these approaches track either the trajectory or the orientation of the moving object, without detecting the translation and rotation of the tagged object simultaneously. Tagball [16] is the closest work to RF-Dial, which studies the motion behavior, including the translation and rotation, of a ball attached with a tag array. However, Tagball solves the problem by the absolute local￾ization on multiple tags. Specifically, it first estimates the absolute positions of multiple tags via the phase values, and then figures out the translation and rotation of the tagged object based on the estimated positions of tags. Hence, the localization errors are further introduced to the estimation of the translation and rotation. Therefore, it requires plenty of tags, i.e., 12 tags in total, to provide enough data to the Extended Kalman Filter-based tracking model to guarantee the tracking accuracy. In comparison, RF-Dial tracks the translation and rotation of tagged objects simultaneously by directly referring to phase variations from at least two tags, thus it achieves more accuracy in the motion tracking. RFID-based Touch Sensing: Besides the traditional studies on localization or tracking, recently researchers also have studied the tag’s physical change when the conductor touches the tag, i.e., the liquid and human beings, and tried to utilize such characteristics for applications like liquid detection [29] and touch interaction [19, 30, 31]. PaperID [30] provides the capability to use COTS tags to sense the finger touch, swipe touch, and other gestures by the support vector machine (SVM). But this sensing capability is very coarse, i.e., it only can detect whether the touch happens but cannot determine where the exact touch position is on the tag. Meanwhile, the machine learning algorithm requires the high training overhead. RIO [19] observes that when a human finger touches the tag, the tag’s impedance changes, which causes the phase change among received RF-signals correspondingly. Based on the phase variation template, it tracks the finger position during a swipe using the segmen￾tal dynamic time warping (SDTW) method. However, RIO can only work with the known start finger position, other￾wise it cannot use the SDTW to track the finger position. Also, the phase is sensitive to the distance from the tag and antenna, so the template is required to be updated when the tag is moved to different positions. Whereas, RF-Dial is designed to track the absolute finger position without the known start position. Our solution is based on the RSSI de￾viation resulted from the impedance change when the touch happens, which is position-independent and orientation￾Data Preprocessing Movement Tracking Movement Calibration Outlier Detection Outlier Elimination Tag Movement Derivation Rigid Transformation Tracking Initial State Estimation Data Segmentation Rigid Motion Tracking Touch Gesture Detection Data Preprocessing Gesture Detection Touch Estimation Motion or touch? Motion Touch RF-signals Interaction Commands Click Press and hold Swipe Translation �� Rotation �� Feature Extraction Re-sampling Fitting Template Generation Fig. 2. System framework TABLE 1 Signal variation of different operations Operations Considerable Signal Variation Movement Tag Touch Tag None No No Movement Yes Yes Touch No Yes insensitive. With only one general RSSI deviation template, we can accomplish the light-weight and fine-grained touch detection accurately among the whole operating area. 3 SYSTEM OVERVIEW RF-Dial is designed to provide two functions: one is to track the rigid motion of the tagged object, including the transla￾tion and the rotation, the other is to detect the touch gesture on one linear tag, such as the click, the press and hold, and the swipe. The basic idea is to use a tag array attached on the side face of an object to track its movement, and use another single tag attached on the top face of the object to detect the touch gesture, as shown in Fig. 1. Suppose the tags in a array that track the motion are movement tags, the single tag that detects the touch gesture is the touch tag. To reduce the mutual coupling between tags, we separate the touch tag and the movement tags with enough distances, i.e., more than 6cm of height difference. Fig. 2 illustrates the system framework. After receiving the raw RF-signals from tags, we first determine which kind of the operation, i.e., touch or movement. The intuition is that for the touch gesture, the signal of the touch tag changes significantly but the signals of movement tags keep steady, while for the motion, the signal of either the touch tag or the movement tags changes along with the continuous motion, as shown in TABLE 1. Therefore, by extracting the features of the signal variation of tags, such as the phase or RSSI difference in time intervals, we can determine which operation the user performs using the random forest algorithm. Details are shown in Section 7.5. As shown in Fig. 2, there are two main function modules to realize our goal of the rigid motion tracking and the touch gesture detection separately. In the following, we first build

IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 a phase-based model for the rigid motion tracking in Sec- Central Beam Region tion 4,and propose a tracking solution based on the motion 180H model in Section 5.Then,we build an RSSI-based model 160 for the touch gesture detection in Section 6,and propose the corresponding solution to realize the absolute touch position detection robustly and accurately in Section 7. 120 4 RIGID MOTION TRACKING 1005 -100-80 -60 -40 -20 0 20 100 In this section,we propose the definition of linear region and Fig.3.Phase contours of RF-signals non-linear region based on phase contour variations.Then, we illustrate the relationship between the tag movement and phase variation in different regions.Further,we model the rigid motion to decompose the translation and rotation. Linear Region 0.6x0.6m2 4.1 Linear Region and Non-linear Region In RFID systems,the RF phase is a common attribute of the Non-linear Region wireless signal,ranging from 0 to 2m.It is very sensitive to the tag-antenna distance.Suppose the distance between the tag and the antenna is d,so the signal traverses a distance Fig.4.Effective scanning area:linear region vs non-linear region of 2d in the backscatter communication.Then,the phase provided by the antenna can be expressed as: of antenna Ay smaller than a certain value,we think this position belongs to the central beam of Ay.Also,as shown 0= ×2d+4 mod 2. (1) in Fig.3,the width of the central beam changes with the perpendicular distance.The larger the perpendicular where A is the wavelength,u represents the phase offset distance is,the wider the central beam is at that distance,the caused by the diversity of hardware characteristics.Accord- central beam region is trapezoidal.Therefore,taking Fig.4, ing to the phase expression in Eq.(1),besides the diversity if setting the small displacement as 5cm,the phase different term,the phase value mainly depends on the distance threshold as 0.4 radians,for the position with the distance between the tag and the antenna.Therefore,the phase of 1.2m from two antennas,the size of the linear region is contours should form concentric circles with the antenna about 0.6 x 0.6m2,centered at that position.Apart from the at the center in an ideal situation.We thus conduct an linear region,the phase variations in the other scanning area experiment to validate the above hypothesis.We build a 2D are not linear to the displacements along either the X-axis coordinate system according to the parallel direction (X- or Y-axis,they depend on the exact tag position instead. axis)and perpendicular direction (Y-axis)of the antenna, Hence,we denote the other area as the non-linear region. and set the origin (0,0)at the center of the antenna.Then, we collect phase values in a rectangle space in front of the 4.2 Rigid Transformation antenna,ranging from -100cm to 100cm along the X-axis During the continuous movement of an object,its position and from 100cm to 180cm along the Y-axis,the step is 5cm. and orientation are changing all the time.For a rigid body, The collected phase values are plotted in Fig.3.Based on such change of the position and orientation in the 2D the experiment results,we have the following observation: space can be defined by the rigid transformation R,S, Observation 1:The phase contours are very close to concen- where R is a 2 x 2 rotation matrix and S is a 2 x 1 tric circles with the antenna at the center.Besides,in the central translation matrix.Here,the rotation means a circular move- beam region marked with blue lines in Fig.3,the phase contours ment that the device rotates around a rotation center,and are almost parallel to each other and stretching along the X-axis. the translation means a linear movement that every point That is,in this region,the phase can be regarded as linearly related of the device moves with the same displacement.As the to the perpendicular distance from the tag to the antenna plane. continuous movement of an object consists of a series of As shown in Fig.4,assume that two antennas Ar and instant movements at different time,we denote the instant Ay are deployed in a mutually orthogonal manner and movement as the micro-movement,each micro-movement can separated with a fairly large distance.Then,according to be expressed with the rotation and translation.Thus,we can Observation 1,in the intersection area of the central beams use the rigid transformation to depict the micro-movement. of the two antennas,the displacement of a tag along the By attaching a tag array on an object,it is possible to X-axis and Y-axis should be linear to the phase variations track the rigid transformation based on the movement of received by the antenna Ar and Au,respectively.We thus each tag in the array.Note that,different from the rigid body, denote this intersection region as the linear region.The size i.e.,the tagged object,the tag attached on the object actually of the linear region depends on two factors:the central beam represents a single point of the object,so its movement can region of each antenna,and the perpendicular distance to be regarded as the particle movement,which only has the each antenna.Specifically,the central beam region relies on translation rather than the rotation.E.g.,assume an object the tolerance of the small displacement along the horizontal is attached with a tag array with the layout of rectangle, direction for the antenna.E.g.,if the small displacement as shown in Fig.5,the tags are denoted as solid points on along the X-axis at one position incurs the phase difference the rectangle.For any micro-movement in the continuous

IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 4 a phase-based model for the rigid motion tracking in Sec￾tion 4, and propose a tracking solution based on the motion model in Section 5. Then, we build an RSSI-based model for the touch gesture detection in Section 6, and propose the corresponding solution to realize the absolute touch position detection robustly and accurately in Section 7. 4 RIGID MOTION TRACKING In this section, we propose the definition of linear region and non-linear region based on phase contour variations. Then, we illustrate the relationship between the tag movement and phase variation in different regions. Further, we model the rigid motion to decompose the translation and rotation. 4.1 Linear Region and Non-linear Region In RFID systems, the RF phase is a common attribute of the wireless signal, ranging from 0 to 2π. It is very sensitive to the tag-antenna distance. Suppose the distance between the tag and the antenna is d, so the signal traverses a distance of 2d in the backscatter communication. Then, the phase provided by the antenna can be expressed as: θ =  2π λ × 2d + µ  mod 2π, (1) where λ is the wavelength, µ represents the phase offset caused by the diversity of hardware characteristics. Accord￾ing to the phase expression in Eq. (1), besides the diversity term, the phase value mainly depends on the distance between the tag and the antenna. Therefore, the phase contours should form concentric circles with the antenna at the center in an ideal situation. We thus conduct an experiment to validate the above hypothesis. We build a 2D coordinate system according to the parallel direction (X￾axis) and perpendicular direction (Y -axis) of the antenna, and set the origin (0, 0) at the center of the antenna. Then, we collect phase values in a rectangle space in front of the antenna, ranging from −100cm to 100cm along the X-axis and from 100cm to 180cm along the Y -axis, the step is 5cm. The collected phase values are plotted in Fig. 3. Based on the experiment results, we have the following observation: Observation 1: The phase contours are very close to concen￾tric circles with the antenna at the center. Besides, in the central beam region marked with blue lines in Fig. 3, the phase contours are almost parallel to each other and stretching along the X-axis. That is, in this region, the phase can be regarded as linearly related to the perpendicular distance from the tag to the antenna plane. As shown in Fig. 4, assume that two antennas Ax and Ay are deployed in a mutually orthogonal manner and separated with a fairly large distance. Then, according to Observation 1, in the intersection area of the central beams of the two antennas, the displacement of a tag along the X-axis and Y -axis should be linear to the phase variations received by the antenna Ax and Ay, respectively. We thus denote this intersection region as the linear region. The size of the linear region depends on two factors: the central beam region of each antenna, and the perpendicular distance to each antenna. Specifically, the central beam region relies on the tolerance of the small displacement along the horizontal direction for the antenna. E.g., if the small displacement along the X-axis at one position incurs the phase difference -100 -80 -60 -40 -20 0 20 40 60 80 100 100 120 140 160 180 1 2 3 4 5 6 Central Beam Region Fig. 3. Phase contours of RF-signals � � �! Linear Region 1.2� 1.2� 0.6×0.6�! Non-linear Region �" � Fig. 4. Effective scanning area: linear region vs non-linear region of antenna Ay smaller than a certain value, we think this position belongs to the central beam of Ay. Also, as shown in Fig. 3, the width of the central beam changes with the perpendicular distance. The larger the perpendicular distance is, the wider the central beam is at that distance, the central beam region is trapezoidal. Therefore, taking Fig. 4, if setting the small displacement as 5cm, the phase different threshold as 0.4 radians, for the position with the distance of 1.2m from two antennas, the size of the linear region is about 0.6 × 0.6m2 , centered at that position. Apart from the linear region, the phase variations in the other scanning area are not linear to the displacements along either the X-axis or Y -axis, they depend on the exact tag position instead. Hence, we denote the other area as the non-linear region. 4.2 Rigid Transformation During the continuous movement of an object, its position and orientation are changing all the time. For a rigid body, such change of the position and orientation in the 2D space can be defined by the rigid transformation R, S , where R is a 2 × 2 rotation matrix and S is a 2 × 1 translation matrix. Here, the rotation means a circular move￾ment that the device rotates around a rotation center, and the translation means a linear movement that every point of the device moves with the same displacement. As the continuous movement of an object consists of a series of instant movements at different time, we denote the instant movement as the micro-movement, each micro-movement can be expressed with the rotation and translation. Thus, we can use the rigid transformation to depict the micro-movement. By attaching a tag array on an object, it is possible to track the rigid transformation based on the movement of each tag in the array. Note that, different from the rigid body, i.e., the tagged object, the tag attached on the object actually represents a single point of the object, so its movement can be regarded as the particle movement, which only has the translation rather than the rotation. E.g., assume an object is attached with a tag array with the layout of rectangle, as shown in Fig. 5, the tags are denoted as solid points on the rectangle. For any micro-movement in the continuous

IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 5 4.3.2 Tag Movement in the Non-linear Region In the non-linear region,since the corresponding phase vari- ations are not linear to the tag movement,we need to figure out their relationship according to the geometric property. Given the phase variations Ar and Aby respectively col- Trajectory lected from the two orthogonally deployed antennas A.and 4 X Ay,according to Eq.(4),we have: Fig.5.Rigid transformation in con-Fig.6.Relationship between tag tinuous moving process movement and phase variation T1△x十△y= 入0r 4π (6 movement,it can be intuitively observed that,the rigid 入9y x,△x+,△y=4 transformation of the tagged object,including the translation and rotation,can be derived from the movement of different tags. where(zlz,.〉and〈rl,h,）denote the normalized vec- tor for the polar axis AP from the antenna A and Ay, 4.3 Model of Tag Movement and Phase Variation respectively.Therefore,as long as the starting position of According to Observation 1,the phase contours can be de- movement s,i.e,P,is known,the values of (,y)and picted as concentric circles with the antenna at the center. can be figured out.Then,by solving the linear Thus,we can build a polar coordinate system by setting equations in Eq.(6),we can directly compute [Ar,Ay]T. the center of the antenna as the origin.Then,given a tag movement s,we can further depict the relationship between 4.4 Model of Rigid Motion Decomposition the phase variation and the movement s in this polar coordi- As aforementioned,during the continuous moving process nate system.As shown in Fig.6,the antenna is deployed at of the rigid body,the micro-movement can be defined by position A,we use the vector s to denote the tag movement, the rigid transformation including the rotation and transla- the starting point of s is P.Besides,we use the vector I to tion.Meanwhile,the tag movement can be regarded as the denote the polar axis AP,and use y to denote the angle particle movement only with the translation.Therefore,we between s and 1.Thus,if we use Ad to denote the projection investigate the relationship between the tag movement and of s on the polar axis 1,then Ad=lls]cosy. the rigid transformation of the tagged object,i.e.,translation, Note that,for any tag movement in the micro-movement, rotation and translation with rotation,respectively. its moving distance should be smaller than half-wavelength, i.e.,lsll≤≥≈l6.4cm.According to Eq.(1),by offsetting 4.4.1 Translation the constant diversity term,the phase variation A caused The translation means a linear movement that every point of the by s is as follows: device moves with the same displacement.Suppose a rigid body 402 ×2Ad=25x×2 silcos7 is attached with a tag array T,when the center of the rigid (2) body translates from position Ps to position Pe,each tag Ti Meanwhile,as 1.s sl cosy,according to Eq.(2), in the tag array has the same translation (ss) 1 1m·sA8. (3)Let and T be the coordinates of tagT Note that,,而 when the rigid body is at position P,and Pe,respectively, is a normalized vector of 1,it depends on the then: position of P relative to A.Assume s=(Az,Ay), 工i,e Ii.s +S. (7) (i,y),then,according to Eq.(3), yi,e yi.s x△x+△y=△， Fig.7(a)shows an example of the translation when the rigid (4) body is attached with a rectangle tag array. x+7=1. 4.4.2 Rotation Then,to compute the tag movement s=(△x,△y）accord- The rotation means a circular movement that the device rotates ing to the phase variations,we investigate their relation- around a rotation center.Suppose a rigid body is attached ships in the linear region and non-linear region,respectively. with a tag array T,when the rigid body rotates around a 4.3.1 Tag Movement in the Linear Region rotation center Pa by the angle of a,all the tags should In the linear region,the phase variations detected from have the same rotation angle.Specifically,letTand the two orthogonally deployed antennas are linear to the [i.e,yi.e]be the coordinates of tag Ti when the rigid body tag's moving distances along the two orthogonal axes,re- starts rotation and ends rotation,respectively,let(a,y)be spectively.E.g,as shown in Fig.4,antenna A detects the coordinates of rotation center Pa,then: the phase variation of the tag movement along the X-axis, Tie-Ta -R Ti,s-Ta (8) whereas antenna Ay detects the phase variations of the tag Yi,e Ya yi,s-Ya movement along the Y-axis.Let A0r and A0u be the phase cosa -sina variations from antenna Az and Ay,respectively,so the tag where R is a rotation matrix sina cosa representing movement s is computed as follows: the counter-clockwise rotation of angle a.Fig.7(b)shows - an example of the rotation when the rigid body is attached (5) with a rectangle tag array

IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 5 ܺ ܻ Trajectory ܱ ܣ ܲ ܛ ܔ ߛ ݀߂ ܺ ܻ ܱ Fig. 5. Rigid transformation in con￾tinuous moving process ܺ ܻ ܣ ܲ ܛ ܔ ߛ ݀߂ ܱ ܣ ܲ ܛ ܔ ߛ ݀߂ ܺ ܻ ܱ Fig. 6. Relationship between tag movement and phase variation movement, it can be intuitively observed that, the rigid transformation of the tagged object, including the translation and rotation, can be derived from the movement of different tags. 4.3 Model of Tag Movement and Phase Variation According to Observation 1, the phase contours can be de￾picted as concentric circles with the antenna at the center. Thus, we can build a polar coordinate system by setting the center of the antenna as the origin. Then, given a tag movement s, we can further depict the relationship between the phase variation and the movement s in this polar coordi￾nate system. As shown in Fig. 6, the antenna is deployed at position A, we use the vector s to denote the tag movement, the starting point of s is P. Besides, we use the vector l to denote the polar axis AP, and use γ to denote the angle between s and l. Thus, if we use ∆d to denote the projection of s on the polar axis l, then ∆d = ksk cos γ. Note that, for any tag movement in the micro-movement, its moving distance should be smaller than half-wavelength, i.e., ksk ≤ λ 2 ≈ 16.4cm. According to Eq. (1), by offsetting the constant diversity term, the phase variation ∆θ caused by s is as follows: ∆θ = 2π λ × 2∆d = 2π λ × 2 ksk cos γ. (2) Meanwhile, as l · s = klk · ksk cos γ, according to Eq. (2), l klk · s = λ 4π ∆θ. (3) Note that, l klk is a normalized vector of l, it depends on the position of P relative to A. Assume s = h∆x, ∆yi, l klk = hxl , yli, then, according to Eq. (3),    xl∆x + yl∆y = λ 4π ∆θ, x 2 l + y 2 l = 1. (4) Then, to compute the tag movement s = h∆x, ∆yi accord￾ing to the phase variations, we investigate their relation￾ships in the linear region and non-linear region, respectively. 4.3.1 Tag Movement in the Linear Region In the linear region, the phase variations detected from the two orthogonally deployed antennas are linear to the tag’s moving distances along the two orthogonal axes, re￾spectively. E.g., as shown in Fig. 4, antenna Ax detects the phase variation of the tag movement along the X-axis, whereas antenna Ay detects the phase variations of the tag movement along the Y -axis. Let ∆θx and ∆θy be the phase variations from antenna Ax and Ay, respectively, so the tag movement s is computed as follows:  ∆x ∆y  =  λ 4π ∆θx λ 4π ∆θy  . (5) 4.3.2 Tag Movement in the Non-linear Region In the non-linear region, since the corresponding phase vari￾ations are not linear to the tag movement, we need to figure out their relationship according to the geometric property. Given the phase variations ∆θx and ∆θy respectively col￾lected from the two orthogonally deployed antennas Ax and Ay, according to Eq. (4), we have:    xlx∆x + ylx∆y = λ 4π ∆θx, xly∆x + yly∆y = λ 4π ∆θy, (6) where hxlx , ylx i and xly , yly  denote the normalized vec￾tor for the polar axis AP from the antenna Ax and Ay, respectively. Therefore, as long as the starting position of movement s , i.e, P, is known, the values of hxlx , ylx i and xly , yly  can be figured out. Then, by solving the linear equations in Eq. (6), we can directly compute [∆x, ∆y] T . 4.4 Model of Rigid Motion Decomposition As aforementioned, during the continuous moving process of the rigid body, the micro-movement can be defined by the rigid transformation including the rotation and transla￾tion. Meanwhile, the tag movement can be regarded as the particle movement only with the translation. Therefore, we investigate the relationship between the tag movement and the rigid transformation of the tagged object, i.e., translation, rotation and translation with rotation, respectively. 4.4.1 Translation The translation means a linear movement that every point of the device moves with the same displacement. Suppose a rigid body is attached with a tag array T, when the center of the rigid body translates from position Ps to position Pe, each tag Ti in the tag array has the same translation  S = sx, sy T  . Let [xi,s, yi,s] T and [xi,e, yi,e] T be the coordinates of tag Ti when the rigid body is at position Ps and Pe, respectively, then:  xi,e yi,e =  xi,s yi,s + S. (7) Fig. 7(a) shows an example of the translation when the rigid body is attached with a rectangle tag array. 4.4.2 Rotation The rotation means a circular movement that the device rotates around a rotation center. Suppose a rigid body is attached with a tag array T, when the rigid body rotates around a rotation center Pa by the angle of α, all the tags should have the same rotation angle. Specifically, let [xi,s, yi,s] T and [xi,e, yi,e] T be the coordinates of tag Ti when the rigid body starts rotation and ends rotation, respectively, let (xa, ya) be the coordinates of rotation center Pa, then:  xi,e − xa yi,e − ya  = R  xi,s − xa yi,s − ya  , (8) where R is a rotation matrix  cos α − sin α sin α cos α  , representing the counter-clockwise rotation of angle α. Fig. 7(b) shows an example of the rotation when the rigid body is attached with a rectangle tag array.