RF-Brush:3D Human-Computer Interaction via Linear Tag Array Yinyin Gong,Lei Xie,Chuyu Wang,Yanling Bu,Sanglu Lu State Key Laboratory for Novel Software Technology,Nanjing University Email:[yygong,wangcyu217)@dislab.nju.edu.cn,[Ixie,sanglu)@nju.edu.cn,yanling@smail.nju.edu.cn Abstract-Nowadays,novel approaches of 3D human-computer interaction have enabled the capability of manipulating in the 3D Tag 3D Orientation Array space rather than 2D space.For example,Microsoft Surface Pen leverages the embedded sensors to sense the 3D manipulations such as inclining the pen to get bolder handwriting.In this paper,we propose RF-Brush,a battery-free and light-weight solution for 3D human-computer interaction based on RFID,by simply attaching a linear RFID tag array onto the linear shaped object like a brush.RF-Brush senses the 3D orientation and 2D movement of the linear shaped object,when the human subject is drawing with this object in the 3D space.Here,the 3D orientation 21 Operating Plane refers to the relative orientation of the linear shaped object to the operating plane,whereas the 2D movement refers to the moving trace in the 2D operating plane.In this way,we are able to Fig.1:Illustration of applications scenario of RF-Brush. transform an ordinary linear shaped object like a brush or pen need extensive training process.They either are sensitive to to an intelligent HCI device.Particularly,we build two geometric the change of background and light condition,or only pro- models to depict the relationship between the RF-signal and the 3D orientation as well as 2D movement,respectively.Based vide coarse-grained classification manipulation.Sensor-based on the geometric model,we propose the linear tag array-based approaches [6]usually can accurately track the motion of the HCI solution,implemented a prototype system,and evaluated device in the 3D space based on the embedded IMU.But they the performance in real environment.The experiments show usually suffer from the limited battery life.As for RFID-based that RF-Brush achieves an average error of 5.7 and 8.6 of elevation and azimuthal angle,respectively,and an average error approaches [7].[8].most of them are focusing on interacting of 3.8cm and 4.2cm in movement tracking along X-axis and Y. with computer in 2D space.Therefore,according to the above axis,respectively.Moreover,RF-Brush achieves 89%in letter considerations,in this paper,we aim to provide a battery-free recognition accuracy. and more natural 3D human-computer interaction by tracking I.INTRODUCTION the fine-grained 3D motion based on cheap ubiquitous objects. With the ubiquity of sensing techniques,human-computer As Fig.I shows,we interact with computer not only by the interaction (HCl)has drawn more and more attention recently, moving trace in 2D space,but also the orientation.Here,the and novel approaches are required to provide a more natural orientation is able to reflect the thickness of lines,and the 2D interaction.Currently,due to the wide utilization of the smart moving trace reflects what we write finally devices (e.g.,smart phone),the touch screen has become the With the development of RFID technique,RFID tag has most popular HCI interaction approach.However,by using the been widely regarded as a light-weight and battery-less sensor finger or point-touch pen for manipulation,the touch screen for locating and sensing in recent years.In this paper,we only provides a 2D interaction with the devices,which may propose RF-Brush,a novel system based on linear tag array prevent the user from interacting with the computer via a attached onto linear shaped object,which can provide both the 3D manner.For example,people may intend to incline the 2D movement and the 3D orientation information of tagged point-touch pen to draw a bolder line,which is a common object.Here,the 2D movement refers to the moving trace manipulation for a real pen.But based on a traditional point-in 2D operating plane,whereas the 3D orientation refers to touch pen,people need to separately configure for the thick- the relative orientation of the object to the operating plane. ness of lines,which makes the manipulation tedious.Some The basic idea is to leverage the phase change of the linear dedicated devices,e.g.,Microsoft Surface Pen,can leverage tag array during the process of the 3D motion to recover the the embedded sensors to sense the 3D manipulations,such 2D movement in 2D operating plane and the 3D orientation as inclining the pen to get bolder handwriting,but they are in 3D environment.Particularly,we first extract the phase usually expensive and only work on specific smart devices.difference between different tags at the same time point,which Existing HCI approaches can be divided into four main can be used for the 3D orientation estimation via our proposed groups:computer vision-based,sensor-based,wifi-based and geometrical model.Moreover,we build another geometrical RFID-based.Computer vision-based approaches [1]-[3]and model,which can accurately track the 2D movement of the wireless techniques [4].[5],e.g.,WiFi or FMCW usually linear shaped object's tip based on the phase variation between
RF-Brush: 3D Human-Computer Interaction via Linear Tag Array Yinyin Gong, Lei Xie, Chuyu Wang, Yanling Bu, Sanglu Lu State Key Laboratory for Novel Software Technology, Nanjing University Email: {yygong,wangcyu217}@dislab.nju.edu.cn, {lxie,sanglu}@nju.edu.cn, yanling@smail.nju.edu.cn Abstract—Nowadays, novel approaches of 3D human-computer interaction have enabled the capability of manipulating in the 3D space rather than 2D space. For example, Microsoft Surface Pen leverages the embedded sensors to sense the 3D manipulations, such as inclining the pen to get bolder handwriting. In this paper, we propose RF-Brush, a battery-free and light-weight solution for 3D human-computer interaction based on RFID, by simply attaching a linear RFID tag array onto the linear shaped object like a brush. RF-Brush senses the 3D orientation and 2D movement of the linear shaped object, when the human subject is drawing with this object in the 3D space. Here, the 3D orientation refers to the relative orientation of the linear shaped object to the operating plane, whereas the 2D movement refers to the moving trace in the 2D operating plane. In this way, we are able to transform an ordinary linear shaped object like a brush or pen to an intelligent HCI device. Particularly, we build two geometric models to depict the relationship between the RF-signal and the 3D orientation as well as 2D movement, respectively. Based on the geometric model, we propose the linear tag array-based HCI solution, implemented a prototype system, and evaluated the performance in real environment. The experiments show that RF-Brush achieves an average error of 5.7 ◦ and 8.6 ◦ of elevation and azimuthal angle, respectively, and an average error of 3.8cm and 4.2cm in movement tracking along X-axis and Yaxis, respectively. Moreover, RF-Brush achieves 89% in letter recognition accuracy. I. INTRODUCTION With the ubiquity of sensing techniques, human-computer interaction (HCI) has drawn more and more attention recently, and novel approaches are required to provide a more natural interaction. Currently, due to the wide utilization of the smart devices (e.g., smart phone), the touch screen has become the most popular HCI interaction approach. However, by using the finger or point-touch pen for manipulation, the touch screen only provides a 2D interaction with the devices, which may prevent the user from interacting with the computer via a 3D manner. For example, people may intend to incline the point-touch pen to draw a bolder line, which is a common manipulation for a real pen. But based on a traditional pointtouch pen, people need to separately configure for the thickness of lines, which makes the manipulation tedious. Some dedicated devices, e.g., Microsoft Surface Pen, can leverage the embedded sensors to sense the 3D manipulations, such as inclining the pen to get bolder handwriting, but they are usually expensive and only work on specific smart devices. Existing HCI approaches can be divided into four main groups:computer vision-based, sensor-based, wifi-based and RFID-based. Computer vision-based approaches [1]–[3] and wireless techniques [4], [5], e.g., WiFi or FMCW usually Z X 𝜷𝒂 Tag Array Y Operating Plane 𝜷𝒂 2D Movement 3D Orientation Fig. 1: Illustration of applications scenario of RF-Brush. need extensive training process. They either are sensitive to the change of background and light condition, or only provide coarse-grained classification manipulation. Sensor-based approaches [6] usually can accurately track the motion of the device in the 3D space based on the embedded IMU. But they usually suffer from the limited battery life. As for RFID-based approaches [7], [8], most of them are focusing on interacting with computer in 2D space. Therefore, according to the above considerations, in this paper, we aim to provide a battery-free and more natural 3D human-computer interaction by tracking the fine-grained 3D motion based on cheap ubiquitous objects. As Fig.1 shows, we interact with computer not only by the moving trace in 2D space, but also the orientation. Here, the orientation is able to reflect the thickness of lines, and the 2D moving trace reflects what we write finally. With the development of RFID technique, RFID tag has been widely regarded as a light-weight and battery-less sensor for locating and sensing in recent years. In this paper, we propose RF-Brush, a novel system based on linear tag array attached onto linear shaped object, which can provide both the 2D movement and the 3D orientation information of tagged object. Here, the 2D movement refers to the moving trace in 2D operating plane, whereas the 3D orientation refers to the relative orientation of the object to the operating plane. The basic idea is to leverage the phase change of the linear tag array during the process of the 3D motion to recover the 2D movement in 2D operating plane and the 3D orientation in 3D environment. Particularly, we first extract the phase difference between different tags at the same time point, which can be used for the 3D orientation estimation via our proposed geometrical model. Moreover, we build another geometrical model, which can accurately track the 2D movement of the linear shaped object’s tip based on the phase variation between
consecutive time points of the same tag.Finally,combining and Y-axis respectively in the 2D movement evaluation. both the 3D orientation and 2D movement,we can recover II.RELATED WORK the 3D motion of the linear shaped object,which can further facilitate the interaction with the computers. RFID-based Trajectory Tracking:State-of-the-art systems Realizing such RFID-based 3D human-computer interaction usually take advantage of the phase variation from RFID via linear tag array entails three key challenges.1)How to tags of consecutive time points to accurately track the 2D track the 2D movement and 3D orientation simultaneously? movement [7].[9]-121.Representative work such as Tago- Since the 3D motion of the linear shaped object can lead ram [13]realizes the real-time 2D movement tracking of the to the 2D movement and 3D rotation simultaneously,and if tagged object by estimating the absolute location of the tag we are tracking some part of the object,the rotation of the at every time point based on multiple fixed RFID antennas. object may also affect the moving trajectory.For example, Pantomime [8]leverages the tag array to track the trajectory the rotation of the object can lead to the movement of tip of the moving object based on a single antenna,which can in the operating plane.The RF phase is changing with both reduce the cost of the multiple fixed RFID antennas.However. of the two kinds of motions at the same time.Therefore,it is all of these existing works regard the object as a point, challenging to decompose the two kinds of phase change from which indicates the displacement of object is equal to the one tag for 3D motion tracking.To handle the problem,we displacement of tag.Besides,they only focus on the 2D utilize a linear tag array for RF phase measurement,and build human-computer interaction,which cannot satisfy the user's two models to depict the relationship between the two kinds of demand in 3D human-computer interaction.Different from motions and RF phase from both the time and space domain. these work,we need to track the 3D motion of the linear 2)How to deal with the diversity caused by tag's orientation? shaped objects,which contains both the 3D orientation and Since the 3D motion of linear shaped object can unavoidably the 2D movement. change the orientation of the tag array.the change of tag RFID-based Orientation Tracking:Recently,several s- orientation will also lead to the phase offset,which is later tudies are proposed to estimate the orientation of the objects studied in section III.To address this challenge,we deploy the based on the RFID technique [14]-[16].Tagyro [16]tracks tag array linearly,so when we calculate the phase difference the 3D orientation of the object by attaching the tag array and between tags,the diversity caused by the tag's orientation is models the relationship between the orientation and the phase thus canceled.due to each tag has the same orientation.3)How offset.Different from Tagyro,RF-Brush need to track both the to improve the robustness of our system to real environment? 3D orientation and 2D trajectory simultaneously by attaching a Since the RF signal is sensitive to the influence such as multi- linear tag array on the linear shaped object.Therefore we also path effect and mutual interference,it is challenging to be need to study the relationship between phase and trajectory robust in the real environments.To address this challenge,on based on the orientation information.Other work such as one hand,we make use of redundant tags attached on the linear PolarDraw [17]reconstructs the handwriting by estimating the shaped object in space domain,and estimate the 3D motion azimuthal angle,so that to track the 2D movement with a of the linear shaped object based on the RF signal of all the single tag,which leverages the RSS and phase trend based tags,which can calibrate the signal noise from part of the tags. on two linearly polarized antennas.In comparison,RF-Brush On the other hand,we leverage the motion continuity,and utilizes the phase received by circularly polarized antenna and set constraints for each motion estimation based on previous provides both the features of 3D orientation and 2D movement estimated result in time domain. which can used to interact with computer in 3D level. We make three key contributions in this paper.First,we propose a novel system for 3D human-computer interaction, III.PRELIMINARIES which not only provides the 2D movement,but also the 3D The RF phase is widely used for 2D localization and orientation of object.By attaching a linear tag array onto the tracking in previous works [13],[18].and has been validat- surface of linear shaped object,we are able to convert the ed to be an efficient wireless attribute for mobile sensing. object to an intelligent HCI device.Second,we build two Theoretically,it represents the degree that the received signal geometric models to depict the relationship between RF-signal offsets from the sender,ranging from 0 to 2,and thus and 3D orientation as well as 2D movement,which study the is determined by the transmitting distance.In a real RFID fundamental features in RFID sensing system and facilitate system,the manufacturing technique of both the reader and great quantity applications of 3D human-computer interaction. tag also introduce some static phase offset,which is usually Based on the proposed geometric models,we are able to regarded as the device diversity of phase. estimate the 3D orientation and 2D movement accurately and In addition to the static device diversity,the RF phase simultaneously.Third,we implemented a prototype system of is also affected by the dynamic status of each tag.Since RF-Brush with COTS RFID and examined its performance the 3D motion of tagged object can unavoidably lead to the in the real environments.The experiments show that the our change of tag orientation in the 3D space,we thus conduct system achieved an average error of 5.7 and 8.6 of elevation experiments to study the influence of tag orientation on the angle and azimuthal angle in 3D orientation evaluation respec- phase as illustrated in Fig.2.Particularly,we rotate one RFID tively,and an average error of 3.8cm and 4.2cm along X-axis tag 180 at a fixed point along three different axes in front
consecutive time points of the same tag. Finally, combining both the 3D orientation and 2D movement, we can recover the 3D motion of the linear shaped object, which can further facilitate the interaction with the computers. Realizing such RFID-based 3D human-computer interaction via linear tag array entails three key challenges. 1) How to track the 2D movement and 3D orientation simultaneously? Since the 3D motion of the linear shaped object can lead to the 2D movement and 3D rotation simultaneously, and if we are tracking some part of the object, the rotation of the object may also affect the moving trajectory. For example, the rotation of the object can lead to the movement of tip in the operating plane. The RF phase is changing with both of the two kinds of motions at the same time. Therefore, it is challenging to decompose the two kinds of phase change from one tag for 3D motion tracking. To handle the problem, we utilize a linear tag array for RF phase measurement, and build two models to depict the relationship between the two kinds of motions and RF phase from both the time and space domain. 2) How to deal with the diversity caused by tag’s orientation? Since the 3D motion of linear shaped object can unavoidably change the orientation of the tag array, the change of tag orientation will also lead to the phase offset, which is later studied in section III. To address this challenge, we deploy the tag array linearly, so when we calculate the phase difference between tags, the diversity caused by the tag’s orientation is thus canceled, due to each tag has the same orientation. 3) How to improve the robustness of our system to real environment? Since the RF signal is sensitive to the influence such as multipath effect and mutual interference, it is challenging to be robust in the real environments. To address this challenge, on one hand, we make use of redundant tags attached on the linear shaped object in space domain, and estimate the 3D motion of the linear shaped object based on the RF signal of all the tags, which can calibrate the signal noise from part of the tags. On the other hand, we leverage the motion continuity, and set constraints for each motion estimation based on previous estimated result in time domain. We make three key contributions in this paper. First, we propose a novel system for 3D human-computer interaction, which not only provides the 2D movement, but also the 3D orientation of object. By attaching a linear tag array onto the surface of linear shaped object, we are able to convert the object to an intelligent HCI device. Second, we build two geometric models to depict the relationship between RF-signal and 3D orientation as well as 2D movement, which study the fundamental features in RFID sensing system and facilitate great quantity applications of 3D human-computer interaction. Based on the proposed geometric models, we are able to estimate the 3D orientation and 2D movement accurately and simultaneously. Third, we implemented a prototype system of RF-Brush with COTS RFID and examined its performance in the real environments. The experiments show that the our system achieved an average error of 5.7 ◦ and 8.6 ◦ of elevation angle and azimuthal angle in 3D orientation evaluation respectively, and an average error of 3.8cm and 4.2cm along X-axis and Y-axis respectively in the 2D movement evaluation. II. RELATED WORK RFID-based Trajectory Tracking: State-of-the-art systems usually take advantage of the phase variation from RFID tags of consecutive time points to accurately track the 2D movement [7], [9]–[12]. Representative work such as Tagoram [13] realizes the real-time 2D movement tracking of the tagged object by estimating the absolute location of the tag at every time point based on multiple fixed RFID antennas. Pantomime [8] leverages the tag array to track the trajectory of the moving object based on a single antenna, which can reduce the cost of the multiple fixed RFID antennas. However, all of these existing works regard the object as a point, which indicates the displacement of object is equal to the displacement of tag. Besides, they only focus on the 2D human-computer interaction, which cannot satisfy the user’s demand in 3D human-computer interaction. Different from these work, we need to track the 3D motion of the linear shaped objects, which contains both the 3D orientation and the 2D movement. RFID-based Orientation Tracking: Recently, several studies are proposed to estimate the orientation of the objects based on the RFID technique [14]–[16]. Tagyro [16] tracks the 3D orientation of the object by attaching the tag array and models the relationship between the orientation and the phase offset. Different from Tagyro, RF-Brush need to track both the 3D orientation and 2D trajectory simultaneously by attaching a linear tag array on the linear shaped object. Therefore we also need to study the relationship between phase and trajectory based on the orientation information. Other work such as PolarDraw [17] reconstructs the handwriting by estimating the azimuthal angle, so that to track the 2D movement with a single tag, which leverages the RSS and phase trend based on two linearly polarized antennas. In comparison, RF-Brush utilizes the phase received by circularly polarized antenna and provides both the features of 3D orientation and 2D movement which can used to interact with computer in 3D level. III. PRELIMINARIES The RF phase is widely used for 2D localization and tracking in previous works [13], [18], and has been validated to be an efficient wireless attribute for mobile sensing. Theoretically, it represents the degree that the received signal offsets from the sender, ranging from 0 to 2π, and thus is determined by the transmitting distance. In a real RFID system, the manufacturing technique of both the reader and tag also introduce some static phase offset, which is usually regarded as the device diversity of phase. In addition to the static device diversity, the RF phase is also affected by the dynamic status of each tag. Since the 3D motion of tagged object can unavoidably lead to the change of tag orientation in the 3D space, we thus conduct experiments to study the influence of tag orientation on the phase as illustrated in Fig.2. Particularly, we rotate one RFID tag 180◦ at a fixed point along three different axes in front
can be regarded as the interaction with computers.Therefore, we focus on tracking such typical 3D motion,where the brush is moving and rotating simultaneously by keeping the tip on the operating plane.Fig.4 presents a simple case when the user writes on the X-Y plane.Here,the whole motion of the brush is composed of a series of instantaneous postures at different Fig.2:Experiment deployment of tag rotation. time points.Each instantaneous posture can be expressed with the absolute position of its tip on the operating plane and the 3D orientation of the brush in 3D environment.Therefore the whole motion of the brush is decomposed as follows: moving the absolute position of the brush tip and rotating the orientation of the brush to the operating plane.For the first part,we only need to extract the relative moving trajectory .3 of the tip,which is the relative displacement of the absolute 020406080100120140160180 0 20406080100120140160180 Rotation(deg) Rotationdeg) position between consecutive time points,which called 2D (a)Phase change along with tag ori-(b)The measured phase when rotat- movement in next Section.For the second part,we need to entation ing along Y-axis extract the 3D orientation of the brush at each time point. Fig.3:Empirical study when rotating the tag. which will affect the relative trajectory estimation.Here,the 3D orientation refers to the relative orientation of the brush of an RFID reader,and present the phase change in Fig.3(a). to the operating plane. According to the results.we find that the rotation along Y-axis Linear shaped object will introduce a linear decrement on the phase measurement, while the rotation along X-axis or Z-axis almost does not Operating plane affect the phase measurements.It means the tag orientation along Y-axis will also introduce some offset to the actual phase measurement.which is defined as the orientation diversity of 2D Movement phase.Therefore,supposing the transmitting distance is dr, then the measured phase value a can be calculated as: X 0=(22m+9am+8,)mod2x, (1) Fig.4:Decomposing 3D motion into 3D rotation and 2D 入 movement. where A is the wavelength of RF signal,Odev denotes the Based on the understanding,we can express the instanta- device diversity,and 0 represents the orientation diversity. neous posture of the brush in the Cartesian Coordinate System Here,the device diversity Odee contains the phase diversity caused by both the reader and the tag.In RFID system,since as shown in Fig.4.X-Y plane represents the operating plane of the brush,and Z-axis is the perpendicular to the X-Y plane, the reader communicates with the tag via backscattering,the representing the height away from the X-Y plane.Therefore, signal traverses 2dr distance in total,which thus leads to 2d2m phase change. we can use(B,yB,0)to represent the absolute position of the brush tip.As for the brush orientation,we can use elevation Furthermore,we investigate the consistency of orientation angle and azimuthal angle to uniquely define the orientation diversity across different tags.Since the orientation diversity is Particularly,the elevation angle,denoted as B.,measures the only related to the rotation along Y-axis,we next rotate four angle between the brush and X-Y plane,which indicates slope different tags along Y-axis in turn at the same fixed point. Fig.3(b)presents the phase change along with the rotation of the brush with respect to the X-Y plane.The azimuthal along Y-axis.It is clear that all the four tags have the same angle,denoted as Ba,measures the angle between the brush's projection on the X-Y plane and the X-axis,which indicates phase trend along with the rotation angle.Therefore,we azimuth shift of the brush,rotating in anti-clockwise from the conclude that all the tag have the same orientation diversity X-axis.Therefore,when the brush is moving on the X-Y plane, of 0 if the rotation angle along Y-axis is the same. Be is ranging from 0 to /2,and Ba is ranging from 0 to IV.MODELING 3D MOTION VIA A LINEAR TAG ARRAY 2.Then,combining the position of the brush tip (B,yB,0) A.Decomposing 3D Motion into 3D Orientation and 2D and the brush orientation (Be,Ba),we can accurately describe the instantaneous posture of the brush.Thus,our goal is to Movement estimate the both brush orientation and the brush position to Before deeply investigating the relationship between the t track the 3D motion of the brush,which is solved via two 3D motion and RF signal of linear shaped object,we first models in the next two subsections. demonstrate how to decompose the 3D motion.Without loss of generality,we use the brush to replace the linear shaped object B.Modeling 3D Orientation via Linear Tag Array in this paper to demonstrate our model.Typically,the brush is We next demonstrate how to estimate the brush orientation, used to write or draw pictures on the operating plane,which i.e.,(Be,Ba),by leveraging the linear tag array.Fig.5 illustrates
Y Z X Fig. 2: Experiment deployment of tag rotation. 0 20 40 60 80 100 120 140 160 180 Rotation(deg.) -4 -3 -2 -1 0 1 2 3 4 Phase(rad.) X Y Z (a) Phase change along with tag orientation 0 20 40 60 80 100 120 140 160 180 Rotation(deg.) 0 1 2 3 4 5 6 Phase(rad.) Tag1 Tag2 Tag3 Tag4 (b) The measured phase when rotating along Y-axis Fig. 3: Empirical study when rotating the tag. of an RFID reader, and present the phase change in Fig.3(a). According to the results, we find that the rotation along Y-axis will introduce a linear decrement on the phase measurement, while the rotation along X-axis or Z-axis almost does not affect the phase measurements. It means the tag orientation along Y-axis will also introduce some offset to the actual phase measurement, which is defined as the orientation diversity of phase. Therefore, supposing the transmitting distance is dT , then the measured phase value θ can be calculated as: θ = (2dT λ 2π + θdev + θγ) mod 2π, (1) where λ is the wavelength of RF signal, θdev denotes the device diversity, and θγ represents the orientation diversity. Here, the device diversity θdev contains the phase diversity caused by both the reader and the tag. In RFID system, since the reader communicates with the tag via backscattering, the signal traverses 2dT distance in total, which thus leads to 2dT λ 2π phase change. Furthermore, we investigate the consistency of orientation diversity across different tags. Since the orientation diversity is only related to the rotation along Y-axis, we next rotate four different tags along Y-axis in turn at the same fixed point. Fig.3(b) presents the phase change along with the rotation along Y-axis. It is clear that all the four tags have the same phase trend along with the rotation angle. Therefore, we conclude that all the tag have the same orientation diversity of θγ, if the rotation angle along Y-axis is the same. IV. MODELING 3D MOTION VIA A LINEAR TAG ARRAY A. Decomposing 3D Motion into 3D Orientation and 2D Movement Before deeply investigating the relationship between the 3D motion and RF signal of linear shaped object, we first demonstrate how to decompose the 3D motion. Without loss of generality, we use the brush to replace the linear shaped object in this paper to demonstrate our model. Typically, the brush is used to write or draw pictures on the operating plane, which can be regarded as the interaction with computers. Therefore, we focus on tracking such typical 3D motion, where the brush is moving and rotating simultaneously by keeping the tip on the operating plane. Fig.4 presents a simple case when the user writes on the X-Y plane. Here, the whole motion of the brush is composed of a series of instantaneous postures at different time points. Each instantaneous posture can be expressed with the absolute position of its tip on the operating plane and the 3D orientation of the brush in 3D environment. Therefore, the whole motion of the brush is decomposed as follows: moving the absolute position of the brush tip and rotating the orientation of the brush to the operating plane. For the first part, we only need to extract the relative moving trajectory of the tip, which is the relative displacement of the absolute position between consecutive time points, which called 2D movement in next Section. For the second part, we need to extract the 3D orientation of the brush at each time point, which will affect the relative trajectory estimation. Here, the 3D orientation refers to the relative orientation of the brush to the operating plane. Z X 2D Movement Y 𝜷𝒂 Linear shaped object Operating plane Fig. 4: Decomposing 3D motion into 3D rotation and 2D movement. Based on the understanding, we can express the instantaneous posture of the brush in the Cartesian Coordinate System as shown in Fig.4. X-Y plane represents the operating plane of the brush, and Z-axis is the perpendicular to the X-Y plane, representing the height away from the X-Y plane. Therefore, we can use (xB, yB, 0) to represent the absolute position of the brush tip. As for the brush orientation, we can use elevation angle and azimuthal angle to uniquely define the orientation. Particularly, the elevation angle, denoted as βe, measures the angle between the brush and X-Y plane, which indicates slope of the brush with respect to the X-Y plane. The azimuthal angle, denoted as βa, measures the angle between the brush’s projection on the X-Y plane and the X-axis, which indicates azimuth shift of the brush, rotating in anti-clockwise from the X-axis. Therefore, when the brush is moving on the X-Y plane, βe is ranging from 0 to π/2, and βa is ranging from 0 to 2π. Then, combining the position of the brush tip (xB, yB, 0) and the brush orientation hβe, βai, we can accurately describe the instantaneous posture of the brush. Thus, our goal is to estimate the both brush orientation and the brush position to track the 3D motion of the brush, which is solved via two models in the next two subsections. B. Modeling 3D Orientation via Linear Tag Array We next demonstrate how to estimate the brush orientation, i.e., hβe, βai, by leveraging the linear tag array. Fig.5 illustrates
X Fig.5:Deployment of RF-Brush. the basic deployment of RF-brush.Particularly,we attach a Fig.6:Relationship between phase difference and brush linear tag array on the surface of the brush for passive sensing. projection. whose deployment is known in advance.For simplicity,we demonstrate our model with only two tags,i.e.,T and T2, △9,12= 201x-2.2元 (4) 入 and the reader can easily extend the model to the array with multiple tags.We use(1,y1,21)and (x2,y2,22)to represent where lir represents the projected distance on the X-axis of the coordinates of the two tags in our model.Here,the two L;as shown in Fig.6.In essence,the phase difference A0.1.2 here measures the distance of brush projection between T and tags Ti and T2 are l and l2 distance away from the brush tip, respectively.Besides,the antenna A and Ay are deployed T2 along X-axis. on the X-Y plane to measure the RF phase of tags from two 2)Estimating Brush Orientation from Brush Projection: perpendicular dimensions.Antenna Ar is on the Y-axis,which Next,we demonstrate how to estimate the brush orientation mainly measures phase change caused by tag displacement based on l1-2..In Fig.6,we can calculate the projection of along X-axis.And antenna A,is on the X-axis,and measures li from the brush orientation (Be,Ba)based on the geometric the phase change due to the displacement along Y-axis.Based position as: li,x=l3×cos Be×cos6a (5) on the deployment,our basic idea is to first calculate the phase difference between two tags at the same time point, Therefore,we further rewrite Eq.(4)with the brush orientation then estimate the brush projection on the X-Y plane from the (Be,Ba）as phase difference of the two tags,and finally leverage the brush (6) projection to calculate the brush orientation. △812=20-12)co×cos2m 1)Calculating Brush Projection from Phase Difference: Since l and l2 have been measured in advance,the unknown Firstly,we demonstrate how to refine the phase difference parameters are only the brush orientation (Be,Ba). between two tags,which can be further used to estimate the So far.we have modeled the brush orientation based on the brush projection.Consider Fig.6 where we use antenna Ar to phase difference received from the antenna A.However,Eg. measure the RF phase,and present the top view of Fig.5.Here, (6)still cannot uniquely express the brush orientation.Since Ba the RF signals from Az first reach T before T2.According ranges from 0 to 2m,cos Ba and cos(2-Ba)have the same to Eq.(1),the phase difference A0r.1.2 between Ti and T2 value.As a result,there is always an ambiguity orientation received by antenna A can be calculated as: with the same phase difference.To solve the ambiguity and △012=(21-22r+△0ae12+△0,12）m0d2x calculate the orientation,we deploy anther antenna A as 入 shown in Fig.6,which is mutually orthogonal manner with the (2) Here,Adev.1.2 calculates the phase difference caused by the antenna A.So the operating plane is the central area where device diversity.Since the device diversity is constant for all the beams of two antennas meets,and the antenna Ay measure the tags,we can remove them in advance.A0.1.2 represents the phase difference from another direction to compensate the the phase difference caused by the orientation diversity.Based antenna A..Then,we can follow Eq.(6)to write the equation on the key observation in Section III,this term is equal to 0 in based on the phase difference measured from antenna Ay as: this model,because the two tags have the same orientation and the same orientation diversity.Moreover,we simply remove △91,2= 201-2)×cos Be×sin2x (7) 入 the mod function,because the human writing is much slower Comparing Eq.(6)with Eg.(7),we have replaced cos Ba with than the sampling rate of RFID,which is later discussed in sin B,and thus we can unique describe the brush orientation Section V.Therefore,the phase difference A0z.1.2 in Eq. with the two equations.By combining Eq.(6)and Eq.(7).we (2)can be simplified as: can build a nonlinear equation set: △012=2-22m= 2△122m (3) [△0x1,2 T2l1-l2)xcos Be×>cos Ba2元 入 入 △9y,12J 244-2xcB.xsina2元 (8) where Az1.2 calculates the coordinate difference between T 入 and T2 along X-axis.By replacing the coordinate difference and the goal is to solve the brush orientation (Be,Ba).In Ax1.2 with the relative distance difference between tags and section V-C,we will demonstrate how to utilize the nonlinear brush tip along X-axis,we can further rewrite Eq.(3)as: least squares to estimate Be and Ba
Z Y X 𝑨𝒙 𝑻𝟏 𝑻𝟐 𝒍2 𝒍1 𝑨𝒚 Fig. 5: Deployment of RF-Brush. the basic deployment of RF-brush. Particularly, we attach a linear tag array on the surface of the brush for passive sensing, whose deployment is known in advance. For simplicity, we demonstrate our model with only two tags, i.e., T1 and T2, and the reader can easily extend the model to the array with multiple tags. We use (x1, y1, z1) and (x2, y2, z2) to represent the coordinates of the two tags in our model. Here, the two tags T1 and T2 are l1 and l2 distance away from the brush tip, respectively. Besides, the antenna Ax and Ay are deployed on the X-Y plane to measure the RF phase of tags from two perpendicular dimensions. Antenna Ax is on the Y-axis, which mainly measures phase change caused by tag displacement along X-axis. And antenna Ay is on the X-axis, and measures the phase change due to the displacement along Y-axis. Based on the deployment, our basic idea is to first calculate the phase difference between two tags at the same time point, then estimate the brush projection on the X-Y plane from the phase difference of the two tags, and finally leverage the brush projection to calculate the brush orientation. 1) Calculating Brush Projection from Phase Difference: Firstly, we demonstrate how to refine the phase difference between two tags, which can be further used to estimate the brush projection. Consider Fig.6 where we use antenna Ax to measure the RF phase, and present the top view of Fig.5. Here, the RF signals from Ax first reach T1 before T2. According to Eq. (1), the phase difference ∆θx,1,2 between T1 and T2 received by antenna Ax can be calculated as: ∆θx,1,2 = (2(x1 − x2) λ 2π + ∆θdev,1,2 + ∆θγ,1,2) mod 2π. (2) Here, ∆θdev,1,2 calculates the phase difference caused by the device diversity. Since the device diversity is constant for all the tags, we can remove them in advance. ∆θγ,1,2 represents the phase difference caused by the orientation diversity. Based on the key observation in Section III, this term is equal to 0 in this model, because the two tags have the same orientation and the same orientation diversity. Moreover, we simply remove the mod function, because the human writing is much slower than the sampling rate of RFID, which is later discussed in Section V. Therefore, the phase difference ∆θx,1,2 in Eq. (2)can be simplified as: ∆θx,1,2 = 2(x1 − x2) λ 2π = 2∆x1,2 λ 2π, (3) where ∆x1,2 calculates the coordinate difference between T1 and T2 along X-axis. By replacing the coordinate difference ∆x1,2 with the relative distance difference between tags and brush tip along X-axis, we can further rewrite Eq. (3) as: 𝜃𝑎 X Z 𝑨𝒙 𝜷𝒂 𝒍𝟐,𝒙 Y 𝑨𝒚 𝑻𝟏 𝑻𝟐 𝒍𝟐,𝒚 ∆𝒙𝟏,𝟐 𝒍𝟏,𝒙 Fig. 6: Relationship between phase difference and brush projection. ∆θx,1,2 = 2(l1,x − l2,x) λ 2π, (4) where li,x represents the projected distance on the X-axis of li as shown in Fig.6. In essence, the phase difference ∆θx,1,2 here measures the distance of brush projection between T1 and T2 along X-axis. 2) Estimating Brush Orientation from Brush Projection: Next, we demonstrate how to estimate the brush orientation based on l1,x−l2,x. In Fig.6, we can calculate the projection of li from the brush orientation hβe, βai based on the geometric position as: li,x = li × cos βe × cos βa. (5) Therefore, we further rewrite Eq. (4) with the brush orientation hβe, βai as: ∆θx,1,2 = 2(l1 − l2) × cos βe × cos βa λ 2π. (6) Since l1 and l2 have been measured in advance, the unknown parameters are only the brush orientation hβe, βai. So far, we have modeled the brush orientation based on the phase difference received from the antenna Ax. However, Eq. (6) still cannot uniquely express the brush orientation. Since βa ranges from 0 to 2π, cos βa and cos(2π − βa) have the same value. As a result, there is always an ambiguity orientation with the same phase difference. To solve the ambiguity and calculate the orientation, we deploy anther antenna Ay as shown in Fig.6, which is mutually orthogonal manner with the antenna Ax. So the operating plane is the central area where the beams of two antennas meets, and the antenna Ay measure the phase difference from another direction to compensate the antenna Ax. Then, we can follow Eq. (6) to write the equation based on the phase difference measured from antenna Ay as: ∆θy,1,2 = 2(l1 − l2) × cos βe × sin βa λ 2π. (7) Comparing Eq. (6) with Eq. (7), we have replaced cos βa with sin βa, and thus we can unique describe the brush orientation with the two equations. By combining Eq. (6) and Eq. (7), we can build a nonlinear equation set: ∆θx,1,2 ∆θy,1,2 = " 2(l1−l2)×cos βe×cos βa λ 2π 2(l1−l2)×cos βe×sin βa λ 2π # , (8) and the goal is to solve the brush orientation hβe, βai. In section V-C, we will demonstrate how to utilize the nonlinear least squares to estimate βe and βa
C.Modeling the 2D Movement via Linear Tag Array Next,we demonstrate how to calculate the 2D movement (x-,y-”0)T of the brush on the operating plane based on the estimated brush orientation.The basic idea is to ignore the absolute t- position of the brush tip of each instantaneous posture,and (t) estimate the relative displacement of the brush tip between the consecutive instantaneous postures.Since we are aware of the distance between each tag and the brush tip,we can thus △xB X estimate the relative displacement of the brush tip from the A relative displacement of each tag. Firstly,we calculate the relative displacement of each tag Fig.7:Modeling the displacement of brush's tip. based on the phase variation.Supposing the phase of tag result of each tag to improve the accuracy.By concatenating T;of the t-th instantaneous posture received by the antenna the displacement of brush tip between each consecutive instan- A is then the phase variation can be calculated taneous posture,we can recover the trajectory of the brush on according to Eq.(1)as: the X-Y plane. 409=09-9u-)三2△02r+a92,+Aa9 V.SYSTEM DESIGN T.i (9) A.System Overview where1)measures the displacement of RF-Brush is a 3D human-homputer interaction system, tag i of the t-th instantaneous posture along X-axis.Since the which tracks the 3D motion of tagged linear shaped object device diversityis constant all the time,the term based on the RFID technique.The basic idea is to deploy a is equal to.In regard to the term since it is proved linear tag array onto the object,and use a pair of antennas to be proportional to the orientation of the tag in section III,to track the 3D orientation and 2D movement of the linear we can measure 0 of each angle y in advance and cancel shaped object based on phase difference and phase variation. the term based on the estimated brush orientation (Be,Ba). Without loss of generality,we use the brush as the target By canceling the influence of bothandwe object to demonstrate our system.Fig.8 presents the system can thus deduce the tag displacement Ar from the phase framework of RF-brush.Particularly,the system first takes variation accordingly. the time-series RF phase of the linear tag array as input, Secondly,we further calculate the relative displacement of which is attached on the surface of brush.Then,the first brush tip from the tag displacement.Supposing in regard to module Data Preprocessing re-samples the time-series RF phase to resolve the random sampling in RFID system due the t-th instantaneous posture,the coordinate of the brush tip and tag T:are ()and ()respectively. to the Frame-Slotted-ALOHA protocol,and then removes the device diversity and periodicity of RF phase.After that,we Then as shown in Fig.7,we can build an equation as: use the 3D Orientation Estimation to estimate the orientation 9=x增+唱， (10)of the brush,which can resolve the orientation from phase where can be calculated based on Eq.(5).Thus,to difference between different tags.Next,the 2D Movement calculate the displacement of brush tip compared with the Tracking module calculates the displacement of the brush tip (-1)-th instantaneous posture,we can calculate as: based on the estimated brush orientation and phase variation △9=△8+9-) between same tag at consecutive time points.Finally,we can (11) revert the 3D motion of the brush based on the estimated 3D Therefore,we can deduce the displacement of the brush tip orientation and 2D movement. A based on the displacement of each tagx and the change of the brush projection As for the RF-signal displacement along Y-axis Au.we can follow Eq.(11)and Data Preprocessing 3D Orientation Estimating 2D Movement Tracking emove the periodicity Remove the orientation calculate it similarly.Finally,the displacement of the brush tip of Phas Diversity Data Interpolation on the X-Y plane can be represented as: Track the 2D Movemer t) △x 「(t-1)- ,(t) Calibrate Calibrate A」 △ -1)- Orientation Trajectory 4x(7 cosB(-1) (t) cosB(t-1) Fig.8:System framework. (t) t-1) △ sinBa sn6.0 cosB(t) B.Data Preprocessing (12) Data preprocessing is used to improve the reliability of In fact,based on the estimated brush orientation,we only need the RF phase by mitigating the noise in real environments one tag to estimate the 2D trajectory.Since we have attached and uniform the sampling rate in RFID system.Particular- a linear tag array on the brush,we can summary the tracking ly,in regard to the time-series RF phase of each tag,we
C. Modeling the 2D Movement via Linear Tag Array Next, we demonstrate how to calculate the 2D movement of the brush on the operating plane based on the estimated brush orientation. The basic idea is to ignore the absolute position of the brush tip of each instantaneous posture, and estimate the relative displacement of the brush tip between the consecutive instantaneous postures. Since we are aware of the distance between each tag and the brush tip, we can thus estimate the relative displacement of the brush tip from the relative displacement of each tag. Firstly, we calculate the relative displacement of each tag based on the phase variation. Supposing the phase of tag Ti of the t-th instantaneous posture received by the antenna Ax is θ (t) x,i, then the phase variation ∆θ (t) x,i can be calculated according to Eq. (1) as: ∆θ (t) x,i = θ (t) x,i − θ (t−1) x,i = 2∆x (t) i λ 2π + ∆θ (t) dev,i + ∆θ (t) γ,i, (9) where ∆x (t) i = x (t) i − x (t−1) i measures the displacement of tag i of the t-th instantaneous posture along X-axis. Since the device diversity θdev,i is constant all the time, the term ∆θ (t) dev,i is equal to 0. In regard to the term ∆θ (t) γ,i, since it is proved to be proportional to the orientation of the tag in section III, we can measure θγ of each angle γ in advance and cancel the term based on the estimated brush orientation hβe, βai. By canceling the influence of both ∆θ (t) γ,i and ∆θ (t) dev,i, we can thus deduce the tag displacement ∆x (t) i from the phase variation ∆θ (t) x,i accordingly. Secondly, we further calculate the relative displacement of brush tip from the tag displacement. Supposing in regard to the t-th instantaneous posture, the coordinate of the brush tip and tag Ti are (x (t) B , y (t) B , 0) and (x (t) i , y (t) i , z (t) i ), respectively. Then as shown in Fig.7, we can build an equation as: x (t) i = x (t) B + l (t) i,x, (10) where l (t) i,x can be calculated based on Eq. (5). Thus, to calculate the displacement of brush tip compared with the (t − 1)-th instantaneous posture, we can calculate ∆x (t) i as: ∆x (t) i = ∆x (t) B + l (t) i,x − l (t−1) i,x . (11) Therefore, we can deduce the displacement of the brush tip ∆x (t) B based on the displacement of each tag ∆x (t) i and the change of the brush projection l (t) i,x − l (t−1) i,x . As for the displacement along Y-axis ∆y (t) B , we can follow Eq. (11) and calculate it similarly. Finally, the displacement of the brush tip on the X-Y plane can be represented as: " ∆x (t) B ∆y (t) B # = " ∆x (t) i ∆y (t) i # + " l (t−1) i,x − l (t) i,x l (t−1) i,y − l (t) i,y# = " ∆x (t) i ∆y (t) i # + li × " cosβ(t−1) a , −cosβ(t) a sinβ(t−1) a , sinβ(t) a # × " cosβ(t−1) e cosβ(t) e # . (12) In fact, based on the estimated brush orientation, we only need one tag to estimate the 2D trajectory. Since we have attached a linear tag array on the brush, we can summary the tracking Z 𝑻𝒊 Y ∆𝒙𝑩 (𝒕) 𝑨𝒙 𝑨𝒚 ∆𝒚𝑩 (𝒕) (𝒙𝒊 (𝒕−𝟏) , 𝒚𝒊 (𝒕−𝟏) , 𝟎) ∆𝒙𝒊 (𝒕) (𝒙𝑩 (𝒕−𝟏) , 𝒚𝑩 (𝒕−𝟏) , 𝟎) 𝒍 𝒊,𝒙 (𝒕−𝟏) 𝒍 𝒊,𝒙 (𝒕) X Fig. 7: Modeling the displacement of brush’s tip. result of each tag to improve the accuracy. By concatenating the displacement of brush tip between each consecutive instantaneous posture, we can recover the trajectory of the brush on the X-Y plane. V. SYSTEM DESIGN A. System Overview RF-Brush is a 3D human-homputer interaction system, which tracks the 3D motion of tagged linear shaped object based on the RFID technique. The basic idea is to deploy a linear tag array onto the object, and use a pair of antennas to track the 3D orientation and 2D movement of the linear shaped object based on phase difference and phase variation. Without loss of generality, we use the brush as the target object to demonstrate our system. Fig.8 presents the system framework of RF-brush. Particularly, the system first takes the time-series RF phase of the linear tag array as input, which is attached on the surface of brush. Then, the first module Data Preprocessing re-samples the time-series RF phase to resolve the random sampling in RFID system due to the Frame-Slotted-ALOHA protocol, and then removes the device diversity and periodicity of RF phase. After that, we use the 3D Orientation Estimation to estimate the orientation of the brush, which can resolve the orientation from phase difference between different tags. Next, the 2D Movement Tracking module calculates the displacement of the brush tip based on the estimated brush orientation and phase variation between same tag at consecutive time points. Finally, we can revert the 3D motion of the brush based on the estimated 3D orientation and 2D movement. Data Preprocessing Data Interpolation Remove the Periodicity of Phase Variation 3D Orientation Estimating Remove the Periodicity of Phase Difference RF-signal [Orientation & Trajectory] Estimate the 3D Orientation Calibrate Abnormal Angle 2D Movement Tracking Remove the Orientation Diversity Track the 2D Movement Calibrate Abnormal Position Fig. 8: System framework. B. Data Preprocessing Data preprocessing is used to improve the reliability of the RF phase by mitigating the noise in real environments and uniform the sampling rate in RFID system. Particularly, in regard to the time-series RF phase of each tag, we