2146 IEEE/ACM TRANSACTIONS ON NETWORKING,VOL.26.NO.5.OCTOBER 2018 Synchronize Inertial Readings From Multiple Mobile Devices in Spatial Dimension Lei Xie,Member,IEEE,Qingliang Cai,Student Member,IEEE,Alex X.Liu,Senior Member,IEEE, Wei Wang,Member;IEEE,Yafeng Yin,Member,IEEE,and Sanglu Lu,Member,IEEE Abstract-In this paper,we investigate the problem of space synchronization,i.e,synchronizing inertial readings from multiple mobile devices in the spatial dimension,in other words,mul- tiple mobile devices are space synchronized to have the same 3-D coordinates except that each device is the origin of its corresponding coordinate.We propose a scheme called MObile Space Synchronization (MOSS)for devices with two sensors: an accelerometer and a gyroscope,which are available on most Local Coordinat (a) mobile devices.Accelerometer readings from multiple mobile 6) devices on a human subject are used to achieve space synchroniza- Fig.1.Synchronize inertial readings from multi-devices in spatial dimension tion when the human subject is moving forward,such as walking (a)Synchronize 3-D coordinates among devices in VR games.(b)Local and running.Gyroscope readings from multiple mobile devices coordinates vs synchronized coordinates. on a human subject are used to maintain space synchronization when the human subject stops moving forward,which means that multiple mobile devices are space synchronized to have the we can no longer obtain the consistent acceleration caused by same 3-D coordinates except that each device is the origin body moving forward.Experiment results show that our MOSS scheme can achieve an average angle deviation of 9.8 and an of its corresponding coordinate.A modern mobile device is average measurement similarity of 97%. often equipped with an Inertial Measurement Unit (IMU), Index Terms-Space synchronization,mobile device which typically includes an accelerometer and a gyroscope, and sometimes a magnetometer as well for advanced models. I.INTRODUCTION A.Motivation The readings from IMU sensors are based on a local coordinate determined by the orientation of its body frame (or its mother TOWADAYS,the mobile devices equipped with inertial board to be more precise).Fig.1(b)shows the two local sensors are widely used to perform motion sensing and coordinates of a smartphone and a smartwatch,respectively, human computer interaction [1]-[4].For example,in virtual in solid arrows.For the multiple mobile devices placed at dif- reality games,a moving human subject may be equipped with ferent body parts of a human subject,due to the different orien- multiple mobile devices on different body parts(such as heads, tation of their body frames,their local coordinates are mostly arms,and legs)to capture the human's movement,as shown likely different.Without space synchronization,the readings in Fig.1(a).In this case,it is very essential to fuse the from the IMU sensors of different mobile devices are difficult readings from multiple mobile devices based on synchronized to be correlated with each other.With space synchronization, coordinates,so that they can be combined together to recover these readings can be used jointly to better describe human better orientation of the users'body parts and recognize the movements.We call the synchronized coordinate of a device as human activities. its global coordinate.Fig.1(a)and (b)shows the synchronized In this paper,we investigate the problem of space synchro- coordinates of the devices in dashed arrows.Space synchro- nization,i.e.,synchronizing inertial readings from multiple nization allows us to capture the movement of different body mobile devices in the spatial dimension,in other words, parts in the aligned 3-D coordinates,which helps to build more Manuscript received October 29,2017:revised May 10,2018;accepted accurate 3-D models of human actions. July 17,2018:approved by IEEE/ACM TRANSACTIONS ON NETWORKING A straightforward solution to mobile device space syn- Editor K.Tang.Date of publication August 14,2018;date of current chronization is to use compasses or magnetometer sensors in version October 15,2018.This work was supported in part by the National Natural Science Foundation of China under Grant 61472185,Grant 61472184 mobile devices.Magnetometer readings allow each device to Grant 61321491,Grant 61502224,and Grant 61702257,in part by the obtain the same magnetic field direction of the earth.Together Jiangsu Natural Science Foundation under Grant BK20151390 and Grant with the same gravity direction of each device,all devices BK20170648.in part by the Fundamental Research Funds for the Central Universities under Grant 020214380035.in part by the National Science can therefore achieve space synchronization.This solution, Foundation under Grant CNS-1421407,in part by the Jiangsu Innovation and although simple,have two weaknesses.First,most state-of- Entrepreneurship (Shuangchuang)Program,and in part by the Collaborative the-art wearable devices (such as Apple Watch S1,Samsung Innovation Center of Novel Software Technology and Industrialization.(Cor responding authors:Alex X.Liu:Sanglu Lu.) Watch Gear 2,and MOTO Watch 360)are not equipped The authors are with the State Key Laboratory for Novel Software with magnetometer sensors;in comparison,accelerometer and Technology.Nanjing University,Nanjing 210023,China (e-mail:Ixie@nju. gyroscope sensors are available in most wearable devices.Sec- edu.cn:qingliangcai@dislab.nju.edu.cn:alexliu@nju.edu.cn:ww@nju.edu.cn: yafeng@nju.edu.cn:sanglu@nju.edu.cn). ond,magnetometer readings are often notoriously inaccurate Digital Object Identifier 10.1109/TNET.2018.2859246 in indoor environments 5,due to the magnetic interferences 1063-66922018 IEEE.Personal use is permitted,but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information
2146 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 26, NO. 5, OCTOBER 2018 Synchronize Inertial Readings From Multiple Mobile Devices in Spatial Dimension Lei Xie , Member, IEEE, Qingliang Cai, Student Member, IEEE, Alex X. Liu, Senior Member, IEEE, Wei Wang, Member, IEEE, Yafeng Yin, Member, IEEE, and Sanglu Lu, Member, IEEE Abstract— In this paper, we investigate the problem of space synchronization, i.e., synchronizing inertial readings from multiple mobile devices in the spatial dimension, in other words, multiple mobile devices are space synchronized to have the same 3-D coordinates except that each device is the origin of its corresponding coordinate. We propose a scheme called MObile Space Synchronization (MOSS) for devices with two sensors: an accelerometer and a gyroscope, which are available on most mobile devices. Accelerometer readings from multiple mobile devices on a human subject are used to achieve space synchronization when the human subject is moving forward, such as walking and running. Gyroscope readings from multiple mobile devices on a human subject are used to maintain space synchronization when the human subject stops moving forward, which means that we can no longer obtain the consistent acceleration caused by body moving forward. Experiment results show that our MOSS scheme can achieve an average angle deviation of 9.8◦ and an average measurement similarity of 97%. Index Terms— Space synchronization, mobile device. I. INTRODUCTION A. Motivation NOWADAYS, the mobile devices equipped with inertial sensors are widely used to perform motion sensing and human computer interaction [1]–[4]. For example, in virtual reality games, a moving human subject may be equipped with multiple mobile devices on different body parts (such as heads, arms, and legs) to capture the human’s movement, as shown in Fig. 1(a) . In this case, it is very essential to fuse the readings from multiple mobile devices based on synchronized coordinates, so that they can be combined together to recover better orientation of the users’ body parts and recognize the human activities. In this paper, we investigate the problem of space synchronization, i.e., synchronizing inertial readings from multiple mobile devices in the spatial dimension, in other words, Manuscript received October 29, 2017; revised May 10, 2018; accepted July 17, 2018; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor K. Tang. Date of publication August 14, 2018; date of current version October 15, 2018. This work was supported in part by the National Natural Science Foundation of China under Grant 61472185, Grant 61472184, Grant 61321491, Grant 61502224, and Grant 61702257, in part by the Jiangsu Natural Science Foundation under Grant BK20151390 and Grant BK20170648, in part by the Fundamental Research Funds for the Central Universities under Grant 020214380035, in part by the National Science Foundation under Grant CNS-1421407, in part by the Jiangsu Innovation and Entrepreneurship (Shuangchuang) Program, and in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization. (Corresponding authors: Alex X. Liu; Sanglu Lu.) The authors are with the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China (e-mail: lxie@nju. edu.cn; qingliangcai@dislab.nju.edu.cn; alexliu@nju.edu.cn; ww@nju.edu.cn; yafeng@nju.edu.cn; sanglu@nju.edu.cn). Digital Object Identifier 10.1109/TNET.2018.2859246 Fig. 1. Synchronize inertial readings from multi-devices in spatial dimension. (a) Synchronize 3-D coordinates among devices in VR games. (b) Local coordinates vs synchronized coordinates. multiple mobile devices are space synchronized to have the same 3-D coordinates except that each device is the origin of its corresponding coordinate. A modern mobile device is often equipped with an Inertial Measurement Unit (IMU), which typically includes an accelerometer and a gyroscope, and sometimes a magnetometer as well for advanced models. The readings from IMU sensors are based on a local coordinate determined by the orientation of its body frame (or its mother board to be more precise). Fig. 1(b) shows the two local coordinates of a smartphone and a smartwatch, respectively, in solid arrows. For the multiple mobile devices placed at different body parts of a human subject, due to the different orientation of their body frames, their local coordinates are mostly likely different. Without space synchronization, the readings from the IMU sensors of different mobile devices are difficult to be correlated with each other. With space synchronization, these readings can be used jointly to better describe human movements. We call the synchronized coordinate of a device as its global coordinate. Fig. 1(a) and (b) shows the synchronized coordinates of the devices in dashed arrows. Space synchronization allows us to capture the movement of different body parts in the aligned 3-D coordinates, which helps to build more accurate 3-D models of human actions. A straightforward solution to mobile device space synchronization is to use compasses or magnetometer sensors in mobile devices. Magnetometer readings allow each device to obtain the same magnetic field direction of the earth. Together with the same gravity direction of each device, all devices can therefore achieve space synchronization. This solution, although simple, have two weaknesses. First, most state-ofthe-art wearable devices (such as Apple Watch S1, Samsung Watch Gear 2, and MOTO Watch 360) are not equipped with magnetometer sensors; in comparison, accelerometer and gyroscope sensors are available in most wearable devices. Second, magnetometer readings are often notoriously inaccurate in indoor environments [5], due to the magnetic interferences 1063-6692 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information
XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2147 caused by the massive steel embedded in building concrete to these three axes.we can build the synchronized coordinates structures and other metallic objects.Our empirical study with each device as the origins. further shows that the magnetometer readings are not reli- Furthermore,we use gyroscope readings from multiple able in typical indoor environment.Specifically,for differ- mobile devices on a human subject to maintain space syn- ent positions in the indoor environment,the angle deviation chronization when the human subject stops moving forward, between the magnetometer measurement and the ground-truth which means that we can no longer obtain the consistent north can be as large as 110,the standard deviation of the acceleration caused by forward body motion.After the human magnetometer readings can be as large as 26.7;for a fixed subject stops moving forward,his body parts may still slightly position in the indoor environment,with the interferences from move or rotate.The gyroscope readings of a mobile device different sources such as smart phone,earphone,metal plate, allow us to continuously track the small rotations along and magnet,the angle deviation between the magnetometer the three axes of the device's local coordinate.We derive measurement and the ground-truth north can be as large as a real-time rotation matrix corresponding to the orientation 133.6,the standard deviation of the angle deviation can be variation by integrating the rotation rates in different axes over as large as 86.3 time.Leveraging the stability of the gravity direction in the synchronized coordinates,we further calibrate the estimated B.Proposed Approach rotation matrix from gyroscope tracking.Thus,we are able to In this paper,we propose a scheme called MObile Space maintain space synchronization even after the human subject Synchronization (MOSS)for devices with two sensors:an stops moving. accelerometer and a gyroscope,which are available on most mobile devices.Accelerometer readings from multiple mobile C.Technical Challenges and Solutions devices on a human subject are used to achieve space synchro- The first challenge is to extract the consistent acceleration nization when the human subject is moving forward,such as of body movement when the human subject moves forward. walking and running.Our insight on using accelerometer sen- This is challenging because both consistent and inconsistent sors to achieve space synchronization is that when the human accelerations are mixed together.As the mobile devices are subject moves forward,all attached mobile devices experience attached to different body locations of the human subject,they the same acceleration along the moving direction of the torso, perceive rather different accelerations in both the direction and which we call consistent acceleration.Although these mobile magnitude during human motion.To address this challenge, devices also experience various other accelerations due to we propose a principal component analysis (PCA)based intra-body movements (such as arm and leg movements), scheme to remove the inconsistent accelerations from the which we call inconsistent acceleration.they are usually much observed accelerations.The key observation is that the con- smaller than the consistent acceleration because intra-body sistent acceleration contributes to the observed accelerations movements are usually orders of magnitude smaller than the of all mobile devices attached to the human subject.In other forward movement.When the human subject is moving for- words,the observed acceleration signal of each device is the ward,the approach of simply adding an inertial sensor on the combination of the consistent acceleration signal,which is human chest to directly measure the consistent acceleration from the forwarding movement and the same for all devices, seems to be a feasible solution.however,it cannot work and the inconsistent acceleration signal,which is from the effectively due to the following reasons:First,it still requires intra-body movement from and unique to the device itself. the other mobile devices to synchronize with this specified Furthermore,we observe that the inconsistent acceleration device in the spatial dimension,as the consistent acceleration from the intra-body movement cancels each other out during can only be extracted directly from this device rather than the back and forth movement,and its expected value is close all devices.Usually a magnetometer is essentially required to 0 within a large enough time interval.Thus,the observed for all devices to achieve this synchronization,however,this acceleration signals of multiple devices are strongly correlated, is contradictory to the situation we need to tackle.Second. and therefore we can use PCA-based approach to cancel out even for this specified device.it may lead to inaccuracy the inconsistent factor and extract the consistent factor. in directly measuring the consistent acceleration,since the The second challenge is to address the accumulated errors moving human body may usually introduce some inconsistent in maintaining space synchronization when the human sub- accelerations,more or less,into the measurements on the hor- ject stops moving forward.It is well known that the errors izontal plane.Hence,in this paper,we choose to compute the in gyroscope based oriental tracking accumulate [6],[7]. forwarding direction by extracting the consistent acceleration Furthermore,the errors are further exacerbated by the large from the accelerometer readings.We can treat the inconsistent angular velocities and linear accelerations in human body acceleration as noises and use signal processing techniques motion.Current solutions primarily rely on Kalman filters; to filter them out mostly.Moreover,using a low pass filter however,they only use a single data source from the gyroscope such as a Butterworth filter,the gravitational acceleration to calibrate the accumulated errors [8],which is not sufficient can be extracted from the acceleration measurements in the to further reduce the errors in maintaining space synchro- local coordinate system.Therefore,based on the forwarding nization.To address this challenge,we propose a simple direction and the gravitational direction as reference axes, but effective complementary filter to calibrate the gyroscope the third axis can also be obtained since it is perpendicular to tracking based on the stability of the gravity direction.We the plane defined by these two reference axes.Thus,according propose a rotation model that defines the rotations in the
XIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2147 caused by the massive steel embedded in building concrete structures and other metallic objects. Our empirical study further shows that the magnetometer readings are not reliable in typical indoor environment. Specifically, for different positions in the indoor environment, the angle deviation between the magnetometer measurement and the ground-truth north can be as large as 110◦, the standard deviation of the magnetometer readings can be as large as 26.7◦; for a fixed position in the indoor environment, with the interferences from different sources such as smart phone, earphone, metal plate, and magnet, the angle deviation between the magnetometer measurement and the ground-truth north can be as large as 133.6◦, the standard deviation of the angle deviation can be as large as 86.3◦. B. Proposed Approach In this paper, we propose a scheme called MObile Space Synchronization (MOSS) for devices with two sensors: an accelerometer and a gyroscope, which are available on most mobile devices. Accelerometer readings from multiple mobile devices on a human subject are used to achieve space synchronization when the human subject is moving forward, such as walking and running. Our insight on using accelerometer sensors to achieve space synchronization is that when the human subject moves forward, all attached mobile devices experience the same acceleration along the moving direction of the torso, which we call consistent acceleration. Although these mobile devices also experience various other accelerations due to intra-body movements (such as arm and leg movements), which we call inconsistent acceleration, they are usually much smaller than the consistent acceleration because intra-body movements are usually orders of magnitude smaller than the forward movement. When the human subject is moving forward, the approach of simply adding an inertial sensor on the human chest to directly measure the consistent acceleration seems to be a feasible solution, however, it cannot work effectively due to the following reasons: First, it still requires the other mobile devices to synchronize with this specified device in the spatial dimension, as the consistent acceleration can only be extracted directly from this device rather than all devices. Usually a magnetometer is essentially required for all devices to achieve this synchronization, however, this is contradictory to the situation we need to tackle. Second, even for this specified device, it may lead to inaccuracy in directly measuring the consistent acceleration, since the moving human body may usually introduce some inconsistent accelerations, more or less, into the measurements on the horizontal plane. Hence, in this paper, we choose to compute the forwarding direction by extracting the consistent acceleration from the accelerometer readings. We can treat the inconsistent acceleration as noises and use signal processing techniques to filter them out mostly. Moreover, using a low pass filter such as a Butterworth filter, the gravitational acceleration can be extracted from the acceleration measurements in the local coordinate system. Therefore, based on the forwarding direction and the gravitational direction as reference axes, the third axis can also be obtained since it is perpendicular to the plane defined by these two reference axes. Thus, according to these three axes, we can build the synchronized coordinates with each device as the origins. Furthermore, we use gyroscope readings from multiple mobile devices on a human subject to maintain space synchronization when the human subject stops moving forward, which means that we can no longer obtain the consistent acceleration caused by forward body motion. After the human subject stops moving forward, his body parts may still slightly move or rotate. The gyroscope readings of a mobile device allow us to continuously track the small rotations along the three axes of the device’s local coordinate. We derive a real-time rotation matrix corresponding to the orientation variation by integrating the rotation rates in different axes over time. Leveraging the stability of the gravity direction in the synchronized coordinates, we further calibrate the estimated rotation matrix from gyroscope tracking. Thus, we are able to maintain space synchronization even after the human subject stops moving. C. Technical Challenges and Solutions The first challenge is to extract the consistent acceleration of body movement when the human subject moves forward. This is challenging because both consistent and inconsistent accelerations are mixed together. As the mobile devices are attached to different body locations of the human subject, they perceive rather different accelerations in both the direction and magnitude during human motion. To address this challenge, we propose a principal component analysis (PCA) based scheme to remove the inconsistent accelerations from the observed accelerations. The key observation is that the consistent acceleration contributes to the observed accelerations of all mobile devices attached to the human subject. In other words, the observed acceleration signal of each device is the combination of the consistent acceleration signal, which is from the forwarding movement and the same for all devices, and the inconsistent acceleration signal, which is from the intra-body movement from and unique to the device itself. Furthermore, we observe that the inconsistent acceleration from the intra-body movement cancels each other out during the back and forth movement, and its expected value is close to 0 within a large enough time interval. Thus, the observed acceleration signals of multiple devices are strongly correlated, and therefore we can use PCA-based approach to cancel out the inconsistent factor and extract the consistent factor. The second challenge is to address the accumulated errors in maintaining space synchronization when the human subject stops moving forward. It is well known that the errors in gyroscope based oriental tracking accumulate [6], [7]. Furthermore, the errors are further exacerbated by the large angular velocities and linear accelerations in human body motion. Current solutions primarily rely on Kalman filters; however, they only use a single data source from the gyroscope to calibrate the accumulated errors [8], which is not sufficient to further reduce the errors in maintaining space synchronization. To address this challenge, we propose a simple but effective complementary filter to calibrate the gyroscope tracking based on the stability of the gravity direction. We propose a rotation model that defines the rotations in the
2148 IEEE/ACM TRANSACTIONS ON NETWORKING,VOL.26.NO.5.OCTOBER 2018 orthogonal and parallel directions of the gravity,respectively. We then use this model to calibrate the gyroscope-based estimation with the Minimum Mean Square Error (MMSE) estimator. Smant Phone:斜Sumoung3 D.Summary of Experimental Results We implemented our MOSS system on mobile devices including Google Glass and Samsung S5 smart phones.They Fig.2.Measurement setup are deployed at different body locations of each human subject. We let the human subjects walk along different types of traces angles of a human subject so that the indoor pathway maps in outdoor environments with different moving modes,i.e., can be obtained [21].Wang et al.[22]used compasss,gyro- walk,run,and jump.We use two main metrics to evaluate scopes,and WiFi landmarks to estimate the absolute walking the performance:(1)angle accuracy:the angle deviation direction of a human subject.Of prior work in this category, between the estimated human body movement direction and only the APT system does not use magnetometer sensors [24]; the ground truth,and (2)coordinate accuracy:the similarity instead,APT uses accelerometer and gyroscope sensors to between the synchronized coordinate and the ground truth. obtain the walking direction of a human subject.Compared Experiment results show that MOSS achieves an average angle with our work,both the WalkCompass and APT does not accuracy of 9.8 and an average coordinate accuracy of 97%. address space synchronization among multiple mobile devices A real-world case study with free activities further shows that as they use only one smartphone.In contrast,in this paper MOSS achieves an average angle accuracy of 12 and an we investigate multiple mobile devices instead of one device average coordinate accuracy of 91%. to synchronize the inertial readings from multiple devices in spatial dimension.We propose a more generalized solution II.RELATED WORK to heading direction estimation for multiple wearable devices Orientation Estimation:Much work has been done on subject to different but correlated accelerations.Our solution estimating the orientation of a mobile device (such as a neither relies on the inaccurate magnetometer measurements smartphone)using accelerometer,gyroscope,and magne- nor uses any application specific features tometer sensors [2],[6].[9]-[17].Compared with our work, most of such work uses magnetometers.For example III.UNDERSTANDING HUMAN MOTIONS Madgwick et al.[13]proposed a quaternion representation to incorporate accelerometer and magnetometer readings for A.Measurements orientation estimation.Zhou et al.[6]used the accelerometer Measurement Setup:We placed six mobile devices,one and magnetometer to assist gyroscope in orientation estimation smart glass(Google Glass 2)and five smartphones(Samsung by selecting the best sensing capabilities.Gowda et al.[14] Galaxy S5).at different body location of a human subject tried to map from a local frame of the sensor to a global frame to continuously collect the inertial measurements in his daily of reference,so as to track a ball's 3D trajectory and spin life.As shown in Fig.2,the Google Glass was placed on with inertial sensors and radios embedded in the ball.Of prior the head and the five phones were placed at five different work in this category,only two systems,Autowitness [18]locations on the body.These devices are all equipped with an and Nericell [19],which finds the rotation matrix between the accelerometer,a gyroscope,and a magnetometer.We use the local coordinate of a smartphone and the reference coordinate three axes obtained from the magnetometers as the reference by the acceleration measurement,did not use magnetometers. global coordinate. They used a vehicle's forwarding accelerations (i.e.,speeding Measurement of Human Body Movements:We observed that up and slowing down)obtained from accelerometer readings. when the human subject attached with multiple mobile devices Compared with our work,they have two key limitations as moves forward,the inconsistent accelerations from intra-body they attach mobile devices to moving vehicles rather than movement have different directions and magnitude.To extract human subjects.First,as there is no intra-body movements the consistent and the inconsistent accelerations,we let the and a vehicle moves much straighter than human moves,they human subject walk along a straight path for 30 seconds.We do not need to deal with our first technical challenge.Second, attached an additional IMU sensor on the chest to estimate the as the mobile devices do not move after a vehicle stops,they ground-truth of the consistent acceleration as the inconsistent do not need to deal with our second technical challenge. accelerations on the chest is negligible.We collected the Direction Estimation:Recently some work explored the acceleration measurements from the six devices as mixed estimation of heading directions for dead reckoning-based accelerations and then extract the inconsistent accelerations navigation schemes [20]-[26].Compared with our work,most from each device by subtracting the consistent acceleration. of such work uses magnetometers to estimate the heading To illustrate the direction and magnitude of both the consistent directions while trying to mitigate the magnetic interference and inconsistent accelerations,we plot them as vectors in the from indoor environments.For example,WalkCompass used polar coordinate system corresponding to the earth coordinate magnetometers to estimate the walking direction of a human system.We plot a vector of the inconsistent acceleration subject [20.Walkie-Markie used the magnetometer and gyro- for every 100 ms during the time interval of 5 seconds. scopes in smartphones to get the walking direction and turning Fig.3 shows the directions and magnitudes of the original
2148 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 26, NO. 5, OCTOBER 2018 orthogonal and parallel directions of the gravity, respectively. We then use this model to calibrate the gyroscope-based estimation with the Minimum Mean Square Error (MMSE) estimator. D. Summary of Experimental Results We implemented our MOSS system on mobile devices including Google Glass and Samsung S5 smart phones. They are deployed at different body locations of each human subject. We let the human subjects walk along different types of traces in outdoor environments with different moving modes, i.e., walk, run, and jump. We use two main metrics to evaluate the performance: (1) angle accuracy: the angle deviation between the estimated human body movement direction and the ground truth, and (2) coordinate accuracy: the similarity between the synchronized coordinate and the ground truth. Experiment results show that MOSS achieves an average angle accuracy of 9.8◦ and an average coordinate accuracy of 97%. A real-world case study with free activities further shows that MOSS achieves an average angle accuracy of 12◦ and an average coordinate accuracy of 91%. II. RELATED WORK Orientation Estimation: Much work has been done on estimating the orientation of a mobile device (such as a smartphone) using accelerometer, gyroscope, and magnetometer sensors [2], [6], [9]–[17]. Compared with our work, most of such work uses magnetometers. For example, Madgwick et al. [13] proposed a quaternion representation to incorporate accelerometer and magnetometer readings for orientation estimation . Zhou et al. [6] used the accelerometer and magnetometer to assist gyroscope in orientation estimation by selecting the best sensing capabilities. Gowda et al. [14] tried to map from a local frame of the sensor to a global frame of reference, so as to track a ball’s 3D trajectory and spin with inertial sensors and radios embedded in the ball. Of prior work in this category, only two systems, Autowitness [18] and Nericell [19], which finds the rotation matrix between the local coordinate of a smartphone and the reference coordinate by the acceleration measurement, did not use magnetometers. They used a vehicle’s forwarding accelerations (i.e., speeding up and slowing down) obtained from accelerometer readings. Compared with our work, they have two key limitations as they attach mobile devices to moving vehicles rather than human subjects. First, as there is no intra-body movements and a vehicle moves much straighter than human moves, they do not need to deal with our first technical challenge. Second, as the mobile devices do not move after a vehicle stops, they do not need to deal with our second technical challenge. Direction Estimation: Recently some work explored the estimation of heading directions for dead reckoning-based navigation schemes [20]–[26]. Compared with our work, most of such work uses magnetometers to estimate the heading directions while trying to mitigate the magnetic interference from indoor environments. For example, WalkCompass used magnetometers to estimate the walking direction of a human subject [20]. Walkie-Markie used the magnetometer and gyroscopes in smartphones to get the walking direction and turning Fig. 2. Measurement setup. angles of a human subject so that the indoor pathway maps can be obtained [21]. Wang et al. [22] used compasss, gyroscopes, and WiFi landmarks to estimate the absolute walking direction of a human subject. Of prior work in this category, only the APT system does not use magnetometer sensors [24]; instead, APT uses accelerometer and gyroscope sensors to obtain the walking direction of a human subject. Compared with our work, both the WalkCompass and APT does not address space synchronization among multiple mobile devices as they use only one smartphone. In contrast, in this paper we investigate multiple mobile devices instead of one device to synchronize the inertial readings from multiple devices in spatial dimension. We propose a more generalized solution to heading direction estimation for multiple wearable devices subject to different but correlated accelerations. Our solution neither relies on the inaccurate magnetometer measurements nor uses any application specific features. III. UNDERSTANDING HUMAN MOTIONS A. Measurements Measurement Setup: We placed six mobile devices, one smart glass (Google Glass 2) and five smartphones (Samsung Galaxy S5), at different body location of a human subject to continuously collect the inertial measurements in his daily life. As shown in Fig. 2, the Google Glass was placed on the head and the five phones were placed at five different locations on the body. These devices are all equipped with an accelerometer, a gyroscope, and a magnetometer. We use the three axes obtained from the magnetometers as the reference global coordinate. Measurement of Human Body Movements: We observed that when the human subject attached with multiple mobile devices moves forward, the inconsistent accelerations from intra-body movement have different directions and magnitude. To extract the consistent and the inconsistent accelerations, we let the human subject walk along a straight path for 30 seconds. We attached an additional IMU sensor on the chest to estimate the ground-truth of the consistent acceleration as the inconsistent accelerations on the chest is negligible. We collected the acceleration measurements from the six devices as mixed accelerations and then extract the inconsistent accelerations from each device by subtracting the consistent acceleration. To illustrate the direction and magnitude of both the consistent and inconsistent accelerations, we plot them as vectors in the polar coordinate system corresponding to the earth coordinate system. We plot a vector of the inconsistent acceleration for every 100 ms during the time interval of 5 seconds. Fig. 3 shows the directions and magnitudes of the original
XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2149 12090180 120 9010 120 90180 120 902%0 150 30 150 30 150 30 150 10 30 180 180 0 180 180 210 330 210 330 210 330 210 330 240270300 240270300 240270300 240270300 (a b (c) (d) Fig.3.The directions (and magnitudes (m/s2)of the original/consistent/inconsistent accelerations in different devices.(a)S1.(b)S2.(c)S3.(d)56. ■出口多塞法口型 口出强口澳类多口类 口光口器口数图 (2) Fig.4.The mean and standard deviation of different accelerations.(a)The mixed accelerations (m/s2).(b)The consistent accelerations (m/s?).(c)The inconsistent accelerations (m/s). accelerations,the consistent accelerations,and the inconsistent acceleration f(t)and inconsistent acceleration f(t),ie., accelerations,respectively,with the color green,red and blue. fi(t)=fe(t)+f(t).The accelerations are originally measured Without loss of generality,we plot these accelerations in the according to the local coordinate system of the device.As polar system for the devices S1,S2,S3 and S6.We observed the human subject is moving,the local coordinate system that,for the inconsistent accelerations,the average magnitudes is rotating over time relative to the earth coordinate system; of the inconsistent accelerations are different among different thus.the acceleration measurements cannot describe human devices,because they depend on the motion amplitudes in motions consistently.Suppose we can build a fixed coordinate different body parts.Moreover,the directions of the incon- system relative to the earth coordinate system.For the fixed sistent accelerations are distributed by and large evenly on coordinate system,we use fi(t),fi.(t),and fi.=(t)to denote the horizontal plane,because the accelerations of intra-body the projections of fi(t)in y and 2 axes,respectively,and movements often mutually offset each other in opposite direc- use f(t),f(t).and f (t)to denote the projections of tions due to back and forth body part movement such as arm f(t)in y.and z axes,respectively.Let the angles between swing. fe(t)and each x,y and z axis at time t be ai(t).Bi(t). We observed that the mean value of inconsistent accel- and i(t),respectively.Since the consistent acceleration fe(t) erations is close to 0 in each direction for a sufficiently has a fixed direction with respect to the coordinate system, large time window (such as 10 seconds),although the stan- i(t),Bi(t),and (t)are all constant values over time;thus, dard deviations of inconsistent accelerations are fairly large. we denote them as ai,Bi,and i,respectively,for simplicity. Fig.4 shows the mean and standard deviation of the mixed, Thus,the direction of fe(t)can be represented as a unit vector consistent,and inconsistent accelerations,respectively,in the (cos ai,cos Bi,cosYi).Let fe(t)denote the magnitude of the forward-back direction and the left-right direction of human consistent acceleration in the forwarding direction;thus,its motions.We observed that although the standard deviations projection on the y and z axes are fe(t)cos ai,fe(t)cos Bi, of all accelerations are fairly large,their mean values are all and fe(t)cosi,respectively.The projections on the three axes relatively small.In regard to the inconsistent accelerations, are as follows: for both directions.the standard deviations are usually in the order of 1m/s2.The standard deviations in the forward-back fi.r(t)=fe(t)cosai+fir(t), direction are significantly greater than the left-right direction, fi.y(t)=fe(t)cosBi+fi.y(t), (1) due to the reason that the human subject is moving for- fi.z(t)=fe(t)cosvi+fz(t). ward.However.for both directions.the mean values of the inconsistent accelerations are all in the order of 10-5m/s2. Fig.5 shows the relationship among fi(t),fe(t),and f(t), This implies that,although the inconsistent accelerations have as well as the projection of fe(t)on the three axes. different directions and magnitudes,the overall impact of the Thus,if we can compute the value of cos ai,cos Bi,and inconsistent accelerations from the intra-body movement can cos i,we can calculate the direction of the consistent acceler- be negligible statistically. ation f(t)in the corresponding coordinate system.Further,for any inconsistent accelerations corresponding to the intra-body movement,we observe that the mean value of the inconsistent B.Modeling of Human Motions accelerations is close to 0,as long as the time interval Let fi(t)denote the mixed acceleration measured from is large enough.The following theorem captures the above device Di at time t.Here fi(t)consists of consistent characteristics:
XIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2149 Fig. 3. The directions (◦) and magnitudes (m/s2) of the original/consistent/inconsistent accelerations in different devices. (a) S1. (b) S2. (c) S3. (d) S6. Fig. 4. The mean and standard deviation of different accelerations. (a) The mixed accelerations (m/s2). (b) The consistent accelerations (m/s2). (c) The inconsistent accelerations (m/s2). accelerations, the consistent accelerations, and the inconsistent accelerations, respectively, with the color green, red and blue. Without loss of generality, we plot these accelerations in the polar system for the devices S1, S2, S3 and S6. We observed that, for the inconsistent accelerations, the average magnitudes of the inconsistent accelerations are different among different devices, because they depend on the motion amplitudes in different body parts. Moreover, the directions of the inconsistent accelerations are distributed by and large evenly on the horizontal plane, because the accelerations of intra-body movements often mutually offset each other in opposite directions due to back and forth body part movement such as arm swing. We observed that the mean value of inconsistent accelerations is close to 0 in each direction for a sufficiently large time window (such as 10 seconds), although the standard deviations of inconsistent accelerations are fairly large. Fig. 4 shows the mean and standard deviation of the mixed, consistent, and inconsistent accelerations, respectively, in the forward-back direction and the left-right direction of human motions. We observed that although the standard deviations of all accelerations are fairly large, their mean values are all relatively small. In regard to the inconsistent accelerations, for both directions, the standard deviations are usually in the order of 1m/s2. The standard deviations in the forward-back direction are significantly greater than the left-right direction, due to the reason that the human subject is moving forward. However, for both directions, the mean values of the inconsistent accelerations are all in the order of 10−5m/s2. This implies that, although the inconsistent accelerations have different directions and magnitudes, the overall impact of the inconsistent accelerations from the intra-body movement can be negligible statistically. B. Modeling of Human Motions Let fi(t) denote the mixed acceleration measured from device Di at time t. Here fi(t) consists of consistent acceleration fc(t) and inconsistent acceleration f i(t), i.e., fi(t) = fc(t)+f i(t). The accelerations are originally measured according to the local coordinate system of the device. As the human subject is moving, the local coordinate system is rotating over time relative to the earth coordinate system; thus, the acceleration measurements cannot describe human motions consistently. Suppose we can build a fixed coordinate system relative to the earth coordinate system. For the fixed coordinate system, we use fi,x(t), fi,y(t), and fi,z(t) to denote the projections of fi(t) in x, y and z axes, respectively, and use f i,x(t), f i,y(t), and f i,z(t) to denote the projections of f i(t) in x, y, and z axes, respectively. Let the angles between fc(t) and each x, y and z axis at time t be αi(t), βi(t), and γi(t), respectively. Since the consistent acceleration fc(t) has a fixed direction with respect to the coordinate system, αi(t), βi(t), and γi(t) are all constant values over time; thus, we denote them as αi, βi, and γi, respectively, for simplicity. Thus, the direction of fc(t) can be represented as a unit vector cos αi, cos βi, cos γi. Let fc(t) denote the magnitude of the consistent acceleration in the forwarding direction; thus, its projection on the x, y and z axes are fc(t) cos αi, fc(t) cos βi, and fc(t) cos γi, respectively. The projections on the three axes are as follows: ⎧ ⎪⎨ ⎪⎩ fi,x(t) = fc(t) cos αi + f i,x(t), fi,y(t) = fc(t) cos βi + f i,y(t), fi,z(t) = fc(t) cos γi + f i,z(t). (1) Fig.5 shows the relationship among fi(t), fc(t), and f i(t), as well as the projection of fc(t) on the three axes. Thus, if we can compute the value of cos αi, cos βi, and cos γi, we can calculate the direction of the consistent acceleration fc(t) in the corresponding coordinate system. Further, for any inconsistent accelerations corresponding to the intra-body movement, we observe that the mean value of the inconsistent accelerations is close to 0, as long as the time interval is large enough. The following theorem captures the above characteristics:
2150 IEEE/ACM TRANSACTIONS ON NETWORKING,VOL.26.NO.5.OCTOBER 2018 Z axis "地 Z axis constant f System Bi Y axis 阳 gravitational Y axis acceleration g Fig.5.fi(t)consists of consistent acceleration f(t)and inconsistent acceleration f(t).the projection of fe(t)on the r.y and 2 axes are Fig.6.Deriving the global coordinate. fe(t)cos ai,fe(t)cos Bi,and fe(t)cos Yi.respectively. Theorem 1:For a fixed coordinate system relative to the A vector quantity v defined in the local coordinate system earth coordinate system,for any inconsistent acceleration f.(t) is equivalent to the vector v=C.v defined in the global caused by intra-body movements,the expected value of f(t) coordinate.The inverse transformation is v=C.v as the is approximately equal to 0 within a sufficiently large time inverse of a rotation matrix is equal to its transpose. interval [ts,te].That is, For the local coordinate of a device,assuming that we can extract a constant acceleration as a vector fe from the E':（训=te-t。J 1 E(t)dt≈0. (2) acceleration measurements and extract a constant gravitational acceleration as a vector g from the low pass filter (such as the Proof:Without loss of generality,we prove this theorem Butterworth filter [271).we can build the global coordinate and for z-axis.Our reasoning is applicable for other axes.Let compute the rotation matrix as follows.After we obtain the Avi.denote the change of the relative velocity between gravity vector g,we derive its opposite value and normalize device Di and the body after the intra-body movements this vector as we then set this vector to represent occurred during time interval [ts,te].According to the rela- the global Z-axis as it is in the opposite direction of the tion between velocity and acceleration,we have Avi.r= gravitational acceleration.We set the vector zo to represent f(t)dt.As body parts move back and forth,the ampli- to be perpendicular to the horizontal plane.After computing tudes of the intra-body movement are usually small and do the cross product y =g x fc,we obtain a vector y that is not change significantly in any direction.Thus,Avi.should perpendicular to the plane determined by the two distinct but be smaller than a constant threshold C.In general situations, intersecting lines corresponding to g and fe.We normalize this C<0.4m/s due to the back-and-forth moving property. vector as yo=y.Since the vector yo is on the horizontal Thus,let At =te-ta,for E[f'i.(t)],i.e.,the expected plane,we set this vector to represent the global Y-axis.After value of the acceleration f(t)during [ts,te],we have that,by computing the cross product x =g x y,we obtain a vector x that is orthogonal to the plane determined by the two EU(. 1 fiz(t)dt △i< C (3) distinct but intersecting lines corresponding to g and y.We △t △t normalize this vector as x=.Since the vector xo is on If the time interval At =te-ts is sufficiently larger than the horizontal plane,and it is orthogonal to the global Y-axis the value of C,then E[fi,z(t)]≈0，e.g,△t=2 Os and and Z-axis,we set it to represent the global X-axis.After we C-0.4ms,△t>C,then Ef(d≤%ms2=2× obtain the vectors xo,yo,and zo in the local coordinate,which 10-2m/s2≈0. correspond to the global X,Y,and Z-axes,we can derive the rotation matrix CT.Fig.6 illustrates the above process. Theorem 1 implies that,during a sufficiently large time interval,e.g.,10~20s,the expected value of the inconsistent acceleration can be negligible,as the inconsistent accelerations B.Maintain Space Synchronization cancel each other out during the back and forth movements As the orientation of a device may be continuously chang- This explains our observations that the mean value of the ing,we need to track and update the rotation matrix C over inconsistent acceleration is in the order of 10-5m/s2. time.For example,if the rotation matrix at time t is given by Ct,then the rotation matrix C+6t at time t+6t can be IV.SPACE SYNCHRONIZATION MODEL computed as the product of two matrices A.Achieve Space Synchronization C1+8t=CiAt.t+8t (4) The objective of space synchronization for mobile devices where At.t+6t is the rotation matrix relating the local coordi- is to align their local coordinate to their global coordinate nate at time t to the one at time t+ot.To estimate the matrix We use the direction cosine representation to quantify the orientation difference between the local and global coordi- At.t+6t,we use gyroscope measurements.According to the nates of each device.In the direction cosine representation, small angle approximation [12],if we use w(t),w(t),and the orientation of the local coordinate relative to the global w(t)to represent the rotation rate of small rotations about its coordinate system is specified by a 3 x 3 rotation matrix C. x,y and z axes between time t and t+ot,then where each column is a unit vector along one axis in the local rt+6t coordinate specified in terms of the global coordinate axes. At.t+6t exp( (t)dt) (5)
2150 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 26, NO. 5, OCTOBER 2018 Fig. 5. fi(t) consists of consistent acceleration fc(t) and inconsistent acceleration f i(t), the projection of fc(t) on the x, y and z axes are fc(t) cos αi, fc(t) cos βi, and fc(t) cos γi, respectively. Theorem 1: For a fixed coordinate system relative to the earth coordinate system, for any inconsistent acceleration f i(t) caused by intra-body movements, the expected value of f i(t) is approximately equal to 0 within a sufficiently large time interval [ts, te]. That is, E[f’i(t)] = 1 te − ts te ts f i(t)dt ≈ 0. (2) Proof: Without loss of generality, we prove this theorem for x-axis. Our reasoning is applicable for other axes. Let Δvi,x denote the change of the relative velocity between device Di and the body after the intra-body movements occurred during time interval [ts, te]. According to the relation between velocity and acceleration, we have Δvi,x = te ts f i,x(t)dt. As body parts move back and forth, the amplitudes of the intra-body movement are usually small and do not change significantly in any direction. Thus, Δvi,x should be smaller than a constant threshold C. In general situations, C ≤ 0.4m/s due to the back-and-forth moving property. Thus, let Δt = te − ts, for E[f i,x(t)], i.e., the expected value of the acceleration f i,x(t) during [ts, te], we have E[f i,x(t)] = 1 Δt te ts f i,x(t)dt = Δvi,x Δt < C Δt . (3) If the time interval Δt = te − ts is sufficiently larger than the value of C, then E[f i,x(t)] ≈ 0, e.g., Δt = 20s and C = 0.4m/s, Δt C, then E[f i,x(t)] ≤ 0.4 20 m/s2 = 2 × 10−2m/s2 ≈ 0. Theorem 1 implies that, during a sufficiently large time interval, e.g., 10∼20s, the expected value of the inconsistent acceleration can be negligible, as the inconsistent accelerations cancel each other out during the back and forth movements. This explains our observations that the mean value of the inconsistent acceleration is in the order of 10−5m/s2. IV. SPACE SYNCHRONIZATION MODEL A. Achieve Space Synchronization The objective of space synchronization for mobile devices is to align their local coordinate to their global coordinate. We use the direction cosine representation to quantify the orientation difference between the local and global coordinates of each device. In the direction cosine representation, the orientation of the local coordinate relative to the global coordinate system is specified by a 3 × 3 rotation matrix C, where each column is a unit vector along one axis in the local coordinate specified in terms of the global coordinate axes. Fig. 6. Deriving the global coordinate. A vector quantity vl defined in the local coordinate system is equivalent to the vector vg = C · vl defined in the global coordinate. The inverse transformation is vl = CT · vg as the inverse of a rotation matrix is equal to its transpose. For the local coordinate of a device, assuming that we can extract a constant acceleration as a vector fc from the acceleration measurements and extract a constant gravitational acceleration as a vector g from the low pass filter (such as the Butterworth filter [27]), we can build the global coordinate and compute the rotation matrix as follows. After we obtain the gravity vector g, we derive its opposite value and normalize this vector as zo = −g g , we then set this vector to represent the global Z-axis as it is in the opposite direction of the gravitational acceleration. We set the vector zo to represent to be perpendicular to the horizontal plane. After computing the cross product y = g × fc, we obtain a vector y that is perpendicular to the plane determined by the two distinct but intersecting lines corresponding to g and fc. We normalize this vector as yo = y y . Since the vector yo is on the horizontal plane, we set this vector to represent the global Y -axis. After that, by computing the cross product x = g × y, we obtain a vector x that is orthogonal to the plane determined by the two distinct but intersecting lines corresponding to g and y. We normalize this vector as xo = x x . Since the vector xo is on the horizontal plane, and it is orthogonal to the global Y -axis and Z-axis, we set it to represent the global X-axis. After we obtain the vectors xo, yo, and zo in the local coordinate, which correspond to the global X, Y , and Z-axes, we can derive the rotation matrix CT . Fig. 6 illustrates the above process. B. Maintain Space Synchronization As the orientation of a device may be continuously changing, we need to track and update the rotation matrix C over time. For example, if the rotation matrix at time t is given by Ct, then the rotation matrix Ct+δt at time t + δt can be computed as the product of two matrices Ct+δt = CtAt,t+δt (4) where At,t+δt is the rotation matrix relating the local coordinate at time t to the one at time t+δt. To estimate the matrix At,t+δt, we use gyroscope measurements. According to the small angle approximation [12], if we use ωx(t), ωy(t), and ωz(t) to represent the rotation rate of small rotations about its x, y and z axes between time t and t + δt, then At,t+δt = exp( t+δt t Ω(t)dt) (5)