计算机科学与技术（参考文献）Meta-Activity Recognition - A Wearable Approach for Logic Cognition-based Activity Sensing

Meta-Activity Recognition:A Wearable Approach for Logic Cognition-based Activity Sensing Lei Xie,Xu Dong,Wei Wang,and Dawei Huang State Key Laboratory for Novel Software Technology,Nanjing University,China Email:Ixie@nju.edu.cn,dongxu@dislab.nju.edu.cn,ww@nju.edu.cn,huangdw @dislab.nju.edu.cn Abstract-Activity sensing has become a key technology for activity.Therefore,traditional activity sensing schemes [1] many ubiquitous applications,such as exercise monitoring and 3]are either based on the user-dependent recognition,which elder care.Most traditional approaches track the human motions requires to record the training data from the current user and perform activity recognition based on the waveform match- ing schemes in the raw data representation level.In regard to to improve the recognition accuracy,or relying on heavy- the complex activities with relatively large moving range,they training,which requires to collect a large quantity of training usually fail to accurately recognize these activities,due to the samples to build the templates.It is essential to propose inherent variations in human activities.In this paper,we propose a brand-new activity sensing scheme,such that the derived a wearable approach for logic cognition-based activity sensing recognition models can be scalable to any arbitrary human scheme in the logical representation level,by leveraging the meta- activity recognition.Our solution extracts the angle profiles from subjects in a user-independent and light-training approach. the raw inertial measurements,to depict the angle variation In this paper,we propose a wearable approach for logic cog- of limb movement in regard to the consistent body coordinate nition-based activity sensing,by leveraging the meta-activity system.It further extracts the meta-activity profiles to depict the recognition in the logical representation level.We mainly focus sequence of small-range activity units in the complex activity By leveraging the least edit distance-based matching scheme,our on the complex activities from human subjects,as shown in solution is able to accurately perform the activity sensing.Based Fig.1.Our approach is based on the observation that when on the logic cognition-based activity sensing,our solution achieves the human subject is performing an arbitrary activity,he/she lightweight-training recognition,which requires a small quantity is experiencing a very similar sequence of small-range-activity of training samples to build the templates,and user-independent units in the logical aspect,despite of the detailed differences in recognition,which requires no training from the specific user. The experiment results in real settings shows that our meta- the waveforms of the raw inertial measurements.We leverage activity recognition achieves an average accuracy of 92%for the notion meta-activity to denote the small-range-activity user-independent activity sensing units which compose a common activity of human subject. Given an arbitrary activity,our approach first extracts the I.INTRODUCTION angle profiles from the raw measurements to depict the angle Nowadays activity sensing has become a key technology for variation of limb movement in the consistent body coordinate many ubiquitous applications such as exercise monitoring and system.Then,it further extracts the meta-activity profiles to elder care.For example,in the daily exercise monitoring,it depict the sequence of small-range-activity units in the specific is essential to figure out what kinds of exercises the human activity.By leveraging the least edit distance-based matching subjects did everyday.The rising of the wearable devices has scheme,our solution is able to accurately perform the activity provided new opportunities for activity sensing during human sensing.Since a scalable recognition model is derived from motion.The wearable devices such as the smart watches are the meta-activity-based templates in the logical representation usually embedded with inertial sensors like the accelerometers, level,our solution achieves lightweight-training recognition, gyroscopes and magnetometers.They are able to continuously which requires a small quantity of training samples to build the track the human subject's movements and classify them into templates,and user-independent recognition,which requires the corresponding activities by matching the waveforms of in- no training from the specific user. ertial measurements against the templates.However,a number of common activities,e.g.,dumbbell curl and rope skipping, belong to the complex activities.The complex activity refers to an activity which has large range of movement and incurs rotations on multiple joints of the limbs,e.g.,the movement 1:Upright 2:Dumbbell 3:Dumbbell 4:Dumbbell 5:Dumbbell has angle change of more than 45 and involves more than arbell Rov ●urH Flies .随tern长速st Triceps Extensio 2 joints of the limbs.Moreover,it usually has two complex aspects:the widespread variations in activity details and the large movement range.Due to the user-specific characters like the heights,limb lengths and moving behaviors,there Rope Skipping 9:Pin 0:Ba exist obvious deviations in the raw inertial measurements from Swing different human subjects during the process of the complex Fig.1.Example Complex Activities

Meta-Activity Recognition: A Wearable Approach for Logic Cognition-based Activity Sensing Lei Xie, Xu Dong, Wei Wang, and Dawei Huang State Key Laboratory for Novel Software Technology, Nanjing University, China Email: lxie@nju.edu.cn, dongxu@dislab.nju.edu.cn, ww@nju.edu.cn, huangdw@dislab.nju.edu.cn Abstract—Activity sensing has become a key technology for many ubiquitous applications, such as exercise monitoring and elder care. Most traditional approaches track the human motions and perform activity recognition based on the waveform match￾ing schemes in the raw data representation level. In regard to the complex activities with relatively large moving range, they usually fail to accurately recognize these activities, due to the inherent variations in human activities. In this paper, we propose a wearable approach for logic cognition-based activity sensing scheme in the logical representation level, by leveraging the meta￾activity recognition. Our solution extracts the angle profiles from the raw inertial measurements, to depict the angle variation of limb movement in regard to the consistent body coordinate system. It further extracts the meta-activity profiles to depict the sequence of small-range activity units in the complex activity. By leveraging the least edit distance-based matching scheme, our solution is able to accurately perform the activity sensing. Based on the logic cognition-based activity sensing, our solution achieves lightweight-training recognition, which requires a small quantity of training samples to build the templates, and user-independent recognition, which requires no training from the specific user. The experiment results in real settings shows that our meta￾activity recognition achieves an average accuracy of 92% for user-independent activity sensing. I. INTRODUCTION Nowadays activity sensing has become a key technology for many ubiquitous applications such as exercise monitoring and elder care. For example, in the daily exercise monitoring, it is essential to figure out what kinds of exercises the human subjects did everyday. The rising of the wearable devices has provided new opportunities for activity sensing during human motion. The wearable devices such as the smart watches are usually embedded with inertial sensors like the accelerometers, gyroscopes and magnetometers. They are able to continuously track the human subject’s movements and classify them into the corresponding activities by matching the waveforms of in￾ertial measurements against the templates. However, a number of common activities, e.g., dumbbell curl and rope skipping, belong to the complex activities. The complex activity refers to an activity which has large range of movement and incurs rotations on multiple joints of the limbs, e.g., the movement has angle change of more than 45◦ and involves more than 2 joints of the limbs. Moreover, it usually has two complex aspects: the widespread variations in activity details and the large movement range. Due to the user-specific characters like the heights, limb lengths and moving behaviors, there exist obvious deviations in the raw inertial measurements from different human subjects during the process of the complex activity. Therefore, traditional activity sensing schemes [1]– [3] are either based on the user-dependent recognition, which requires to record the training data from the current user to improve the recognition accuracy, or relying on heavy￾training, which requires to collect a large quantity of training samples to build the templates. It is essential to propose a brand-new activity sensing scheme, such that the derived recognition models can be scalable to any arbitrary human subjects in a user-independent and light-training approach. In this paper, we propose a wearable approach for logic cog￾nition-based activity sensing, by leveraging the meta-activity recognition in the logical representation level. We mainly focus on the complex activities from human subjects, as shown in Fig. 1. Our approach is based on the observation that when the human subject is performing an arbitrary activity, he/she is experiencing a very similar sequence of small-range-activity units in the logical aspect, despite of the detailed differences in the waveforms of the raw inertial measurements. We leverage the notion meta-activity to denote the small-range-activity units which compose a common activity of human subject. Given an arbitrary activity, our approach first extracts the angle profiles from the raw measurements to depict the angle variation of limb movement in the consistent body coordinate system. Then, it further extracts the meta-activity profiles to depict the sequence of small-range-activity units in the specific activity. By leveraging the least edit distance-based matching scheme, our solution is able to accurately perform the activity sensing. Since a scalable recognition model is derived from the meta-activity-based templates in the logical representation level, our solution achieves lightweight-training recognition, which requires a small quantity of training samples to build the templates, and user-independent recognition, which requires no training from the specific user. 1:Upright Barbell Row 2:Dumbbell Curl 3:Dumbbell Flies 4:Dumbbell Lateral Raise 6: Rope Skipping 7: Butterfly 8: Cable Crossover 9: Ping-Pong Swing 5:Dumbbell Triceps Extension 10: Badminton Swing Fig. 1. Example Complex Activities

There are two key technical challenges in realizing the consistent approach,regardless of the exact orientation of the activity sensing scheme.The first challenge is to realize the human bodies.3)We have implemented a prototype system to activity sensing in a user-independent approach,such that the evaluate the real performance,the experiment results in real derived recognition model can be extended to recognize the settings shows that our meta-activity recognition achieves an activities of any arbitrary human subjects,regardless of the average accuracy of 92%for user-independent activity sensing. detailed differences and inherent deviations in the activities from different human subjects.To address this challenge,we II.PROBLEM FORMULATION propose to leverage the angle profiles,i.e.,the angles between In this paper,we investigate the wearable approach for the specified limb and the coordinate axes,to depict the limb activity sensing,i.e.,a wearable device is worn by the human movements.The angle profiles are able to capture the angle subject to continuously collect the inertial measurements of variation of the limb movements relative to the human body, human motion,then an activity sensing scheme is required to which tackle the deviation details caused by the user-specific accurately recognize the complex activities of limb movements characters like the height.Moreover,we propose the method of from human subjects.The complex activity refers to the kind "meta-activity recognition"to perform activity sensing in the of activity with a large range of movement,such as sit-ups logical representation level,based on the sequence of meta- and dumbbell lateral raise.Without loss of generality,we activity profiles,so as to tackle the variations in the long se- leverage the smart watch to sense the human motions,which quence of small-range activities.Specifically,according to the is embedded with inertial sensors including the accelerometer, inertial measurements collected from human motion,instead of gyroscope and magnetometer. performing the waveform-based matching like dynamic time In this paper,we aim to design an activity sensing scheme, warping,we decompose the complex activity into a sequence by considering the following metrics in system performance: of meta-activities,and use this sequence to recognize the 1)Accuracy:The expected accuracy for the activity sensing complex activity via the least edit distance-based matching. scheme to successfully match a specific activity to a correct The second challenge is to build a consistent scheme to activity should be greater than a specified threshold,e.g.,85%. depict the human motion according to the inertial measure- 2)Time-efficiency:The time delay of the activity recognition ments from the wearable devices.Since the human subjects process should be less than a specified threshold,e.g.,500ms. may perform the activities towards any arbitrary direction 3)User-independence:When performing activity sensing,no during the human motion,this causes the templates for activity training data should be required from the specified user. recognition to depend heavily on the actual direction the 4)Lightweight-training:The essential quantity of the training human body is facing,and further enhances the complexities samples to build the templates should be small enough.. in performing activity sensing due to the inconsistency.To III.MODELING THE HUMAN MOTION address this challenge,we depict all the inertial measurements of human motion in terms of a body coordinate system in A.Coordinate System Transformation a consistent approach.Specifically,according to the gravity In regard to activity sensing,as the raw inertial measure- direction and the magnetic direction extracted from the inertial ments are collected from the embedded inertial sensors in measurements.we transform the measurements from the watch the smart watch,they are measured by reference to the body coordinate system (WCS)to the global coordinate system frame of the smart watch.However,the watch coordinate (GCS).Then,by specifying two signal gestures,i.e.,extending system is continuously changing with the arm/wrist movement the arm to the front and dropping the arm downward,we can during the process of human motion,thus the measurements figure out the orientation of the human body in the global from the watch coordinate system cannot be used as a stable coordinate system according to the measurements in the signal reference for the specified activities.In fact,since the human gestures,thus we further transform the measurements to the subject may be performing the activity towards any arbitrary body coordinate system (BCS). direction,the movements should be depicted as the movement To the best of our knowledge,this paper presents the of arms or legs relative to the human body,regardless of the first study of using the method "meta-activity recognition" absolute moving direction of the limbs.Therefore,in order to for logical cognition-based activity sensing.Specifically,we perform activity sensing in a scalable approach,it is essential make three key contributions in this paper.1)Instead of to transform the measurement of limb movements from the performing waveform matching on the inertial measurements watch coordinate system to the body coordinate system. in the raw data level,we extract the angle profiles to depict 1)From Watch Coordinate System to Global Coordinate the angle variation of limb movements,and leverage the System:Fig.2(a)shows the three axes of the watch coordinate meta-activity profiles to depict the complex activities in the system.The X-axis refers to the direction of the lower arm logical representation level,such that the derived recognition when the watch is worn on the wrist,the Yio-axis refers to the model is scalable enough for the activity recognition on any direction of the strap of the watch,and the 2-axis refers to arbitrary human subjects.2)We build a coordinate system the direction which is perpendicular to the watch surface. transformation scheme to transform the inertial measurement According to the acceleration measurements from the ac- from the watch coordinate system to the body coordinate celerometer,we can extract a constant gravitational accel- system,such that the limb movement can be depicted in a eration as a vector g from the low pass filter (such as

There are two key technical challenges in realizing the activity sensing scheme. The first challenge is to realize the activity sensing in a user-independent approach, such that the derived recognition model can be extended to recognize the activities of any arbitrary human subjects, regardless of the detailed differences and inherent deviations in the activities from different human subjects. To address this challenge, we propose to leverage the angle profiles, i.e., the angles between the specified limb and the coordinate axes, to depict the limb movements. The angle profiles are able to capture the angle variation of the limb movements relative to the human body, which tackle the deviation details caused by the user-specific characters like the height. Moreover, we propose the method of “meta-activity recognition” to perform activity sensing in the logical representation level, based on the sequence of meta￾activity profiles, so as to tackle the variations in the long se￾quence of small-range activities. Specifically, according to the inertial measurements collected from human motion, instead of performing the waveform-based matching like dynamic time warping, we decompose the complex activity into a sequence of meta-activities, and use this sequence to recognize the complex activity via the least edit distance-based matching. The second challenge is to build a consistent scheme to depict the human motion according to the inertial measure￾ments from the wearable devices. Since the human subjects may perform the activities towards any arbitrary direction during the human motion, this causes the templates for activity recognition to depend heavily on the actual direction the human body is facing, and further enhances the complexities in performing activity sensing due to the inconsistency. To address this challenge, we depict all the inertial measurements of human motion in terms of a body coordinate system in a consistent approach. Specifically, according to the gravity direction and the magnetic direction extracted from the inertial measurements, we transform the measurements from the watch coordinate system (WCS) to the global coordinate system (GCS). Then, by specifying two signal gestures, i.e., extending the arm to the front and dropping the arm downward, we can figure out the orientation of the human body in the global coordinate system according to the measurements in the signal gestures, thus we further transform the measurements to the body coordinate system (BCS). To the best of our knowledge, this paper presents the first study of using the method “meta-activity recognition” for logical cognition-based activity sensing. Specifically, we make three key contributions in this paper. 1) Instead of performing waveform matching on the inertial measurements in the raw data level, we extract the angle profiles to depict the angle variation of limb movements, and leverage the meta-activity profiles to depict the complex activities in the logical representation level, such that the derived recognition model is scalable enough for the activity recognition on any arbitrary human subjects. 2) We build a coordinate system transformation scheme to transform the inertial measurement from the watch coordinate system to the body coordinate system, such that the limb movement can be depicted in a consistent approach, regardless of the exact orientation of the human bodies. 3) We have implemented a prototype system to evaluate the real performance, the experiment results in real settings shows that our meta-activity recognition achieves an average accuracy of 92% for user-independent activity sensing. II. PROBLEM FORMULATION In this paper, we investigate the wearable approach for activity sensing, i.e., a wearable device is worn by the human subject to continuously collect the inertial measurements of human motion, then an activity sensing scheme is required to accurately recognize the complex activities of limb movements from human subjects. The complex activity refers to the kind of activity with a large range of movement, such as sit-ups and dumbbell lateral raise. Without loss of generality, we leverage the smart watch to sense the human motions, which is embedded with inertial sensors including the accelerometer, gyroscope and magnetometer. In this paper, we aim to design an activity sensing scheme, by considering the following metrics in system performance: 1) Accuracy: The expected accuracy for the activity sensing scheme to successfully match a specific activity to a correct activity should be greater than a specified threshold, e.g., 85%. 2) Time-efficiency: The time delay of the activity recognition process should be less than a specified threshold, e.g., 500ms. 3) User-independence: When performing activity sensing, no training data should be required from the specified user. 4) Lightweight-training: The essential quantity of the training samples to build the templates should be small enough. . III. MODELING THE HUMAN MOTION A. Coordinate System Transformation In regard to activity sensing, as the raw inertial measure￾ments are collected from the embedded inertial sensors in the smart watch, they are measured by reference to the body frame of the smart watch. However, the watch coordinate system is continuously changing with the arm/wrist movement during the process of human motion, thus the measurements from the watch coordinate system cannot be used as a stable reference for the specified activities. In fact, since the human subject may be performing the activity towards any arbitrary direction, the movements should be depicted as the movement of arms or legs relative to the human body, regardless of the absolute moving direction of the limbs. Therefore, in order to perform activity sensing in a scalable approach, it is essential to transform the measurement of limb movements from the watch coordinate system to the body coordinate system. 1) From Watch Coordinate System to Global Coordinate System: Fig. 2(a) shows the three axes of the watch coordinate system. The Xw-axis refers to the direction of the lower arm when the watch is worn on the wrist, the Yw-axis refers to the direction of the strap of the watch, and the Zw-axis refers to the direction which is perpendicular to the watch surface. According to the acceleration measurements from the ac￾celerometer, we can extract a constant gravitational accel￾eration as a vector g from the low pass filter (such as

Za directions.Therefore,it is essential to build a body coordinate system (BCS)to depict the limb movements in a consistent approach by reference to the human body. In regard to the body coordinate system.we set the vector corresponding to the heading direction of the human subject to represent the 2o axis.For the horizontal plane which is Yw orthogonal to the Z axis,we set the vector which is parallel Yg to the physical plane of the body to represent the X axis,and (a)The relationships between WCS (b)The relationships between GCS set the vector which is perpendicular to the physical plane of and GCS and BCS the body to represent the Yo axis.Fig.2(b)shows the three axes Fig.2.The relationship between different coordinate systems (X,Yo,Zo)of BCS and the three axes (Xg:Ya Zg)of GCS the Butterworth filter [4])in the watch coordinate system. in regard to the physical plane of the human body,respectively. Moreover,according to the magnetic measurements from the Considering that the human subject can perform the activity magnetometer,we can extract the magnetic force as a vector with different orientations of the physical plane of the body. m in the watch coordinate system.Then,we can build a e.g.,standing on the floor or lying on the floor,in all situations, global coordinate system (GCS)based on the gravity direction we can transform any inertial measurement from the GCS to and magnetic direction in the watch coordinate system.The BCS by also using the direction cosine representation.The procedure is as follows:After we obtain the gravity vector orientation of the body coordinate system relative to the global g,we derive its opposite value and normalize this vector as coordinate system is specified by a 3 x 3 rotation matrix C' Zg=g we then set this vector zg to represent the global in which each column is a unit vector along one of the global Zgaxis as it is in the opposite direction of the gravitational coordinate axes specified in terms of the body coordinate axes. acceleration and it is perpendicular to the horizontal plane. A vector quantity vo defined in GCS is equivalent to the vector After computing the cross product y =g x m,we obtain v=C'.v defined in BCS.In this way,we can transform any a vector y that is perpendicular to the plane determined by inertial measurement from the GCS to the BCS.In Section IV. the two distinct but intersecting lines corresponding to g we will introduce the approach to compute the rotation matrix and m.We normalize this vector as ya= 前·Since the C',by leveraging two signal gestures. vector y is on the horizontal plane,we set this vector ya In regard to the activities where the physical plane of the to represent the global Ya-axis.After that,by computing the human body is continuously changing,e.g.,sit-ups,we can set cross product x=gxy,we obtain a vector x that is orthogonal the initial physical plane of the human body as the reference to the plane determined by the two distinct but intersecting body coordinate system.In this way,each of the following lines corresponding to g and y.We normalize this vector as inertial measurements are measured in terms of the reference xg=to represent the global Xg-axis..Fig.2(a)further body coordinate system shows the relationship between the three axes(x,y and z) of WCS and the three axes (xgy and zg)of GCS. B.Modeling the Human Motion with Meta-Activity To quantify the orientation difference between the watch Each complex activity,e.g.,dumbbell side raise and bent- coordinates and global coordinates,we use the direction cosine over dumbbell laterals,is performed with a large range of representation [5].In the direction cosine representation,the movement.So it can be decomposed into a series of small- orientation of the global coordinate relative to the watch range movements which are sequentially performed over time. coordinate system is specified by a 3 x 3 rotation matrix C, Therefore,we leverage the term meta-activities to denote these in which each column is a unit vector along one of the watch small-range movements.Each meta-activity is defined as a unit coordinate axes specified in terms of the global coordinate movement with logically the minimal granularity in regard to axes.A vector quantity ve defined in the watch coordinate the moving range.We can define the whole set of complex system is equivalent to the vector v=C.v defined in the activities as a set C.and the whole set of meta-activities as a global coordinate system.In this way,we are able to transform set M.Then,according to the above definition,each complex any inertial measurement v from WCS to the corresponding activity cC can be depicted as a series of meta-activities, inertial measurement vg in GCS.During the human motion,i.e.,ci=(mj,...,mj),where mjE M. the directions of g and m are continuously updated in WCS 1)Angle Profiles:In regard to the activity sensing,due to track the three axes of GCS,so as to further update the to the differences in human-specific characters such as the rotation matrix C in a real-time approach. height,arm length,and moving behavior,different human 2)From Global Coordinate System to Body Coordinate subjects may perform the same activity with different speeds System:During the human motion,the human subject may and amplitudes.This causes nonnegligible deviations among be facing any arbitrary direction in regard to the global the inertial measurements of the same activities in both time coordinate system.Hence,although we can derive the inertial domain and space domain.Therefore,the meta-activity should measurement of limb movements in GCS,these measurements be depicted in a scalable approach,such that the activity may not be consistent with each other even if they belong sensing scheme can be tolerant to the variances in the limb to the same activity,due to the differences in the facing movements.However,traditional inertial measurements such

Xw Yw Zw Zg Xg Yg (a) The relationships between WCS and GCS Zg(Zb) Yb Xb G Yg Xg ! ! Watch (b) The relationships between GCS and BCS Fig. 2. The relationship between different coordinate systems the Butterworth filter [4]) in the watch coordinate system. Moreover, according to the magnetic measurements from the magnetometer, we can extract the magnetic force as a vector m in the watch coordinate system. Then, we can build a global coordinate system (GCS) based on the gravity direction and magnetic direction in the watch coordinate system. The procedure is as follows: After we obtain the gravity vector g, we derive its opposite value and normalize this vector as zg = −g kgk , we then set this vector zg to represent the global Zg-axis as it is in the opposite direction of the gravitational acceleration and it is perpendicular to the horizontal plane. After computing the cross product y = g × m, we obtain a vector y that is perpendicular to the plane determined by the two distinct but intersecting lines corresponding to g and m. We normalize this vector as yg = y kyk . Since the vector yg is on the horizontal plane, we set this vector yg to represent the global Yg-axis. After that, by computing the cross product x = g×y, we obtain a vector x that is orthogonal to the plane determined by the two distinct but intersecting lines corresponding to g and y. We normalize this vector as xg = x kxk to represent the global Xg-axis.. Fig. 2(a) further shows the relationship between the three axes (xw, yw and zw) of WCS and the three axes (xg, yg and zg) of GCS. To quantify the orientation difference between the watch coordinates and global coordinates, we use the direction cosine representation [5]. In the direction cosine representation, the orientation of the global coordinate relative to the watch coordinate system is specified by a 3 × 3 rotation matrix C, in which each column is a unit vector along one of the watch coordinate axes specified in terms of the global coordinate axes. A vector quantity vw defined in the watch coordinate system is equivalent to the vector vg = C · vw defined in the global coordinate system. In this way, we are able to transform any inertial measurement vw from WCS to the corresponding inertial measurement vg in GCS. During the human motion, the directions of g and m are continuously updated in WCS to track the three axes of GCS, so as to further update the rotation matrix C in a real-time approach. 2) From Global Coordinate System to Body Coordinate System: During the human motion, the human subject may be facing any arbitrary direction in regard to the global coordinate system. Hence, although we can derive the inertial measurement of limb movements in GCS, these measurements may not be consistent with each other even if they belong to the same activity, due to the differences in the facing directions. Therefore, it is essential to build a body coordinate system (BCS) to depict the limb movements in a consistent approach by reference to the human body. In regard to the body coordinate system, we set the vector corresponding to the heading direction of the human subject to represent the Zb axis. For the horizontal plane which is orthogonal to the Zb axis, we set the vector which is parallel to the physical plane of the body to represent the Xb axis, and set the vector which is perpendicular to the physical plane of the body to represent the Yb axis. Fig. 2(b) shows the three axes (Xb, Yb, Zb) of BCS and the three axes (Xg, Yg, Zg) of GCS in regard to the physical plane of the human body, respectively. Considering that the human subject can perform the activity with different orientations of the physical plane of the body, e.g., standing on the floor or lying on the floor, in all situations, we can transform any inertial measurement from the GCS to BCS by also using the direction cosine representation. The orientation of the body coordinate system relative to the global coordinate system is specified by a 3 × 3 rotation matrix C’, in which each column is a unit vector along one of the global coordinate axes specified in terms of the body coordinate axes. A vector quantity vg defined in GCS is equivalent to the vector vb = C’·vg defined in BCS. In this way, we can transform any inertial measurement from the GCS to the BCS. In Section IV, we will introduce the approach to compute the rotation matrix C’, by leveraging two signal gestures. In regard to the activities where the physical plane of the human body is continuously changing, e.g., sit-ups, we can set the initial physical plane of the human body as the reference body coordinate system. In this way, each of the following inertial measurements are measured in terms of the reference body coordinate system. B. Modeling the Human Motion with Meta-Activity Each complex activity, e.g., dumbbell side raise and bent￾over dumbbell laterals, is performed with a large range of movement. So it can be decomposed into a series of small￾range movements which are sequentially performed over time. Therefore, we leverage the term meta-activities to denote these small-range movements. Each meta-activity is defined as a unit movement with logically the minimal granularity in regard to the moving range. We can define the whole set of complex activities as a set C, and the whole set of meta-activities as a set M. Then, according to the above definition, each complex activity ci ∈ C can be depicted as a series of meta-activities, i.e., ci = hmj1 , · · · , mjk i, where mj ∈ M. 1) Angle Profiles: In regard to the activity sensing, due to the differences in human-specific characters such as the height, arm length, and moving behavior, different human subjects may perform the same activity with different speeds and amplitudes. This causes nonnegligible deviations among the inertial measurements of the same activities in both time domain and space domain. Therefore, the meta-activity should be depicted in a scalable approach, such that the activity sensing scheme can be tolerant to the variances in the limb movements. However, traditional inertial measurements such

as the linear accelerations are very sensitive to the speeds obvious variances in the acceleration measurements,whereas and amplitudes of the limb movements,which fail to depict the variances in the angle profiles are relatively small.We the meta-activity in a scalable approach.Fortunately,it is further compute the DTW distances between each pair of found that,during the process of limb movements,the angle measurements from different human subjects,and obtain the variations between the limb and the body are much more average distance as the metric to quantify the corresponding stable than the traditional inertial measurements,which are variances.Fig.4(c)shows the DTW distances,respectively,for regardless of the human-specific characters such as the height the activity Dumbbell Curl and Sit-Up.It is found that for both and arm length.Therefore,in this paper,we propose to cases the angle profiles achieve much smaller distances than leverage the angle profiles,i.e.,the angles between the lower the acceleration measurement,which implies that the angle arm and the three axes in the body coordinate system,to depict profile is a more stable metric to depict the human motion. the meta-activities of the limb movements.Specifically,since Body the direction of X axis in the watch coordinate system is zCoordinate consistent with the lower arm direction,we can use the vector System X x to depict the lower arm direction in the body coordinate a●● system'.Fig.3(a)shows the vector x to depict the lower arm Lower Arm direction in the BCS.As shown in Fig.3(b),we respectively Direction denote the angle profiles,i.e.,the angles between the lower arm and the X,Y and Z axes in the BCS,as a,B and In order to compute the angle profiles,we take the angle a as an example,suppose the lower arm vector and the vector X axis of the X-axis are v(v=x)and u,respectively,in the (a)The lower arm direction (b)The angle profiles BCS.Then a can be computed according to the cosine value Fig.3.Derive the angle profiles in the body coordinate system as follows: 2)Meta-Activity Profiles:Ideally,in order to depict the V·u Uruz Vyuy Vzuz limb movements of human subject,the angle profiles of all cos a (1) skeletons in the body coordinate system are required to be √喔+哈+吃V+吃+喔 captured.Nevertheless,since the lower arm usually experi- For any specified value of cos a,there exist two solutions of ences a movement with fairly large range during the process a in the range between 0 and 360.Hence,we first compute of human motion,it is already representative to perform the the corresponding solution a within the range [0,180],we activity sensing based on the angle profiles of the lower arm. For the angle profiles a.B and y,according to the definition. then further determine the value of o as follows: they have the following relationship: ifvy≥0 a= 360°-a if vy<0. (2) (cosa)2+(cosB)2+(cosy)2=1 (5) Similarly,we can compute cos B and cosy accordingly,then Given any two values of the a,B and y,the other one can the values of B and y can be determined as follows: be computed according to Eg.(5).However,it still has two candidate solutions according to the corresponding cosine ifv2≥0 (3) value.Therefore,the arm-direction in the BCS can be uniquely 360°-ifv2<0. determined via the three parameters (o,B,) ifvz≥0 For any specified meta-activity,while it is being performed, 360°-ifvx<0. (4) the angle profiles (a,B,are continuously changing.The meta-activity should have the following properties for any of In this way,the angle profiles (o,B,in the BCS can be the parameters (o,B,):1)The variation range of any angle determined within the range[0°，36o]. profile should be less than a threshold 6,e.g.,30.2)The We further conducted empirical studies to validate the above variation trend of any angle profile should be monotonic,e.g., judgement.We invite four human subjects (a,b,c and d) monotonically increasing or decreasing.3)The time duration with different heights and genders to perform the specified of the meta-activity should be less than a threshold t,e.g.. complex activities,and respectively record the corresponding 500ms.The first property implies that the moving range of acceleration measurements and the angle profiles in regard to the meta-activity should be small enough,the second property each of the axes in the body coordinate system.We normalize implies that the moving direction of the meta-activity should all the measurements to the range [0,1]for fair comparison. be monotonic,the third property implies that the time duration Fig.4(a)and Fig.4(b)respectively shows the acceleration mea- of the meta-activity should be limited,even if the moving surements and angle profiles of the activity Dumbbell Curl. range is still small enough. It is found that,among different human subjects,there exist Therefore,for each dimension of the angle profiles (o,B,Y), As mentioned in Section III,the vector x in BCS can be computed we can uniformly divide the rotation range [0,360]into according to the direction cosine representation,it can be continuously multiple sectors,while the angle of each sector is no greater updated in a real time approach. than the threshold 6.In this way,we can use the specified

as the linear accelerations are very sensitive to the speeds and amplitudes of the limb movements, which fail to depict the meta-activity in a scalable approach. Fortunately, it is found that, during the process of limb movements, the angle variations between the limb and the body are much more stable than the traditional inertial measurements, which are regardless of the human-specific characters such as the height and arm length. Therefore, in this paper, we propose to leverage the angle profiles, i.e., the angles between the lower arm and the three axes in the body coordinate system, to depict the meta-activities of the limb movements. Specifically, since the direction of Xw axis in the watch coordinate system is consistent with the lower arm direction, we can use the vector xw to depict the lower arm direction in the body coordinate system1 . Fig. 3(a) shows the vector xw to depict the lower arm direction in the BCS. As shown in Fig. 3(b), we respectively denote the angle profiles, i.e., the angles between the lower arm and the X, Y and Z axes in the BCS, as α, β and γ. In order to compute the angle profiles, we take the angle α as an example, suppose the lower arm vector and the vector of the X-axis are v (v = xw ) and u, respectively, in the BCS. Then α can be computed according to the cosine value as follows: cos α = v · u |v||u| = q vxux + vyuy + vzuz v 2 x + v 2 y + v 2 z q u 2 x + u 2 y + u 2 z . (1) For any specified value of cos α, there exist two solutions of α in the range between 0 ◦ and 360◦ . Hence, we first compute the corresponding solution αb within the range [0◦ , 180◦ ], we then further determine the value of α as follows: α =  αb if vy ≥ 0 360◦ − αb if vy < 0. (2) Similarly, we can compute cos β and cos γ accordingly, then the values of β and γ can be determined as follows: β = ( βb if vz ≥ 0 360◦ − βb if vz < 0. (3) γ =  γb if vx ≥ 0 360◦ − γb if vx < 0. (4) In this way, the angle profiles hα, β, γi in the BCS can be determined within the range [0◦ , 360◦ ]. We further conducted empirical studies to validate the above judgement. We invite four human subjects (a, b, c and d) with different heights and genders to perform the specified complex activities, and respectively record the corresponding acceleration measurements and the angle profiles in regard to each of the axes in the body coordinate system. We normalize all the measurements to the range [0, 1] for fair comparison. Fig.4(a) and Fig.4(b) respectively shows the acceleration mea￾surements and angle profiles of the activity Dumbbell Curl. It is found that, among different human subjects, there exist 1As mentioned in Section III, the vector xw in BCS can be computed according to the direction cosine representation, it can be continuously updated in a real time approach. obvious variances in the acceleration measurements, whereas the variances in the angle profiles are relatively small. We further compute the DTW distances between each pair of measurements from different human subjects, and obtain the average distance as the metric to quantify the corresponding variances. Fig.4(c) shows the DTW distances, respectively, for the activity Dumbbell Curl and Sit-Up. It is found that for both cases the angle profiles achieve much smaller distances than the acceleration measurement, which implies that the angle profile is a more stable metric to depict the human motion. Z X Y Body Coordinate System Lower Arm Direction xw (a) The lower arm direction Z axis X axis Y axis xw α β γ (b) The angle profiles Fig. 3. Derive the angle profiles in the body coordinate system 2) Meta-Activity Profiles: Ideally, in order to depict the limb movements of human subject, the angle profiles of all skeletons in the body coordinate system are required to be captured. Nevertheless, since the lower arm usually experi￾ences a movement with fairly large range during the process of human motion, it is already representative to perform the activity sensing based on the angle profiles of the lower arm. For the angle profiles α, β and γ, according to the definition, they have the following relationship: (cos α) 2 + (cos β) 2 + (cos γ) 2 = 1. (5) Given any two values of the α, β and γ, the other one can be computed according to Eq.(5). However, it still has two candidate solutions according to the corresponding cosine value. Therefore, the arm-direction in the BCS can be uniquely determined via the three parameters hα, β, γi. For any specified meta-activity, while it is being performed, the angle profiles hα, β, γi are continuously changing. The meta-activity should have the following properties for any of the parameters hα, β, γi: 1) The variation range of any angle profile should be less than a threshold δ, e.g., 30◦ . 2) The variation trend of any angle profile should be monotonic, e.g., monotonically increasing or decreasing. 3) The time duration of the meta-activity should be less than a threshold t, e.g., 500ms. The first property implies that the moving range of the meta-activity should be small enough, the second property implies that the moving direction of the meta-activity should be monotonic, the third property implies that the time duration of the meta-activity should be limited, even if the moving range is still small enough. Therefore, for each dimension of the angle profiles hα, β, γi, we can uniformly divide the rotation range [0◦ , 360◦ ] into multiple sectors, while the angle of each sector is no greater than the threshold δ. In this way, we can use the specified

配 20 00 120 100 12 100 Dumbbell Curl Sit-Uo (a)Accleration measurements of Dumbbell Curl (b)Angle profiles of Dumbbell Curl (c)DTW distance in different complex activities Fig.4.The acceleration measurements and angle profiles in X,Y and Z-axes sector to depict the corresponding meta-activity in the spec- Data Acquisition and Preprocessing ified dimension.Moreover,considering the arm rotation can Raw Coordinate ngle Profile be anti-clock-wise or clock-wise,it is essential to further label Extraction Data each sector according to the rotation trend.Therefore,suppose Meta-Activity Segmentation and Classification the number of sectors is m,we can label each sector with a different ID from 0 to m-1 in an anti-clock-wise approach: Meta-Activity Meta-Activity for the ith sector,if the rotation direction is anti-clock-wise, then we label it with s;,otherwise,we label it with S,.Fig Complex Activity Recognition 5 shows an example of these meta-activity sectors,where Least Edit Distance-based Matching each sector has an angle of 30.These sectors are labeled from so to s11 if the rotation direction is anti-clock-wise, Fig.6.The system framework and they are labeled from So to S11 otherwise.In this way, BCS,by using the Direction Cosine method.To figure out we can use these discrete states rather than the continuous the orientation difference.i.e..the rotation matrix C between waveforms to represent the meta-activities.In comparison to BCS and GCS.before the human subject performs the complex the continuous waveform-based representation in the raw data activities,he/she is required to perform the following two level,this discrete state-based representation is based on the signal gestures in advance:1)Extend the arm to the front:let logic cognition of the human motion,which is more scalable the human subject extend his/her arm to the front of the body, to the inherent variances caused by user specific characters. the arm direction is consistent with the Y axis in the BCS:2) Drop the arm downward:let the human subject drop the arm s11 downward along his/her legs,the arm direction is opposite 81 10 s0:0°-30° to the Zo axis in the BCS.Fig.7(a)and Fig.7(b)shows an s1:30°-60 example of the two signal gestures,respectively.In this way, S2 S9s11:330°-360° by computing the corresponding vector of the arm direction S0:30°-0° in the GCS,we are able to figure out the rotation matrix C S1:60°-30° whatever the human subject is standing or lying on the floor. S11:360°-330° Glob Fig.5.The sectors to depict meta-activity in each dimension of angle profiles IV.SYSTEM DESIGN The overall system is composed of three major modules, as shown in Fig.6:Data Acquisition and Preprocessing takes the raw inertial measurements as input.It first performs the (a)Signal gesture 1:extend the arm (b)Signal gesture 2:drop the arm to the front downward coordinate transformation to transform the measurement from Fig.7.The signal gestures WCS to BCS.Then,it extracts the angle profiles and further 2)Angle Profiles Extraction:As aforementioned in Section split the series into separate complex activities.Meta-Activity III,take the smart watch as an example,we use the vector x, Segmentation and Classification segments a single complex i.e.,the direction of X axis in the WCS,to depict the arm activity into a series of meta-activities,and classifies the seg- direction in the BCS.Then,according to the arm vector x(t) mented meta-activities into corresponding categories.Complex at time t,we can extract the angle profiles (a(t),B(t),(t)) Activity Recognition performs activity recognition based on the over time in the BCS according to Eg.(4)-(7). sequences of meta-activities from the test complex activity,by 3)Segmentation:In practice,the human subject may con- leveraging the least edit distance-based matching scheme. tinuously perform a series of complex activities.Therefore,the recognition system should first split these series of complex A.Data Acquisition and Preprocessing activities into separate activities,then we can further identify 1)Coordinate Transformation:As mentioned in Section which activity pattern the current movement belongs to.As III,we can transform the measurement from the WCS to the human subject usually takes a short pause between two

0 20 40 60 80 100 120 140 0 0.5 1 a b c d 0 20 40 60 80 100 120 140 0 0.5 1 a b c d 0 20 40 60 80 100 120 140 0 0.5 1 a b c d (a) Accleration measurements of Dumbbell Curl 0 20 40 60 80 100 120 140 0 0.5 1 a b c d 0 20 40 60 80 100 120 140 0 0.5 1 a b c d 0 20 40 60 80 100 120 140 0 0.5 1 a b c d (b) Angle profiles of Dumbbell Curl Dumbbell Curl Sit-Up 0 1 2 3 4 Acceleration Angle Profiles (c) DTW distance in different complex activities Fig. 4. The acceleration measurements and angle profiles in X, Y and Z-axes sector to depict the corresponding meta-activity in the spec￾ified dimension. Moreover, considering the arm rotation can be anti-clock-wise or clock-wise, it is essential to further label each sector according to the rotation trend. Therefore, suppose the number of sectors is m, we can label each sector with a different ID from 0 to m − 1 in an anti-clock-wise approach: for the ith sector, if the rotation direction is anti-clock-wise, then we label it with sj , otherwise, we label it with Sj . Fig. 5 shows an example of these meta-activity sectors, where each sector has an angle of 30◦ . These sectors are labeled from s0 to s11 if the rotation direction is anti-clock-wise, and they are labeled from S0 to S11 otherwise. In this way, we can use these discrete states rather than the continuous waveforms to represent the meta-activities. In comparison to the continuous waveform-based representation in the raw data level, this discrete state-based representation is based on the logic cognition of the human motion, which is more scalable to the inherent variances caused by user specific characters. 0 1 2 3 4 5 6 7 8 9 10 11 s0: 0º-30º s1: 30º-60º … s11: 330º-360º S0: 30º-0º S1: 60º-30º … S11: 360º-330º S0 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 s s s s s s s s s s s s Fig. 5. The sectors to depict meta-activity in each dimension of angle profiles IV. SYSTEM DESIGN The overall system is composed of three major modules, as shown in Fig.6: Data Acquisition and Preprocessing takes the raw inertial measurements as input. It first performs the coordinate transformation to transform the measurement from WCS to BCS. Then, it extracts the angle profiles and further split the series into separate complex activities. Meta-Activity Segmentation and Classification segments a single complex activity into a series of meta-activities, and classifies the seg￾mented meta-activities into corresponding categories. Complex Activity Recognition performs activity recognition based on the sequences of meta-activities from the test complex activity, by leveraging the least edit distance-based matching scheme. A. Data Acquisition and Preprocessing 1) Coordinate Transformation: As mentioned in Section III, we can transform the measurement from the WCS to Data Acquisition and Preprocessing Coordinate Transformation Angle Profile Extraction Segmentation Meta-Activity Segmentation and Classification Meta-Activity Segmentation Meta-Activity Classification Complex Activity Recognition Least Edit Distance-based Matching Raw Sensor Data Recognition Result Fig. 6. The system framework BCS, by using the Direction Cosine method. To figure out the orientation difference, i.e., the rotation matrix C 0 between BCS and GCS, before the human subject performs the complex activities, he/she is required to perform the following two signal gestures in advance: 1) Extend the arm to the front: let the human subject extend his/her arm to the front of the body, the arm direction is consistent with the Yb axis in the BCS; 2) Drop the arm downward: let the human subject drop the arm downward along his/her legs, the arm direction is opposite to the Zg axis in the BCS. Fig.7(a) and Fig.7(b) shows an example of the two signal gestures, respectively. In this way, by computing the corresponding vector of the arm direction in the GCS, we are able to figure out the rotation matrix C 0 , whatever the human subject is standing or lying on the floor. ! Global Coordinate System Arm Direction (a) Signal gesture 1: extend the arm to the front Global Coordinate System Arm Direction (b) Signal gesture 2: drop the arm downward Fig. 7. The signal gestures 2) Angle Profiles Extraction: As aforementioned in Section III, take the smart watch as an example, we use the vector xw, i.e., the direction of Xw axis in the WCS, to depict the arm direction in the BCS. Then, according to the arm vector xw(t) at time t, we can extract the angle profiles hα(t), β(t), γ(t)i over time in the BCS according to Eq. (4)-(7). 3) Segmentation: In practice, the human subject may con￾tinuously perform a series of complex activities. Therefore, the recognition system should first split these series of complex activities into separate activities, then we can further identify which activity pattern the current movement belongs to. As the human subject usually takes a short pause between two