Tracking Human Motions in Photographing 29:11 Body Level Arm Level Wrist Level Ss: S S3: Rotating the phone Motionlessness Lifting up the arm So: Fine-tuning S2: Body movement: S4: e.g..walking. Laying down the arm jogging,etc. S7: Shotting Fig.8.The activity state machine. Here.If the variancesis below the corresponding threshold,that is,satisfy- ing Equation(3),then the window will be regarded as the start/end segment of an activity, as the pause window A,B,D shown in Figure 6.The left side of the window is regarded as the end point of the last activity,while the right side of the window is regarded as the start point of the next activity.Based on extensive observations,we set ea to 1,and set the size the sliding pause window wp to 0.5s, sa Ea. (3) -Step 2:We move the sliding pause window with the step lengthI=wp represents 0.5s.If two sliding windows have overlaps and are both regarded as start/end segments, then it may indicate that micro activities (or motionlessness)occur.At this time,we will use the corresponding gyroscope data in the pause window to calculate the variancess as described in Step 1.Similarly,if the variances satisfy Equation(4),the window will be regarded as the stat/end segment of an micro activity(or continuous pauses).Otherwise,the sliding window will keep moving forward until it finds the pause between micro activities, as shown in Figure 7.Based on extensive observations,we set e to 0.012: sg Eg (4) -Step 3:For a detected pause,it indicates the end of last activity and the start of next activity. Therefore,when we detect a pause,we start a new activity window to obtain the sensor data of the following activity.As shown in Figure 6,we mark the activity window with a rectangle Z.Obviously,an activity window represents a potential activity.We recognize the user's activity in the activity window and adopt the corresponding energy-saving strategy until the next pause occurs.Based on extensive observations,we set the size wa of the activity window to 2s,because 2s is usually enough to obtain the features of an activity, as shown in Figure 6.If the next pause without overlaps occurs in 2s,then we adaptively change wa to the duration between two consecutive pauses. 4.1.4 Recognition among Different Levels.When we get sensor data of an activity,we should first know which level the activity belongs to,then we can recognize the activity in the corre- sponding level.To describe the relationship between the levels and activities,we introduce an activity state machine,as shown in Figure 8.In the state machine,each activity is represented as a state,and the states are classified into three levels accordingly.Considering the behavior features during photographing,the user cannot transfer from one state to any other state arbitrarily.For ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
Tracking Human Motions in Photographing 29:11 Fig. 8. The activity state machine. Here, a = 1 n n i=1 ai . If the variance s2 a is below the corresponding threshold, that is, satisfying Equation (3), then the window will be regarded as the start/end segment of an activity, as the pause window A, B, D shown in Figure 6. The left side of the window is regarded as the end point of the last activity, while the right side of the window is regarded as the start point of the next activity. Based on extensive observations, we set ϵa to 1, and set the size the sliding pause window wp to 0.5s, s2 a < ϵa . (3) —Step 2: We move the sliding pause window with the step length l = wp 2 , wp represents 0.5s. If two sliding windows have overlaps and are both regarded as start/end segments, then it may indicate that micro activities (or motionlessness) occur. At this time, we will use the corresponding gyroscope data in the pause window to calculate the variances s2 д, as described in Step 1. Similarly, if the variances satisfy Equation (4), the window will be regarded as the stat/end segment of an micro activity (or continuous pauses). Otherwise, the sliding window will keep moving forward until it finds the pause between micro activities, as shown in Figure 7. Based on extensive observations, we set ϵд to 0.012: s2 д < ϵд . (4) —Step 3: For a detected pause, it indicates the end of last activity and the start of next activity. Therefore, when we detect a pause, we start a new activity window to obtain the sensor data of the following activity. As shown in Figure 6, we mark the activity window with a rectangle Z. Obviously, an activity window represents a potential activity. We recognize the user’s activity in the activity window and adopt the corresponding energy-saving strategy until the next pause occurs. Based on extensive observations, we set the size wa of the activity window to 2s, because 2s is usually enough to obtain the features of an activity, as shown in Figure 6. If the next pause without overlaps occurs in 2s, then we adaptively change wa to the duration between two consecutive pauses. 4.1.4 Recognition among Different Levels. When we get sensor data of an activity, we should first know which level the activity belongs to, then we can recognize the activity in the corresponding level. To describe the relationship between the levels and activities, we introduce an activity state machine, as shown in Figure 8. In the state machine, each activity is represented as a state, and the states are classified into three levels accordingly. Considering the behavior features during photographing, the user cannot transfer from one state to any other state arbitrarily. For ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
29:12 Y.Yin et al. Gravity (a)Walking (b)Lifting up the phone (c)Rotating the phone Fig.9.Analysis of activity features. example,if the user has laid down the arm,he/her will probably not lay down the arm again.For body level,the user keeps still or moving (e.g.,walking,jogging),it means he/she stays in the same state.Therefore,there is no self-cycle activities in body level.For arm level and wrist level, the short pause between two activities is used for segmenting two consecutive activities,as de- scribed in Section 4.1.3.The transfer relationship of states is shown in Figure 8.By maintaining the activity state machine,we can determine the activity state progressively and reduce the error of activity recognition.For example,we may wrongly recognize the activity "Rotating the phone" as "Fine-tuning".Nevertheless,the two states have similar energy-saving strategies,there will not be a sudden decrease of user experience.Usually,we will not recognize "Rotating the phone"as "Walking,"which turns off the screen for energy saving and leads to a bad user experience.There- fore,the activity state machine can control the recognition error in a tolerable range and guarantee a good user experience. For different activities,the amplitude and direction of linear-acc changes differently,as shown in Figure 9.For body level,the linear-acc changes periodically,as shown in Figure 9(a).For arm level,the user usually lifts his/her arm one time,as shown in Figure 9(b).While considering dif- ferent holding gestures of the phone,the direction variation of linear-acc cannot be mapped to the activities of lifting up or laying down the arm directly.Thus,we introduce the gravity data,which reflects the direction of gravity to assist for activity recognition.In regard to the wrist level,the user tends to rotate the phone to adjust the camera view,as shown in Figure 9(c).Therefore,we combine the linear-acc and gyroscope data to recognize the activities. With the activity segment from Section 4.1.3,we show how to classify an activity into one of the three levels in Figure 10.We utilize the variance of linear-acc,gyroscope data and the periodicity of sensor data to distinguish body level,arm level and wrist level.For body level,if the user stays motionless,the value of linear-acc and gyroscope data is close to zero.The rectangle B in Figure 6 and the rectangle C1 in Figure 7 represent the linear-acc and gyroscope data of pause (or motionlessness),respectively.If the sensor data in a relative long time (e.g.,larger than 15s) satisfies Equations(3)and(4),then the activity will be recognized as motionlessness,that is,be classified into body level In regard to body movement,it does not satisfy Equations(3)and(4). However,we can utilize the periodicity to distinguish body movement from other activities,as the sensor data of "Walking"shown in Figure 6.Here,we simply use the period tp of the activity for body movement detection.We usepp to represent the period of last k activities in body movement,respectively.If tp satisfy Equation(5),then the activity will be classified into body movement,that is,body level.Based on extensive experiments,we set ep to 0.2: p一走1p <Ep (5) ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:12 Y. Yin et al. Fig. 9. Analysis of activity features. example, if the user has laid down the arm, he/her will probably not lay down the arm again. For body level, the user keeps still or moving (e.g., walking, jogging), it means he/she stays in the same state. Therefore, there is no self-cycle activities in body level. For arm level and wrist level, the short pause between two activities is used for segmenting two consecutive activities, as described in Section 4.1.3. The transfer relationship of states is shown in Figure 8. By maintaining the activity state machine, we can determine the activity state progressively and reduce the error of activity recognition. For example, we may wrongly recognize the activity “Rotating the phone” as “Fine-tuning”. Nevertheless, the two states have similar energy-saving strategies, there will not be a sudden decrease of user experience. Usually, we will not recognize “Rotating the phone” as “Walking,” which turns off the screen for energy saving and leads to a bad user experience. Therefore, the activity state machine can control the recognition error in a tolerable range and guarantee a good user experience. For different activities, the amplitude and direction of linear-acc changes differently, as shown in Figure 9. For body level, the linear-acc changes periodically, as shown in Figure 9(a). For arm level, the user usually lifts his/her arm one time, as shown in Figure 9(b). While considering different holding gestures of the phone, the direction variation of linear-acc cannot be mapped to the activities of lifting up or laying down the arm directly. Thus, we introduce the gravity data, which reflects the direction of gravity to assist for activity recognition. In regard to the wrist level, the user tends to rotate the phone to adjust the camera view, as shown in Figure 9(c). Therefore, we combine the linear-acc and gyroscope data to recognize the activities. With the activity segment from Section 4.1.3, we show how to classify an activity into one of the three levels in Figure 10. We utilize the variance of linear-acc, gyroscope data and the periodicity of sensor data to distinguish body level, arm level and wrist level. For body level, if the user stays motionless, the value of linear-acc and gyroscope data is close to zero. The rectangle B in Figure 6 and the rectangle C1 in Figure 7 represent the linear-acc and gyroscope data of pause (or motionlessness), respectively. If the sensor data in a relative long time (e.g., larger than 15s) satisfies Equations (3) and (4), then the activity will be recognized as motionlessness, that is, be classified into body level. In regard to body movement, it does not satisfy Equations (3) and (4). However, we can utilize the periodicity to distinguish body movement from other activities, as the sensor data of “Walking” shown in Figure 6. Here, we simply use the period tp of the activity for body movement detection. We use tp1 , tp2 , ... , tpk−1 , tpk to represent the period of last k activities in body movement, respectively. If tp satisfy Equation (5), then the activity will be classified into body movement, that is, body level. Based on extensive experiments, we set ϵp to 0.2: tp − 1 k k j=1 tpk 1 k k j=1 tpk < ϵp . (5) ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017.
Tracking Human Motions in Photographing 29:13 Sensor data Linear-acc, gyroscope data Which level it belongs to Variance,periodicity Negligible variance, Large variance in linear-acc and Low variance in linear-acc,moderate or periodical change gyroscope data,no periodicity variance ingyroscope data Activity Recognition in Activity Recognition in Activity Recognition in Body Level Arm Level Wrist Level (motionlessness,body movement) (arm up,arm down) (rotating,tuning,shooting) Fig.10.Activity recognition among different levels. For arm level,the user lifts up the arm or lays down the arm.There are no periodicity of consecutive activities,for example,the user cannot continuously lift up the arm.Therefore,if the activity does not satisfy Equations(3),(4),and(5),it will be classified into arm level.For wrist level,the user rotates the phone,makes fine-tuning,shoots a photo.The micro movements of the phone satisfy Equation(3)while not satisfying Equation(4).Without loss of generality,we assume the initial activity of the user belongs to body level.That is to say,before the phone runs our scheme "SenSave,"the user is motionless or keeping moving (e.g.,walking,jogging).The process of activity recognition among three levels is shown in Algorithm 1.When we know which level the current activity belongs to,then we can do activity recognition in the corresponding level,as described in Section 4.1.5. ALGORITHM 1:Recognition among Three Levels Input:Activity segment Ai. Output:Classified level Li. Calculate the variance s2 of linear-acc of Ai with Equation(2). Calculate the variance s of gyroscope data of Ai like Equation(2). if s2 satisfies Equation(3)and s2 satisfies Equation (4)then LClassify Ai into Body Level and recognize the activity as motionlessness. if sa dissatisfies Equation(3)and s dissatisfies Equation(4)then if Period tp of Ai satisfies Equation(5)then LClassify Ai into Body level and recognize it in body movement. else L Classify Ai into Arm level. if sa satisfies Equation (3)and sa dissatisfies Equation(4)then LClassify A;into Wrist level. Ruturn the classified level Li. 4.1.5 Activity Recognition in Each Level.According to Section 4.1.4 and Figure 9,the activities in different levels usually have different features in sensor data.For body-level activities,they usually have large variation in linear-acc and gyroscope data.For arm level activities,we need to combine linear-acc and gravity data to distinguish the lifting-up activity and laying-down activity. For wrist level activities,we combine linear-acc and gyroscope data to distinguish the three micro ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
Tracking Human Motions in Photographing 29:13 Fig. 10. Activity recognition among different levels. For arm level, the user lifts up the arm or lays down the arm. There are no periodicity of consecutive activities, for example, the user cannot continuously lift up the arm. Therefore, if the activity does not satisfy Equations (3), (4), and (5), it will be classified into arm level. For wrist level, the user rotates the phone, makes fine-tuning, shoots a photo. The micro movements of the phone satisfy Equation (3) while not satisfying Equation (4). Without loss of generality, we assume the initial activity of the user belongs to body level. That is to say, before the phone runs our scheme “SenSave,” the user is motionless or keeping moving (e.g., walking, jogging). The process of activity recognition among three levels is shown in Algorithm 1. When we know which level the current activity belongs to, then we can do activity recognition in the corresponding level, as described in Section 4.1.5. ALGORITHM 1: Recognition among Three Levels Input: Activity segment Ai . Output: Classified level Li . Calculate the variance s2 a of linear-acc of Ai with Equation (2). Calculate the variance s2 д of gyroscope data of Ai like Equation (2). if s2 a satisfies Equation (3) and s2 д satisfies Equation (4) then Classify Ai into Body Level and recognize the activity as motionlessness. if s2 a dissatisfies Equation (3) and s2 д dissatisfies Equation (4) then if Period tp of Ai satisfies Equation (5) then Classify Ai into Body level and recognize it in body movement. else Classify Ai into Arm level. if s2 a satisfies Equation (3) and s2 д dissatisfies Equation (4) then Classify Ai into Wrist level. Ruturn the classified level Li . 4.1.5 Activity Recognition in Each Level. According to Section 4.1.4 and Figure 9, the activities in different levels usually have different features in sensor data. For body-level activities, they usually have large variation in linear-acc and gyroscope data. For arm level activities, we need to combine linear-acc and gravity data to distinguish the lifting-up activity and laying-down activity. For wrist level activities, we combine linear-acc and gyroscope data to distinguish the three micro ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017