29:6 Y.Yin et al. Table 1.Energy Consumption for Different Operations Operation Turn on Turn off Turn on Turn off Phone camera camera screen screen Preview (uAh) (uAh) (uAh) (uAh) (uAh/s) Nexus 324.61 314.39 301.36 275.89 169.18 S5 456.57 380.40 311.86 267.83 198.42 Note4 763.02 714.58 439.64 255.22 272.01 on the screen.Each user participating in our experiments unlocks the screen after he/she turns on the screen.Thus,unlocking the screen is included in the "Turn ON/OFF Screen."In regard to "Turn ON/OFF Camera and Screen,"it means the user turns off both camera and screen after photographing,and then turns on screen and camera for retaking photos. In Figure 2(b),we show the probability that the user "Turn ON/OFF Camera,""Turn ON/OFF Screen,"and "Turn ON/OFF Camera and Screen"during photographing.On average,the users adopt one of the above energy-saving schemes with 65.8%probability,instead of doing nothing. Here,"Do Nothing"means that the user keeps the camera working in preview mode during pho- tographing.It usually occurs in the case when taking multiple photos in a short time."Do Nothing" leads to large energy consumption,as described in Section 2.2.1.Among the three energy-saving schemes,the users prefer to turn off both camera and screen for further energy consumption. On average,the probability of "Turn ON/OFF Camera and Screen"achieves 32.9%.However,fre- quently user interaction(e.g.,touch the screen,press the button)degrades the user experience and it can also increase the energy consumption(Dietrich and Chakraborty 2013). In Table 1,by using Samsung Galaxy Nexus,Samsung Galaxy S5,and Samsung Galaxy Note4, we show the energy consumption of turning ON/OFF camera or screen one time.In regard to the measurement,take "ON/OFF Screen"as an example,we first keep the screen on and then turn off the screen.The "turning off"operation incurs the power saltation,then we record the energy consumption E1 in the following 20s from the moment of saltation.After that,we record the en- ergy consumption E2 while keeping the screen off for 20s.Then,we use E1-E2 to represent the energy consumption of "Turning OFF Screen."Similarly,we can measure the energy consumption of other operations.We invite 10 volunteers to participate in the experiment.Each volunteer re- peats the operations 50 times,and we average the experiment results.After that,we compare the energy consumption of turning ON/OFF camera or screen with that of keeping camera on.Take Samsung Galaxy Nexus for an example,Equation(1)shows that "Turn ON/OFF Camera,""Turn ON/OFF Screen,"and "Turn ON/OFF Camera and Screen"one time is equivalent to keeping cam- era in preview mode for 3.78s,3.41s,and 7.19s,respectively.Similarly,"Turn ON/OFF Camera and Screen"one time in Samsung S5 and Samsung Note4 is equivalent to keeping camera in preview mode for 7.14s and 7.99s,respectively. 324.61uAh+314.39uAh ON/OFF Camera: =3.78s 169.18Ah/s 301.36uAh+275.89uAh ON/OFF Screen: =3.41s (1) 169.18uAh/s ON/OFF Camera and Screen:3.78s +3.41s 7.19s It indicates that frequently turning ON/OFF the camera or screen manually is indeed inconvenient and rather energy consuming.If we can decrease the energy consumption while not requiring the ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:6 Y. Yin et al. Table 1. Energy Consumption for Different Operations Phone Operation Turn on camera (uAh) Turn off camera (uAh) Turn on screen (uAh) Turn off screen (uAh) Preview (uAh/s) Nexus 324.61 314.39 301.36 275.89 169.18 S5 456.57 380.40 311.86 267.83 198.42 Note4 763.02 714.58 439.64 255.22 272.01 on the screen. Each user participating in our experiments unlocks the screen after he/she turns on the screen. Thus, unlocking the screen is included in the “Turn ON/OFF Screen.” In regard to “Turn ON/OFF Camera and Screen,” it means the user turns off both camera and screen after photographing, and then turns on screen and camera for retaking photos. In Figure 2(b), we show the probability that the user “Turn ON/OFF Camera,” “Turn ON/OFF Screen,” and “Turn ON/OFF Camera and Screen” during photographing. On average, the users adopt one of the above energy-saving schemes with 65.8% probability, instead of doing nothing. Here, “Do Nothing” means that the user keeps the camera working in preview mode during photographing. It usually occurs in the case when taking multiple photos in a short time. “Do Nothing” leads to large energy consumption, as described in Section 2.2.1. Among the three energy-saving schemes, the users prefer to turn off both camera and screen for further energy consumption. On average, the probability of “Turn ON/OFF Camera and Screen” achieves 32.9%. However, frequently user interaction (e.g., touch the screen, press the button) degrades the user experience and it can also increase the energy consumption (Dietrich and Chakraborty 2013). In Table 1, by using Samsung Galaxy Nexus, Samsung Galaxy S5, and Samsung Galaxy Note4, we show the energy consumption of turning ON/OFF camera or screen one time. In regard to the measurement, take “ON/OFF Screen” as an example, we first keep the screen on and then turn off the screen. The “turning off” operation incurs the power saltation, then we record the energy consumption E1 in the following 20s from the moment of saltation. After that, we record the energy consumption E2 while keeping the screen off for 20s. Then, we use E1 − E2 to represent the energy consumption of “Turning OFF Screen.” Similarly, we can measure the energy consumption of other operations. We invite 10 volunteers to participate in the experiment. Each volunteer repeats the operations 50 times, and we average the experiment results. After that, we compare the energy consumption of turning ON/OFF camera or screen with that of keeping camera on. Take Samsung Galaxy Nexus for an example, Equation (1) shows that “Turn ON/OFF Camera,” “Turn ON/OFF Screen,”and “Turn ON/OFF Camera and Screen” one time is equivalent to keeping camera in preview mode for 3.78s, 3.41s, and 7.19s, respectively. Similarly, “Turn ON/OFF Camera and Screen” one time in Samsung S5 and Samsung Note4 is equivalent to keeping camera in preview mode for 7.14s and 7.99s, respectively. ON/OFF Camera: 324.61uAh + 314.39uAh 169.18uAh/s = 3.78s ON/OFF Screen: 301.36uAh + 275.89uAh 169.18uAh/s = 3.41s ON/OFF Camera and Screen: 3.78s + 3.41s = 7.19s (1) It indicates that frequently turning ON/OFF the camera or screen manually is indeed inconvenient and rather energy consuming. If we can decrease the energy consumption while not requiring the ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
Tracking Human Motions in Photographing 29:7 Data Collection Energy Saving (Low power-consuming sensors) (Applying different strategies based on recognized activities/states) Body-level Arm-level Wrist-level Accelerometer energy-saving strategies energy-saving strategies cnergy-saving strategies 热 Gyroscope Activity state Activity state Activity state in body-level in arm-level in wrist-level Sensor data Body level Arm level Wrist level Activity Segmentation Motionlessness Lifting up the arm Rotating the phone】 (Based on pause detection) Fine-tuning Data Body movement Laying down the arm Detecting the start/end of an Shooting activity based on the sensor data with low variances. Activity Recognition (Classifying an activity into one of the three levels,and recognizing it in the corresponding level) Fig.3.System architecture. user to frequently turn ON/OFF the camera or screen,then we can extend the battery life and provide a better user experience. 2.2.3 Energy Consumption of Using Built-in Sensors.It is possible to use low power-consuming built-in sensors for activity sensing and reduce the energy consumption during photographing. Figure 2(c)shows the power consumption of running the built-in sensor in Samsung Nexus,Sam- sung S5,and Samsung Note4.All these sensors work with the same sampling rate,that is,100Hz, which is the maximal sampling rate of sensors in Samsung Nexus.When we measure the power consumption of a sensor,we turn off other sensors.We first record the power consumption with- out running any sensors as Pw1.Then,we record the power consumption by running an sensor as Pw2.For Pwi and Pw2,the phone has the same screen display.Besides,we average the power consumption in 20s when the phone is in a steady state,that is,eliminating the power saltation of starting the sensor.After that,we use Pw2-PwI to obtain the power consumption of the sensor. In Figure 2(c),the sensors include accelerometer,linear accelerometer,gravity sensor,gyroscope, and camera.Based on the definition of SensorEvent in Android APIs(Google Inc.2016a,2016c),lin- ear acceleration(linear-acc for short)and gravity data are generated from accelerometer,that is,we use a low-pass filter to isolate the force of gravity,then we use the acceleration to minus the grav- ity data in each axis to obtain the linear acceleration.Therefore,the energy consumption of linear accelerometer is a little larger than that of gravity sensor.Both of them are larger than that of ac- celerometer.In regard to gyroscope,it usually has larger energy consumption than accelerometer (Park et al.2011).However,when we turn on the camera and keep the camera working in preview mode,the increase of power consumption is much larger than that of other sensors.Therefore,we can utilize the low power-consuming built-in sensors (i.e.,accelerometer and gyroscope)of the phone to detect the user's activities and reduce the energy consumption during photographing.A simple example could be decreasing the screen brightness or turning off the screen,when we find that the user is not taking photos. 3 SYSTEM OVERVIEW Based on the above observations,we consider using the built-in sensors of smart phones to detect the user's activities and reduce the unnecessary energy cost in photographing.The architecture of our system is shown in Figure 3.First,we collect the data from low power-consuming built-in sensors,that is,accelerometer and gyroscope,as shown in the Data Collection module.Second,we ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
Tracking Human Motions in Photographing 29:7 Fig. 3. System architecture. user to frequently turn ON/OFF the camera or screen, then we can extend the battery life and provide a better user experience. 2.2.3 Energy Consumption of Using Built-in Sensors. It is possible to use low power-consuming built-in sensors for activity sensing and reduce the energy consumption during photographing. Figure 2(c) shows the power consumption of running the built-in sensor in Samsung Nexus, Samsung S5, and Samsung Note4. All these sensors work with the same sampling rate, that is, 100Hz, which is the maximal sampling rate of sensors in Samsung Nexus. When we measure the power consumption of a sensor, we turn off other sensors. We first record the power consumption without running any sensors as Pw1. Then, we record the power consumption by running an sensor as Pw2. For Pw1 and Pw2, the phone has the same screen display. Besides, we average the power consumption in 20s when the phone is in a steady state, that is, eliminating the power saltation of starting the sensor. After that, we use Pw2 − Pw1 to obtain the power consumption of the sensor. In Figure 2(c), the sensors include accelerometer, linear accelerometer, gravity sensor, gyroscope, and camera. Based on the definition of SensorEvent in Android APIs (Google Inc. 2016a, 2016c), linear acceleration (linear-acc for short) and gravity data are generated from accelerometer, that is, we use a low-pass filter to isolate the force of gravity, then we use the acceleration to minus the gravity data in each axis to obtain the linear acceleration. Therefore, the energy consumption of linear accelerometer is a little larger than that of gravity sensor. Both of them are larger than that of accelerometer. In regard to gyroscope, it usually has larger energy consumption than accelerometer (Park et al. 2011). However, when we turn on the camera and keep the camera working in preview mode, the increase of power consumption is much larger than that of other sensors. Therefore, we can utilize the low power-consuming built-in sensors (i.e., accelerometer and gyroscope) of the phone to detect the user’s activities and reduce the energy consumption during photographing. A simple example could be decreasing the screen brightness or turning off the screen, when we find that the user is not taking photos. 3 SYSTEM OVERVIEW Based on the above observations, we consider using the built-in sensors of smart phones to detect the user’s activities and reduce the unnecessary energy cost in photographing. The architecture of our system is shown in Figure 3. First, we collect the data from low power-consuming built-in sensors, that is, accelerometer and gyroscope, as shown in the Data Collection module. Second, we ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
29:8 Y.Yin et al. extract the activity segment from the sensor data,as shown in the Activity Segmentation module. Third,we classify an activity into one of the three levels:body level,arm level,wrist level.Then, we recognize the activity in the corresponding level,as shown in the Activity Recognition module. Fourth,we adopt an appropriate energy-saving strategy based on the recognized activity,as shown in the Energy Saving module.In the following paragraphs,we will describe how to do activity sensing and adopt energy-saving strategies. 3.1 Activity Sensing Based on Section 2.1,the user's activities are categorized into three levels:body level,arm level, wrist level.In each level,there may be more than one activity.Besides,the user can transfer from one activity to another.To reduce the error of activity recognition,we maintain an activity state machine to describe the transfer relationship of the activities and determine the user's activity progressively.When we get the sensor data of an activity,we first utilize the variance and period- icity of sensor data to classify the activity into one of the three levels.Then,we combine the data from different sensors to recognize the specific activity in each level. -Body level:It includes motionlessness and body movement.Motionlessness can be recog- nized with its low variance of linear acceleration (linear-acc for short)and gyroscope data. In regard to body movement,which can be walking,jogging,and so on,we utilize the peri- odicity of an activity to infer whether the current activity belongs to body movement.That is to say,we do not distinguish walking,jogging,and so on,we aim to recognize an activity as body movement or not. -Arm level:It includes lifting up the arm and laying down the arm.We utilize the rela- tionship between the gravity data and linear acceleration to distinguish the two activities. Besides,we use a voting mechanism to improve the recognition accuracy. -Wrist level:It includes rotating the phone,making fine-tuning,and shooting a photo.We make use of a linear SVM model to distinguish them with the following features:variance, mean,maximum value and minimum value of linear acceleration and gyroscope data in each axis. 3.2 Energy Saving Considering the specific feature of each activity,we propose an adaptive energy-saving scheme called "SenSave"for photographing.Our solution SenSave does not need user interaction(e.g.,turn ON/OFF the camera or screen manually)for energy-saving,it utilizes the low power-consuming sensors to detect the user's activity and adopts a corresponding energy-saving strategy adaptively. For example,when you walk,jog or stay motionless,we can keep the screen off.When you lift your arm up,it is better to turn on the screen and adjust the screen brightness based on the ambient light.When you make fine-tuning to observe the camera view before shooting a photo,it is better to make the camera work in preview mode.In this way,we can make context-aware energy-saving strategies for camera phones. 4 SYSTEM DESIGN In this section,we will show how to use the built-in sensors(i.e.,accelerometer and gyroscope)for activity sensing and design an efficient energy-saving scheme for photographing in smart camera phones. ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:8 Y. Yin et al. extract the activity segment from the sensor data, as shown in the Activity Segmentation module. Third, we classify an activity into one of the three levels: body level, arm level, wrist level. Then, we recognize the activity in the corresponding level, as shown in the Activity Recognition module. Fourth, we adopt an appropriate energy-saving strategy based on the recognized activity, as shown in the Energy Saving module. In the following paragraphs, we will describe how to do activity sensing and adopt energy-saving strategies. 3.1 Activity Sensing Based on Section 2.1, the user’s activities are categorized into three levels: body level, arm level, wrist level. In each level, there may be more than one activity. Besides, the user can transfer from one activity to another. To reduce the error of activity recognition, we maintain an activity state machine to describe the transfer relationship of the activities and determine the user’s activity progressively. When we get the sensor data of an activity, we first utilize the variance and periodicity of sensor data to classify the activity into one of the three levels. Then, we combine the data from different sensors to recognize the specific activity in each level. —Body level: It includes motionlessness and body movement. Motionlessness can be recognized with its low variance of linear acceleration (linear-acc for short) and gyroscope data. In regard to body movement, which can be walking, jogging, and so on, we utilize the periodicity of an activity to infer whether the current activity belongs to body movement. That is to say, we do not distinguish walking, jogging, and so on, we aim to recognize an activity as body movement or not. —Arm level: It includes lifting up the arm and laying down the arm. We utilize the relationship between the gravity data and linear acceleration to distinguish the two activities. Besides, we use a voting mechanism to improve the recognition accuracy. —Wrist level: It includes rotating the phone, making fine-tuning, and shooting a photo. We make use of a linear SVM model to distinguish them with the following features: variance, mean, maximum value and minimum value of linear acceleration and gyroscope data in each axis. 3.2 Energy Saving Considering the specific feature of each activity, we propose an adaptive energy-saving scheme called “SenSave” for photographing. Our solution SenSave does not need user interaction (e.g., turn ON/OFF the camera or screen manually) for energy-saving, it utilizes the low power-consuming sensors to detect the user’s activity and adopts a corresponding energy-saving strategy adaptively. For example, when you walk, jog or stay motionless, we can keep the screen off. When you lift your arm up, it is better to turn on the screen and adjust the screen brightness based on the ambient light. When you make fine-tuning to observe the camera view before shooting a photo, it is better to make the camera work in preview mode. In this way, we can make context-aware energy-saving strategies for camera phones. 4 SYSTEM DESIGN In this section, we will show how to use the built-in sensors (i.e., accelerometer and gyroscope) for activity sensing and design an efficient energy-saving scheme for photographing in smart camera phones. ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
Tracking Human Motions in Photographing 299 y-axis x-axis 3-axis Top -axis x-axis y-ax Top z-axis x-axis (a)Coordinates of sensors (b)Hold horizontally naturally (c)Hold horizontally backward Fig.4.Coordinate system of the phone and the attitude of a phone being held. 10 Raw dae eprocs5 ed data 10 100 200 300 400 500 600 Sampling sequence (50Hz) Fig.5.Data preprocessing. 4.1 Activity Sensing 4.1.1 Raw Data Collection.We collect sequential data from accelerometer and gyroscope.From the acceleration,we use a low-pass filter to isolate the gravity data.Then,we use the acceleration to minus the gravity data in each axis to obtain the linear acceleration(Google Inc.2016a,2016c).In the phone,the sensors use a standard 3-axis coordinate system,as shown in Figure 4(a),which may be different from the earth coordinate system.For example,when we hold the phone as shown in Figure 4(b),the gravity data in x-axis x almost equals to g(ie.,9.8m/s2).When we hold the phone as shown in Figure 4(c),x equals to-g(i.e.,-9.8m/s2).Here,g represents the gravity acceleration, which always points towards the ground.It is noteworthy that the direction of the gravity sensor data is opposite to that of the gravity acceleration,according to the definition of SensorEvent in Android APIs(Google Inc.2016a).In this article,we utilize gravity data for activity recognition, while not transforming gravity data to the gravity acceleration.From sensors,we obtain the raw data of linear acceleration(linear-acc for short),gravity data,and gyroscope data,which will be used for following data processing and activity recognition. 4.1.2 Data Preprocessing.The raw sensor data collected from the built-in sensors is usually noisy.To use the sensor data for activity recognition,the data offset and noises should be removed first.For data offset,it means when we keep the device static,the linear-acc and gyroscope data are not equal to zero.Thus,we remove the data offset in each axis.For noises,we use a smooth function to mitigate the effect of noise.As shown in Figure 5,when the sampling rate of the sensor is 50Hz,the blue line shows the raw sensor data of linear acceleration in x-axis.After removing the data offset,we smooth the sensor data in blue line with a 9-point moving average.The smooth function is like a low-pass filter,which removes the high-frequency noise.The red line in Figure 5 represents the smoothed sensor data.The effect of outliers and noises has been mitigated. Unless otherwise specified,we will use the preprocessed sensor data to do activity segmentation for activity detection and recognition,as described in the following sections. ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
Tracking Human Motions in Photographing 29:9 Fig. 4. Coordinate system of the phone and the attitude of a phone being held. Fig. 5. Data preprocessing. 4.1 Activity Sensing 4.1.1 Raw Data Collection. We collect sequential data from accelerometer and gyroscope. From the acceleration, we use a low-pass filter to isolate the gravity data. Then, we use the acceleration to minus the gravity data in each axis to obtain the linear acceleration (Google Inc. 2016a, 2016c). In the phone, the sensors use a standard 3-axis coordinate system, as shown in Figure 4(a), which may be different from the earth coordinate system. For example, when we hold the phone as shown in Figure 4(b), the gravity data in x-axis xv almost equals to д (i.e., 9.8m/s2). When we hold the phone as shown in Figure 4(c), xv equals to −д (i.e., −9.8m/s2). Here, д represents the gravity acceleration, which always points towards the ground. It is noteworthy that the direction of the gravity sensor data is opposite to that of the gravity acceleration, according to the definition of SensorEvent in Android APIs (Google Inc. 2016a). In this article, we utilize gravity data for activity recognition, while not transforming gravity data to the gravity acceleration. From sensors, we obtain the raw data of linear acceleration (linear-acc for short), gravity data, and gyroscope data, which will be used for following data processing and activity recognition. 4.1.2 Data Preprocessing. The raw sensor data collected from the built-in sensors is usually noisy. To use the sensor data for activity recognition, the data offset and noises should be removed first. For data offset, it means when we keep the device static, the linear-acc and gyroscope data are not equal to zero. Thus, we remove the data offset in each axis. For noises, we use a smooth function to mitigate the effect of noise. As shown in Figure 5, when the sampling rate of the sensor is 50Hz, the blue line shows the raw sensor data of linear acceleration in x-axis. After removing the data offset, we smooth the sensor data in blue line with a 9-point moving average. The smooth function is like a low-pass filter, which removes the high-frequency noise. The red line in Figure 5 represents the smoothed sensor data. The effect of outliers and noises has been mitigated. Unless otherwise specified, we will use the preprocessed sensor data to do activity segmentation for activity detection and recognition, as described in the following sections. ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
29:10 Y.Yin et al. Aetivity D Laying Walking up the arm down the arm Walking 200 400 600 800 000 1200 1400 1600 Sampling sequence (50Hz) Fig.6.Linear acceleration. ”y-aXs C3 C4 Rptating the phone Fine-和ning Shooting 600 650 700 750 80 850 900 950 1000 1050 1100 Sampling sequence (50Hz) Fig.7.Gyroscope data in rectangle C. 4.1.3 Activity Segmentation.To detect the user's activity,it is necessary to segment the data of an activity from the sensor data.In Figure 6,we show the sensor data of linear accelerometer during photographing.For the activities in photographing,there is often a short pause between two different activities.In Figure 6,we mark each short pause with a black rectangle,that is, rectangle A,B,and D.Therefore,the linear-acc can be used to separate the following activities: motionlessness,walking(body movement),lifting up the arm,and laying down the arm. However,there still exist some micro activities like rotating the phone,fine-tuning and shoot- ing,which are too gentle to be distinguished by the linear acceleration,as the rectangle C shown in Figure 6.In this case,we use gyroscope data to assist for activity segmentation.This is because gyroscope is more sensitive to micro activities like rotating,fine-tuning and shooting,when com- pared to the linear accelerometer.In Figure 7,we show the gyroscope data corresponding to the activities in rectangle C of Figure 6,and mark the detected short pauses with black rectangles C1, C2,C3,and C4.Therefore,we can use gyroscope data to detect the short pause between different micro activities and separate rotating,fine-tuning and shooting.For shooting,we can also use the screen touch operation to assist for activity recognition. According to the above observations,we utilize the linear acceleration and gyroscope data to do activity segmentation,as shown in the following steps. -Step 1:We introduce a sliding pause window,and then calculate the variances of linear-acc during the pause window.Suppose there are n sampling data of linear-acc located in the slid- ing pause window,we represent the data in x-axis,y-axis,and z-axis as {xa,xa2,...,xan. yal,ya2,...,yanl,and (zal,za2.....zan),respectively.The corresponding resultant linear- acc is ai,ie [1.n].Here,ai=y).The variances of linear-acc during the pause window is s2.which can be calculated based on Equation(2): 后= n-1 ai-a)2. (2) ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:10 Y. Yin et al. Fig. 6. Linear acceleration. Fig. 7. Gyroscope data in rectangle C. 4.1.3 Activity Segmentation. To detect the user’s activity, it is necessary to segment the data of an activity from the sensor data. In Figure 6, we show the sensor data of linear accelerometer during photographing. For the activities in photographing, there is often a short pause between two different activities. In Figure 6, we mark each short pause with a black rectangle, that is, rectangle A, B, and D. Therefore, the linear-acc can be used to separate the following activities: motionlessness, walking (body movement), lifting up the arm, and laying down the arm. However, there still exist some micro activities like rotating the phone, fine-tuning and shooting, which are too gentle to be distinguished by the linear acceleration, as the rectangle C shown in Figure 6. In this case, we use gyroscope data to assist for activity segmentation. This is because gyroscope is more sensitive to micro activities like rotating, fine-tuning and shooting, when compared to the linear accelerometer. In Figure 7, we show the gyroscope data corresponding to the activities in rectangle C of Figure 6, and mark the detected short pauses with black rectangles C1, C2, C3, and C4. Therefore, we can use gyroscope data to detect the short pause between different micro activities and separate rotating, fine-tuning and shooting. For shooting, we can also use the screen touch operation to assist for activity recognition. According to the above observations, we utilize the linear acceleration and gyroscope data to do activity segmentation, as shown in the following steps. —Step 1: We introduce a sliding pause window, and then calculate the variances of linear-acc during the pause window. Suppose there are n sampling data of linear-acc located in the sliding pause window, we represent the data in x-axis, y-axis, and z-axis as {xa1, xa2,..., xan }, {ya1,ya2,...,yan }, and {za1, za2,..., zan }, respectively. The corresponding resultant linearacc is ai,i ∈ [1,n]. Here, ai = (x2 ai + y2 ai + z2 ai ). The variances of linear-acc during the pause window is s2 a, which can be calculated based on Equation (2): s2 a = 1 n − 1 n i=1 (ai − a) 2 . (2) ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017.