AirContour:Building Contour-based Model for in-Air Writing Gesture Recognition YAFENG YIN,State Key Laboratory for Novel Software Technology,Nanjing University,China LEI XIE (corresponding author),State Key Laboratory for Novel Software Technology,Nanjing University,China TAO GU,RMIT University,Australia YIJIA LU,State Key Laboratory for Novel Software Technology,Nanjing University,China SANGLU LU,State Key Laboratory for Novel Software Technology,Nanjing University,China Recognizing in-air hand gestures will benefit a wide range of applications such as sign language recognition,remote control with hand gestures,and "writing"in the air as a new way of text input.This paper presents AirContour,which focuses on in-air writing gesture recognition with a wrist-worn device.We propose a novel contour-based gesture model which converts human gestures to contours in 3D space,and then recognize the contours as characters.Different from 2D contours, the 3D contours may have the problems such as contour distortion caused by different viewing angles,contour difference caused by different writing directions,and the contour distribution across different planes.To address the above problem,we introduce Principal Component Analysis(PCA) to detect the principal/writing plane in 3D space,and then tune the projected 2D contour in the principal plane through reversing,rotating and normalizing operations,to make the 2D contour in right orientation and normalized size under a uniform view.After that,we propose both an online approach AC-Vec and an offline approach AC-CNN for character recognition.The experimental results show that AC-Vec achieves an accuracy of 91.6%and AC-CNN achieves an accuracy of 94.3%for gesture/character recognition,both outperform the existing approaches. CCS Concepts:.Human-centered computing-Ubiquitous and mobile computing de- sign and evaluation methods;Empirical studies in ubiquitous and mobile computing. Additional Key Words and Phrases:AirContour,in-air writing,contour-based gesture model, principal component analysis(PCA),gesture recognition ACM Reference Format: Yafeng Yin,Lei Xie (corresponding author),Tao Gu,Yijia Lu,and Sanglu Lu.2019.AirContour: Building Contour-based Model for in-Air Writing Gesture Recognition.ACM Trans.Sensor Netw. 1,1,Article 1 (January 2019),26 pages.https://doi.org/10.1145/3343855 Authors'addresses:Yafeng Yin,State Key Laboratory for Novel Software Technology,Nanjing University, Nanjing,China,yafeng@nju.edu.cn;Lei Xie(corresponding author),State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing,China,Ixie@nju.edu.cn;Tao Gu,RMIT University,Melbourne, Australia,tao.gu@rmit.edu.au;Yijia Lu,State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing,China,lyj@smail.nju.edu.cn;Sanglu Lu,State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing,China,sanglu@nju.edu.cn. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.Copyrights for components of this work owned by others than ACM must be honored.Abstracting with credit is permitted.To copy otherwise, or republish,to post on servers or to redistribute to lists,requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. C)2019 Association for Computing Machinery. 1550-4859/2019/1-ART1$15.00 https://doi.org/10.1145/3343855 ACM Trans.Sensor Netw.,Vol.1,No.1,Article 1.Publication date:January 2019
1 AirContour: Building Contour-based Model for in-Air Writing Gesture Recognition YAFENG YIN, State Key Laboratory for Novel Software Technology, Nanjing University, China LEI XIE (corresponding author), State Key Laboratory for Novel Software Technology, Nanjing University, China TAO GU, RMIT University, Australia YIJIA LU, State Key Laboratory for Novel Software Technology, Nanjing University, China SANGLU LU, State Key Laboratory for Novel Software Technology, Nanjing University, China Recognizing in-air hand gestures will benefit a wide range of applications such as sign language recognition, remote control with hand gestures, and “writing” in the air as a new way of text input. This paper presents AirContour, which focuses on in-air writing gesture recognition with a wrist-worn device. We propose a novel contour-based gesture model which converts human gestures to contours in 3D space, and then recognize the contours as characters. Different from 2D contours, the 3D contours may have the problems such as contour distortion caused by different viewing angles, contour difference caused by different writing directions, and the contour distribution across different planes. To address the above problem, we introduce Principal Component Analysis (PCA) to detect the principal/writing plane in 3D space, and then tune the projected 2D contour in the principal plane through reversing, rotating and normalizing operations, to make the 2D contour in right orientation and normalized size under a uniform view. After that, we propose both an online approach AC-Vec and an offline approach AC-CNN for character recognition. The experimental results show that AC-Vec achieves an accuracy of 91.6% and AC-CNN achieves an accuracy of 94.3% for gesture/character recognition, both outperform the existing approaches. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing design and evaluation methods; Empirical studies in ubiquitous and mobile computing. Additional Key Words and Phrases: AirContour, in-air writing, contour-based gesture model, principal component analysis (PCA), gesture recognition ACM Reference Format: Yafeng Yin, Lei Xie (corresponding author), Tao Gu, Yijia Lu, and Sanglu Lu. 2019. AirContour: Building Contour-based Model for in-Air Writing Gesture Recognition. ACM Trans. Sensor Netw. 1, 1, Article 1 (January 2019), 26 pages. https://doi.org/10.1145/3343855 Authors’ addresses: Yafeng Yin, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China, yafeng@nju.edu.cn; Lei Xie (corresponding author), State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China, lxie@nju.edu.cn; Tao Gu, RMIT University, Melbourne, Australia, tao.gu@rmit.edu.au; Yijia Lu, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China, lyj@smail.nju.edu.cn; Sanglu Lu, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China, sanglu@nju.edu.cn. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2019 Association for Computing Machinery. 1550-4859/2019/1-ART1 $15.00 https://doi.org/10.1145/3343855 ACM Trans. Sensor Netw., Vol. 1, No. 1, Article 1. Publication date: January 2019
1:2 Y.Yin et al. 1 INTRODUCTION With the advancement of rich embedded sensors,mobile or wearable devices(e.g.,smartphone, smartwatch)have been largely used in activity recognition [37]21][23][45][26][31][41],and benefit many human-computer interactions,e.g.,motion-sensing games [25],sign language recognition [12],in-air writing [1],etc.As a typical interaction mode,writing in the air has aroused wide attentions [9][10][39][36][6].It allows user to write characters with arm and hand freely in the air without focusing the attentions on the small screen or tiny keys on a device [2.As shown in Fig.1,a user carrying/wearing a sensor-embedded device writes in the air,and the gesture will be recognized as a character.Recognizing in-air writing gestures is a key technology to facilitate writing gesture-based interactions in the air and can be used in many scenarios.For example,"writing"commands in the air to control a unmanned aerial vehicle(UAV),while looking at the scene transmitted from the UAV in a virtual reality(VR)headset,to avoid taking off the VR headset and inputting the commands with a controller.Another example could be replacing the traditional on-screen text input by "writing"the text message in the air,thus allowing to interact with mobile or wearable devices having tiny or no screen.Besides,when one hand of the user is occupied,typing with a keyboard becomes inconvenient,the sensor-assisted in-air input technology can be used to capture hand gestures and lay them out in text or image [1].When comparing to the existing handwriting,voice or camera-based input,in-air writing with inertial sensors can tolerate the limited screen,environmental noises and poor light conditions.In this paper, we focus on recognizing in-air writing gestures as characters. Wrist-worn device (eg.,smartwatch)】 In-air In-air/3D Output writing gesture contours (recognized characters) Fig.1.AirContour:in-air writing gesture recognition based on contours In inertial sensor based gesture recognition,many approaches have been proposed.Some data-driven approaches [10][2][7]35][15]tend to extract features from sensor data to train classifiers for gesture recognition,while paying little attention on human activity analysis.If the user performs gestures with more degrees of freedom,i.e.,the gestures may have large variations in speeds,sizes,or orientations,the type of approaches may fail to recognize them with high accuracy.On the Contrast,some pattern-driven approaches [1][32][13]try to capture the moving patterns of gestures for activity recognition.For example,Agrawal et al. [1]utilize the segmented strokes and grammar tree to recognize capital letters in a 2D plane. However,due to the complexity of analyzing human activities,the type of approaches may redefine the gesture patterns or constrain the gestures in a limited area(e.g.,on a limited 2D plane),which may decrease user experience.To track the continuous in-air gestures, Shen et al.[29]utilize the 5-DoF arm model and HMM to track the 3D posture of the arm. However,in 3D space,tracking is not directly linked to recognition,especially when the ACM Trans.Sensor Netw.,Vol.1,No.1,Article 1.Publication date:January 2019
1:2 Y. Yin et al. 1 INTRODUCTION With the advancement of rich embedded sensors, mobile or wearable devices (e.g., smartphone, smartwatch) have been largely used in activity recognition [37][21][23][45][26][31][41], and benefit many human-computer interactions, e.g., motion-sensing games [25], sign language recognition [12], in-air writing [1], etc. As a typical interaction mode, writing in the air has aroused wide attentions [9][10][39][36][6]. It allows user to write characters with arm and hand freely in the air without focusing the attentions on the small screen or tiny keys on a device [2]. As shown in Fig. 1, a user carrying/wearing a sensor-embedded device writes in the air, and the gesture will be recognized as a character. Recognizing in-air writing gestures is a key technology to facilitate writing gesture-based interactions in the air and can be used in many scenarios. For example, “writing” commands in the air to control a unmanned aerial vehicle (UAV), while looking at the scene transmitted from the UAV in a virtual reality (VR) headset, to avoid taking off the VR headset and inputting the commands with a controller. Another example could be replacing the traditional on-screen text input by “writing” the text message in the air, thus allowing to interact with mobile or wearable devices having tiny or no screen. Besides, when one hand of the user is occupied, typing with a keyboard becomes inconvenient, the sensor-assisted in-air input technology can be used to capture hand gestures and lay them out in text or image [1]. When comparing to the existing handwriting, voice or camera-based input, in-air writing with inertial sensors can tolerate the limited screen, environmental noises and poor light conditions. In this paper, we focus on recognizing in-air writing gestures as characters. Fig. 1. AirContour: in-air writing gesture recognition based on contours In inertial sensor based gesture recognition, many approaches have been proposed. Some data-driven approaches [10][2][7][35][15] tend to extract features from sensor data to train classifiers for gesture recognition, while paying little attention on human activity analysis. If the user performs gestures with more degrees of freedom, i.e., the gestures may have large variations in speeds, sizes, or orientations, the type of approaches may fail to recognize them with high accuracy. On the Contrast, some pattern-driven approaches [1][32][13] try to capture the moving patterns of gestures for activity recognition. For example, Agrawal et al. [1] utilize the segmented strokes and grammar tree to recognize capital letters in a 2D plane. However, due to the complexity of analyzing human activities, the type of approaches may redefine the gesture patterns or constrain the gestures in a limited area (e.g., on a limited 2D plane), which may decrease user experience. To track the continuous in-air gestures, Shen et al. [29] utilize the 5-DoF arm model and HMM to track the 3D posture of the arm. However, in 3D space, tracking is not directly linked to recognition, especially when the ACM Trans. Sensor Netw., Vol. 1, No. 1, Article 1. Publication date: January 2019
AirContour 1:3 trajectory (e.g.,handwriting trajectory)locates in different planes.Therefore,it is still a challenging task to apply the existing approaches to recognize in-air writing gestures which occurs in 3D space with more degrees of freedom,while guaranteeing user experience. To address aforementioned issues,in this paper we explore contours to represent in-air writing gestures,and propose a novel contour-based gesture model,where the 'contour'is represented with a sequence of coordinate points over time.We use an off-the-shelf wrist-worn device (e.g.,smartwatch)to collect sensor data,and our basic idea is to build a 3D contour model for each gesture and utilize the contour feature to recognize gestures as characters, as illustrated in Fig.1.Since the gesture contour keeps the essential movement patterns of in-air gestures,it can tolerate the intra-class variability of gestures.It is worth noting that while the proposed 'contour-gesture'model is applied in in-air writing gesture recognition for this work,it can also be used in sign language recognition and remote control with hand gestures [40].However,different from 2D contours,building 3D contours presents several challenges,i.e.,contour distortion caused by different viewing angles,contour difference caused by different writing directions,contour distribution across different planes,making it difficult to recognize 3D contours as 2D characters.To solve this problem,we first describe the range of viewing angles based on the way that the device is worn,which indicates the possible writing directions.We then apply Principal Component Analysis(PCA)to detect the principal/writing plane,i.e.,most of the contour is located in or close to the plane.After that,we calibrate the 2D projected contour in the principal plane for gesture/character recognition,while considering the distortion caused by dimensionality reduction and the difference of gesture sizes. We make the following contributions in this paper. To the best of our knowledge,we are the first to propose the contour-based gesture model to recognize in-air writing gestures.The model is designed to solve the new challenges in 3D gesture contours,e.g.,observation ambiguity,uncertain orientation and distribution of 3D contours,and tolerate the intra-class variability of gestures.The contour-based gesture model can be applied in not only in-air writing gesture recognition,but also many other scenarios such as sign language recognition,motion-sensing games and remote control with hand gestures. To recognize gesture contours in 3D space to characters in a 2D plane,we introduce PCA for dimensionality reduction and a series of calibrations for 2D contours.Specifically, we first utilize PCA to detect the principal/writing plane,and then project the 3D contour into the principal plane for dimensionality reduction.After that,we calibrate the 2D contour in the principal plane through reversing,rotating and normalizing operations,to make it in right orientation and normalized size under a uniform view, i.e.,to make the 2D contour suitable for character recognition. We conduct extensive experiments to verify the efficiency of the proposed contour- based gesture model.In addition,based on the model,we propose an online approach AC-Vec and an offline approach AC-CNN to recognize 2D contours as characters.The experimental results show that AC-Vec and AC-CNN achieve an accuracy of 91.6% and 94.3%,respectively,for gesture/character recognition,and both outperform the existing approaches. 2 RELATED WORK In this section,we describe and analyze the state-of-the-art related to in-air gesture recogni- tion,tracking,writing in the air,and handwritten character recognition,especially focus on inertial sensor based techniques. ACM Trans.Sensor Netw.,Vol.1,No.1,Article 1.Publication date:January 2019
AirContour 1:3 trajectory (e.g., handwriting trajectory) locates in different planes. Therefore, it is still a challenging task to apply the existing approaches to recognize in-air writing gestures which occurs in 3D space with more degrees of freedom, while guaranteeing user experience. To address aforementioned issues, in this paper we explore contours to represent in-air writing gestures, and propose a novel contour-based gesture model, where the ‘contour’ is represented with a sequence of coordinate points over time. We use an off-the-shelf wrist-worn device (e.g., smartwatch) to collect sensor data, and our basic idea is to build a 3D contour model for each gesture and utilize the contour feature to recognize gestures as characters, as illustrated in Fig. 1. Since the gesture contour keeps the essential movement patterns of in-air gestures, it can tolerate the intra-class variability of gestures. It is worth noting that while the proposed ‘contour-gesture’ model is applied in in-air writing gesture recognition for this work, it can also be used in sign language recognition and remote control with hand gestures [40]. However, different from 2D contours, building 3D contours presents several challenges, i.e., contour distortion caused by different viewing angles, contour difference caused by different writing directions, contour distribution across different planes, making it difficult to recognize 3D contours as 2D characters. To solve this problem, we first describe the range of viewing angles based on the way that the device is worn, which indicates the possible writing directions. We then apply Principal Component Analysis (PCA) to detect the principal/writing plane, i.e., most of the contour is located in or close to the plane. After that, we calibrate the 2D projected contour in the principal plane for gesture/character recognition, while considering the distortion caused by dimensionality reduction and the difference of gesture sizes. We make the following contributions in this paper. ∙ To the best of our knowledge, we are the first to propose the contour-based gesture model to recognize in-air writing gestures. The model is designed to solve the new challenges in 3D gesture contours, e.g., observation ambiguity, uncertain orientation and distribution of 3D contours, and tolerate the intra-class variability of gestures. The contour-based gesture model can be applied in not only in-air writing gesture recognition, but also many other scenarios such as sign language recognition, motion-sensing games and remote control with hand gestures. ∙ To recognize gesture contours in 3D space to characters in a 2D plane, we introduce PCA for dimensionality reduction and a series of calibrations for 2D contours. Specifically, we first utilize PCA to detect the principal/writing plane, and then project the 3D contour into the principal plane for dimensionality reduction. After that, we calibrate the 2D contour in the principal plane through reversing, rotating and normalizing operations, to make it in right orientation and normalized size under a uniform view, i.e., to make the 2D contour suitable for character recognition. ∙ We conduct extensive experiments to verify the efficiency of the proposed contourbased gesture model. In addition, based on the model, we propose an online approach AC-Vec and an offline approach AC-CNN to recognize 2D contours as characters. The experimental results show that AC-Vec and AC-CNN achieve an accuracy of 91.6% and 94.3%, respectively, for gesture/character recognition, and both outperform the existing approaches. 2 RELATED WORK In this section, we describe and analyze the state-of-the-art related to in-air gesture recognition, tracking, writing in the air, and handwritten character recognition, especially focus on inertial sensor based techniques. ACM Trans. Sensor Netw., Vol. 1, No. 1, Article 1. Publication date: January 2019
1:4 Y.Yin et al. In-air gesture recognition:Parate et al.[26]design a mobile solution called RisQ to detect smoking gestures and sessions with a wristband and use a machine learning pipeline to process sensor data.Blank et al.[7]present a system for table tennis stroke detection and classification by attaching inertial sensors to table tennis rackets.Thomaz et al.[31]describe the implementation and evaluation of an approach to infer eating moments using a 3-axis accelerometer in smartwatch.Xu et al.[35]build a classifier to identify user's hand and finger gestures utilizing the essential features of accelerometer and gyroscope data measured from smartwatch.Huang et al.[18]build a system to monitor the brushing quality using a manual toothbrush modified by attaching small magnets to the handle and an off-the-shelf smart watch.These approaches typically extract features from sensor data and apply machine learning techniques for gesture recognition. In-air gesture tracking:Zhou et al.[42][44][43]utilize a kinematic chain to track the human upper limb motion by placing multiple devices on the arm.Cutti et al.[11]utilize the joint angles to track the movements of upper limbs by placing sensors on the chest,shoulder, arm,and wrist.Chen et al.[8]design a wearable system consisting of a pair of magnetometers on fingers and a permanent magnet affixed to the thumb,and introduce uTrack to convert the thumb and fingers into a continuous input system (e.g.,3D pointing).Shen et al.[29] utilize the 5-DoF arm model and HMM to track the 3D posture of the arm,using both motion and magnetic sensors in smartwatch.In fact,accurate in-air gesture tracking in real time can be very challenging.Besides,obtaining the 3D moving trajectory does not mean recognizing the in-air gesture.In this paper,we do not require accurate trajectory tracking, while aiming to obtain the gesture contour and recognize it as a character. Writing in the air:Zhang et al.[39]quantify data into small integral vectors based on acceleration orientation,and then use HMM to recognize 10 Arabic numerals.Wang et al.[32] present IMUPEN to reconstruct motion trajectory and recognize handwritten digits.Bashir et al.[6]use a pen equipped with inertial sensors and apply DTW to recognize handwritten characters.Agrawal et al.[1]recognize handwritten capital letters and Arabic numerals in a 2D plane based on strokes and a grammar tree,by using the built-in accelerometer in smartphone.Amma et al.[2]design a glove equipped with inertial sensors,and use SVM, HMM and statistical language model to recognize capital letters,sentences,etc.Deselaers et al.[13]present GyroPen to reconstruct the writing path for pen-like interaction.Xu et al.[36] utilize the continuous density HMM and Viterbi algorithm to recognize handwritten digits and letters using inertial sensors.In this paper,we focus on single in-air character recognition without the assistance of language model.For a character,we do not define specific strokes or require pen-up for stroke segmentation,while tolerating the intra-class variability caused by writing speeds,gesture sizes,writing directions and observation ambiguity caused by viewing angles etc in 3D space. Handwritten character recognition:In addition to inertial sensor-based approaches, many image processing techniques [31416 are also adopted for recognizing handwritten characters in a 2D plane (i.e.,image).Bahlmann et al.[4]combine DTW and SVMs to establish a Gaussian DTW(GDTW)kernel for on-line recognition of UNIPEN handwriting data.Rayar et al.[28]propose preselection method for CNN-based classification and evaluate it in handwritten character recognition in images.Rao et al.[27 propose a newly designed network structure based on an extended nonlinear kernel residual network,to recognize the handwritten characters over MINIST and SVHN dataset.These approaches focus on recognizing hand-moving trajectories in a 2D plane,while our paper focuses on transforming the 3D gesture into a proper 2D contour,and then utilizes the contour's space-time feature to recognize contours as characters. ACM Trans.Sensor Netw.,Vol.1,No.1,Article 1.Publication date:January 2019
1:4 Y. Yin et al. In-air gesture recognition: Parate et al. [26] design a mobile solution called RisQ to detect smoking gestures and sessions with a wristband and use a machine learning pipeline to process sensor data. Blank et al. [7] present a system for table tennis stroke detection and classification by attaching inertial sensors to table tennis rackets. Thomaz et al. [31] describe the implementation and evaluation of an approach to infer eating moments using a 3-axis accelerometer in smartwatch. Xu et al. [35] build a classifier to identify user’s hand and finger gestures utilizing the essential features of accelerometer and gyroscope data measured from smartwatch. Huang et al. [18] build a system to monitor the brushing quality using a manual toothbrush modified by attaching small magnets to the handle and an off-the-shelf smart watch. These approaches typically extract features from sensor data and apply machine learning techniques for gesture recognition. In-air gesture tracking: Zhou et al. [42][44][43] utilize a kinematic chain to track the human upper limb motion by placing multiple devices on the arm. Cutti et al. [11] utilize the joint angles to track the movements of upper limbs by placing sensors on the chest, shoulder, arm, and wrist. Chen et al. [8] design a wearable system consisting of a pair of magnetometers on fingers and a permanent magnet affixed to the thumb, and introduce uTrack to convert the thumb and fingers into a continuous input system (e.g., 3D pointing). Shen et al. [29] utilize the 5-DoF arm model and HMM to track the 3D posture of the arm, using both motion and magnetic sensors in smartwatch. In fact, accurate in-air gesture tracking in real time can be very challenging. Besides, obtaining the 3D moving trajectory does not mean recognizing the in-air gesture. In this paper, we do not require accurate trajectory tracking, while aiming to obtain the gesture contour and recognize it as a character. Writing in the air: Zhang et al. [39] quantify data into small integral vectors based on acceleration orientation, and then use HMM to recognize 10 Arabic numerals. Wang et al. [32] present IMUPEN to reconstruct motion trajectory and recognize handwritten digits. Bashir et al. [6] use a pen equipped with inertial sensors and apply DTW to recognize handwritten characters. Agrawal et al. [1] recognize handwritten capital letters and Arabic numerals in a 2D plane based on strokes and a grammar tree, by using the built-in accelerometer in smartphone. Amma et al. [2] design a glove equipped with inertial sensors, and use SVM, HMM and statistical language model to recognize capital letters, sentences, etc. Deselaers et al. [13] present GyroPen to reconstruct the writing path for pen-like interaction. Xu et al. [36] utilize the continuous density HMM and Viterbi algorithm to recognize handwritten digits and letters using inertial sensors. In this paper, we focus on single in-air character recognition without the assistance of language model. For a character, we do not define specific strokes or require pen-up for stroke segmentation, while tolerating the intra-class variability caused by writing speeds, gesture sizes, writing directions and observation ambiguity caused by viewing angles etc in 3D space. Handwritten character recognition: In addition to inertial sensor-based approaches, many image processing techniques [3][14][16] are also adopted for recognizing handwritten characters in a 2D plane (i.e., image). Bahlmann et al. [4] combine DTW and SVMs to establish a Gaussian DTW (GDTW) kernel for on-line recognition of UNIPEN handwriting data. Rayar et al. [28] propose preselection method for CNN-based classification and evaluate it in handwritten character recognition in images. Rao et al. [27] propose a newly designed network structure based on an extended nonlinear kernel residual network, to recognize the handwritten characters over MINIST and SVHN dataset. These approaches focus on recognizing hand-moving trajectories in a 2D plane, while our paper focuses on transforming the 3D gesture into a proper 2D contour, and then utilizes the contour’s space-time feature to recognize contours as characters. ACM Trans. Sensor Netw., Vol. 1, No. 1, Article 1. Publication date: January 2019
AirContour 1:5 3 TECHNICAL CHALLENGES AND DEFINITIONS IN IN-AIR GESTURE RECOGNITION 3.1 Intra-class Variability in Sensor Data As shown in Fig.2,even when the user performs the same type of gestures (e.g.,writes 't'), the sensor data can be quite different,due to the variation of writing speeds(Fig.2(a)), gesture sizes (Fig.2(b)),writing directions (Fig.2(c)),etc.Directly using the extracted features from sensor data may fail to recognize in-air gestures accurately. To handle the intra-class variability of in-air gestures,e.g.,the variation of speed,amplitude, and orientation of gestures,we present the contour-based gesture model,which utilizes contours to correlate sensor data with human gestures.The 'contour'is represented with a sequence of coordinate points over time.Additionally,to avoid the differences caused by facing directions,we transform the sensor data from device coordinate system to human coordinate system shown in Fig.5(a),i.e.,we analyze the 3D contours in human coordinate system.In this paper,we take the instance of writing characters in the air to illustrate the contour-based gesture model.The characters are referred to the alphabet,i.e.,a'z',and we use the term“character'”,“letter'”interchangeably throughout the paper.It is worth mentioning that in-air writing letters can be different from printed letters,due to joined-up writing.In particular,we remove the point of 'i'and j',and use't'to represent the letter I' for simplification. 2420 25 nong eqene 5npg8net120 4 5nptnms9ne65we120 hamal 40的 Below 一 —6—上的 一泡将日 28 一dm6 -od- (0mgn 的mng0coI网e,0 (a)Different speeds (b)Different sizes (c)Different directions Fig.2.Linear acceleration of writing the same character 't' 3.2 Difference between 2D Contours and 3D Contours Usually,people get used to recognizing and reading handwritten characters in a 2D plane, e.g.,on a piece of paper.Therefore,we can map a 2D gesture contour with a 2D character for recognition.However,based on extensive observations and experimental study,we find that 3D contour recognition is quite different from 2D contour recognition.In fact,recognizing 3D contours as 2D characters can be very challenging,due to the contour distortion caused by viewing angles,contour difference caused by writing directions,and contour distribution across different planes,as described below. 3.2.1 Viewing Angles.There is a uniform viewing angle for a 2D character contour,while multiple viewing angles for a 3D character contour.In a predefined plane-coordinate system, ACM Trans.Sensor Netw.,Vol.1,No.1,Article 1.Publication date:January 2019
AirContour 1:5 3 TECHNICAL CHALLENGES AND DEFINITIONS IN IN-AIR GESTURE RECOGNITION 3.1 Intra-class Variability in Sensor Data As shown in Fig. 2, even when the user performs the same type of gestures (e.g., writes ‘t’), the sensor data can be quite different, due to the variation of writing speeds (Fig. 2(a)), gesture sizes (Fig. 2(b)), writing directions (Fig. 2(c)), etc. Directly using the extracted features from sensor data may fail to recognize in-air gestures accurately. To handle the intra-class variability of in-air gestures, e.g., the variation of speed, amplitude, and orientation of gestures, we present the contour-based gesture model, which utilizes contours to correlate sensor data with human gestures. The ‘contour’ is represented with a sequence of coordinate points over time. Additionally, to avoid the differences caused by facing directions, we transform the sensor data from device coordinate system to human coordinate system shown in Fig. 5(a), i.e., we analyze the 3D contours in human coordinate system. In this paper, we take the instance of writing characters in the air to illustrate the contour-based gesture model. The characters are referred to the alphabet, i.e., ‘a’ ∼ ‘z’, and we use the term “character”, “letter” interchangeably throughout the paper. It is worth mentioning that in-air writing letters can be different from printed letters, due to joined-up writing. In particular, we remove the point of ‘i’ and ‘j’, and use ‘ι’ to represent the letter ‘l’ for simplification. User13, D2-Front, D4-Right User6, Yinyafeng20, D3; 60-D5, 40-D2 YinyafengFast, D1-Fast， D5-Normal, Gezefan D2-Slow (a) Different speeds User13, D2-Front, D4-Right User6, Yinyafeng20, D3; 60-D5, 40-D2 YinyafengFast, D1-Fast， D5-Normal, Gezefan D2-Slow (b) Different sizes User13, D2-Front, D4-Right, D5-Down User6, Yinyafeng20, D3; 60-D4, 40-D2 YinyafengFast, D1-Fast D5-Normal, Gezefan D2-Slow Below (c) Different directions Fig. 2. Linear acceleration of writing the same character ‘t’ 3.2 Difference between 2D Contours and 3D Contours Usually, people get used to recognizing and reading handwritten characters in a 2D plane, e.g., on a piece of paper. Therefore, we can map a 2D gesture contour with a 2D character for recognition. However, based on extensive observations and experimental study, we find that 3D contour recognition is quite different from 2D contour recognition. In fact, recognizing 3D contours as 2D characters can be very challenging, due to the contour distortion caused by viewing angles, contour difference caused by writing directions, and contour distribution across different planes, as described below. 3.2.1 Viewing Angles. There is a uniform viewing angle for a 2D character contour, while multiple viewing angles for a 3D character contour. In a predefined plane-coordinate system, ACM Trans. Sensor Netw., Vol. 1, No. 1, Article 1. Publication date: January 2019