上海交通大学：《学术英语写作与演讲 Academic Writing and Presentation in English》教学资源（学生论文）Research Status of Human-Computer Interaction

上游充通大 SHANGHAI JIAO TONG UNIVERSITY RESEARCH STATUS OF HUMAN-COMPUTER INTERACTION Linghan Zheng,Rui Zhang,Chuankai Zhao An Article Submitted to Prof.Li Zhang For the Course of Academic Communication in English 2015.1.1

RESEARCH STATUS OF HUMAN-COMPUTER INTERACTION Linghan Zheng, Rui Zhang, Chuankai Zhao An Article Submitted to Prof. Li Zhang For the Course of Academic Communication in English 2015.1.1 SHANGHAI JIAO TONG UNIVERSITY

上游充通大学 Research Status of Human-Computer Interaction Research Status of Human-Computer Interaction Linghan Zhenga,Rui Zhang,Chuankai Zhao a School of Electronic,Information and Electrical Engineering,Shanghai Jiao Tong University bSchool of Mechanical Engineering.Shanghai Jiao Tong University School of Materials Science and Engineering,Shanghai Jiao Tong University Abstract With the progress of science and technology,human-computer interaction(HCI)expanded rapidly and steadily,having greatly reshaped our lifestyles.However,there is a lack of easily-understood papers to introduce these technologies to the common people.By summarizing the introductive contents in lots of papers,we show here five stages in the development of HCI:keyboard and character display,the mouse pointing device and graphical display,touch technology,multimedia and multimodal interaction,and virtual reality.Furthermore,the latest two phases,the multimodal interaction and the virtual reality,are introduced detailedly.The multimodal interaction is shown in four different aspects,including the speech recognition,the lip-reading,the facial expression recognition and the action recognition.The virtual reality includes the virtual reality system,the virtual reality hardware and the virtual reality software.At last,we predict that these two technologies will become the main directions of future HCI development.This paper provides comprehensive introduction of human-computer interaction and can be a great help to the common people and the junior researchers. Keywords Human-computer interaction;keyboard;mouse;touch technology;multimodal interaction;virtual reality. 1/19

Research Status of Human-Computer Interaction 1 / 19 Research Status of Human-Computer Interaction Linghan Zhenga , Rui Zhangb , Chuankai Zhaoc a School of Electronic, Information and Electrical Engineering, Shanghai Jiao Tong University b School of Mechanical Engineering, Shanghai Jiao Tong University c School of Materials Science and Engineering, Shanghai Jiao Tong University Abstract With the progress of science and technology, human-computer interaction (HCI) expanded rapidly and steadily, having greatly reshaped our lifestyles. However, there is a lack of easily-understood papers to introduce these technologies to the common people. By summarizing the introductive contents in lots of papers, we show here five stages in the development of HCI: keyboard and character display, the mouse pointing device and graphical display, touch technology, multimedia and multimodal interaction, and virtual reality. Furthermore, the latest two phases, the multimodal interaction and the virtual reality, are introduced detailedly. The multimodal interaction is shown in four different aspects, including the speech recognition, the lip-reading, the facial expression recognition and the action recognition. The virtual reality includes the virtual reality system, the virtual reality hardware and the virtual reality software. At last, we predict that these two technologies will become the main directions of future HCI development. This paper provides comprehensive introduction of human-computer interaction and can be a great help to the common people and the junior researchers. Keywords Human-computer interaction; keyboard; mouse; touch technology; multimodal interaction; virtual reality

上海充通大学 Research Status of Human-Computer Interaction Table of Contents1 1.Introduction-- 4 2.Keyboard and Character Display--- 3.Mouse and Graphical Display---------- --5 4.Touch Technology------ -6 5.Multimedia and Multimodal Interaction-------- -6 5.1 Speech Recognition-------- 7 5.1.1 The development of speech recognition 5.1.2 Three technologies of speech recognition 5.1.3 The application of speech recognition 5.2 Lip-reading-------- -8 5.2.1 The brief introduction of lip-reading 5.2.2 The methods applied in lip-reading technology 5.2.3 The application of lip-reading system 5.3 Facial Expression Recognition-------------------- 5.3.1 The significance of facial expression research 5.3.2 The recognition of facial expression 5.3.3 The application of facial expression technology 5.4 Action Recognition-- 11 5.4.1 The recognition of human action 5.4.2 The application of action recognition 6.Virtual Reality ------- 12 6.1 Virtual Reality System------- 12 6.1.1 Desktop Virtual Reality 6.1.2 Immersive Virtual Reality 6.1.3 Enhancement Virtual Reality 6.1.4 Distributed Virtual Reality 6.2 Virtual Reality Hardware------ 13 6.2.1 Helmet Mounted Display 6.2.2 Data Glove 6.2.3 3D Stereo Display 6.3 Virtual Reality Software-- -15 6.3.1 OpenGL 6.3.2ega 6.3.3 Webmax 6.4 The Application of Virtual Reality -16 6.4.1 Military Field 6.4.2 Industrial Field 6.4.3 Medicine Field 7.Conclusion--- -17 Part 1,2,3,4 and 7 were written by Chuankai Zhao;Part 5 was written by Linghan Zheng Part 6 was written by Rui Zhang They contributed equally to the other parts of the article. 2/19

Research Status of Human-Computer Interaction 2 / 19 Table of Contents1 1. Introduction ------------------------------------------------------------------------------------------ 4 2. Keyboard and Character Display ----------------------------------------------------------------- 4 3. Mouse and Graphical Display --------------------------------------------------------------------- 5 4. Touch Technology ----------------------------------------------------------------------------------- 6 5. Multimedia and Multimodal Interaction ----------------------------------------------------------6 5.1 Speech Recognition -------------------------------------------------------------------------- 7 5.1.1 The development of speech recognition 5.1.2 Three technologies of speech recognition 5.1.3 The application of speech recognition 5.2 Lip-reading ------------------------------------------------------------------------------------ 8 5.2.1 The brief introduction of lip-reading 5.2.2 The methods applied in lip-reading technology 5.2.3 The application of lip-reading system 5.3 Facial Expression Recognition ------------------------------------------------------------- 9 5.3.1 The significance of facial expression research 5.3.2 The recognition of facial expression 5.3.3 The application of facial expression technology 5.4 Action Recognition ------------------------------------------------------------------------- 11 5.4.1 The recognition of human action 5.4.2 The application of action recognition 6. Virtual Reality ------------------------------------------------------------------------------------- 12 6.1 Virtual Reality System --------------------------------------------------------------------- 12 6.1.1 Desktop Virtual Reality 6.1.2 Immersive Virtual Reality 6.1.3 Enhancement Virtual Reality 6.1.4 Distributed Virtual Reality 6.2 Virtual Reality Hardware ------------------------------------------------------------------ 13 6.2.1 Helmet Mounted Display 6.2.2 Data Glove 6.2.3 3D Stereo Display 6.3 Virtual Reality Software ------------------------------------------------------------------- 15 6.3.1 OpenGL 6.3.2 Vega 6.3.3 Webmax 6.4 The Application of Virtual Reality ------------------------------------------------------ 16 6.4.1 Military Field 6.4.2 Industrial Field 6.4.3 Medicine Field 7. Conclusion ------------------------------------------------------------------------------------------ 17 1 Part 1, 2, 3, 4 and 7 were written by Chuankai Zhao; Part 5 was written by Linghan Zheng; Part 6 was written by Rui Zhang. They contributed equally to the other parts of the article

上浒究通大学 Research Status of Human-Computer Interaction List of Figures Fig.1 Five stages in the development of HCI Fig.2 Schematic of the development of the keyboard Fig.3 Apple's Magic Mouse Fig.4 Multi-touch technology Fig.5 iTranslate Wice Fig.6 Some pictures from JAFFE Fig.7 Motion capture via Kinect Fig.8 Action recognition applied in somatosensory games Fig.9 Immersive Virtual Reality Fig.10 A cartoon character made by Enhancement Virtual Reality Fig.11 Distributed Virtual Reality used in military training Fig.12 Helmet Mounted Display Fig.13 Data Glove Fig.14 3D Stereo Display Fig.15 Virtual Surgery 3/19

Research Status of Human-Computer Interaction 3 / 19 List of Figures Fig. 1 Five stages in the development of HCI Fig. 2 Schematic of the development of the keyboard Fig. 3 Apple’s Magic Mouse Fig. 4 Multi-touch technology Fig. 5 iTranslate Voice Fig. 6 Some pictures from JAFFE Fig. 7 Motion capture via Kinect Fig. 8 Action recognition applied in somatosensory games Fig. 9 Immersive Virtual Reality Fig. 10 A cartoon character made by Enhancement Virtual Reality Fig. 11 Distributed Virtual Reality used in military training Fig. 12 Helmet Mounted Display Fig. 13 Data Glove Fig. 14 3D Stereo Display Fig. 15 Virtual Surgery

上游充通大学 Research Status of Human-Computer Interaction 1.Introduction Human-computer interaction (HCI)is a complex research and practice area on interaction and communication between human and computer through mutual understandings.In the early 1980s, Association for Computing Machinery first established a special interest group on HCI(Myers, 1998).Since then,HCI has attracted experts and professionals from many other disciplines and incorporated a wide range of concepts and approaches,expanding rapidly and steadily to solve problems of information managements,services and processing in maximizing degree for human needs(Carroll,2010).Over the past three decades,revolutions of HCI technologies have greatly reshaped our life.It is safe to say that human-computer technology is one of the most significant inventions of the 20th century. (a) (b) (c) (d) (e Fig.1 Five stages in the development of HCI (a)Keyboard (b)Mouse (c)Touch (d)Multimedia (e)Virtual Reality In general,as shown in Fig.1,human-computer interaction styles have experienced five phases, each of which can be represented by its milestone technology:keyboard and character display,the mouse pointing device and graphical display,touch technology,multimedia and multimodal interaction,and virtual reality (Karat et al.,2012;Kelner,2000).On the basis of relative literatures, we summarized the technologies used in different stages,and we laid our stress on the latest two phases.Firstly,we briefly introduced the first three stages,including their developmental processes and features.Next we gave a detailed description of four kinds of multimodal interaction-the recognition of speech,lip-reading,facial expression and action.Then we talked more deeply about virtual reality from its system,hardware,software and application.Finally,we discussed the trend of future development of HCI.We hope that this paper could provide some guidance to related research on HCI to be carried out in future. 2.Keyboard Character Display The early stage of HCI originated from the typewriter,based on which keyboard was then invented. The emergence of keyboard brought us into the era of characters user interface,which can be called the first generation of HCI.Keyboard and character display were the main interaction tools,and characters,texts,and commands were the main interaction contents.Thus,the interaction between human and computer in this stage seemed quite stupid and monotonous. Keyboard has been used as the input device of computer and has kept on upgrading since it was invented by Christopher Latham Sholes in the 1878(Myers,1998).According to the difference in switch,keyboards can be classified into several categories:mechanical,membrane,conductive,and capacitive keyboard(Hu,2012).From the late 19th century to the early 21st century,research on 4/19

Research Status of Human-Computer Interaction 4 / 19 1. Introduction Human-computer interaction (HCI) is a complex research and practice area on interaction and communication between human and computer through mutual understandings. In the early 1980s, Association for Computing Machinery first established a special interest group on HCI (Myers, 1998). Since then, HCI has attracted experts and professionals from many other disciplines and incorporated a wide range of concepts and approaches, expanding rapidly and steadily to solve problems of information managements, services and processing in maximizing degree for human needs (Carroll,2010). Over the past three decades, revolutions of HCI technologies have greatly reshaped our life. It is safe to say that human-computer technology is one of the most significant inventions of the 20th century. (a) (b) (c) (d) (e) Fig. 1 Five stages in the development of HCI (a) Keyboard (b) Mouse (c) Touch (d) Multimedia (e) Virtual Reality In general, as shown in Fig. 1, human-computer interaction styles have experienced five phases, each of which can be represented by its milestone technology: keyboard and character display, the mouse pointing device and graphical display, touch technology, multimedia and multimodal interaction, and virtual reality (Karat et al., 2012; Kelner, 2000). On the basis of relative literatures, we summarized the technologies used in different stages, and we laid our stress on the latest two phases. Firstly, we briefly introduced the first three stages, including their developmental processes and features. Next we gave a detailed description of four kinds of multimodal interaction - the recognition of speech, lip-reading, facial expression and action. Then we talked more deeply about virtual reality from its system, hardware, software and application. Finally, we discussed the trend of future development of HCI. We hope that this paper could provide some guidance to related research on HCI to be carried out in future. 2. Keyboard & Character Display The early stage of HCI originated from the typewriter, based on which keyboard was then invented. The emergence of keyboard brought us into the era of characters user interface, which can be called the first generation of HCI. Keyboard and character display were the main interaction tools, and characters, texts, and commands were the main interaction contents. Thus, the interaction between human and computer in this stage seemed quite stupid and monotonous. Keyboard has been used as the input device of computer and has kept on upgrading since it was invented by Christopher Latham Sholes in the 1878 (Myers, 1998). According to the difference in switch, keyboards can be classified into several categories: mechanical, membrane, conductive, and capacitive keyboard(Hu, 2012). From the late 19th century to the early 21st century, research on