CHAPTER4Introduction to Multimedia1.1WHATISMULTIMEDIA?People who use the term “multimedia"often seem to have quite different, even opposing.viewpoints.APC vendor would like us to think of multimedia as a PC that has soundcapability, a DVD-ROM drive, and perhaps the superiority of multimedia-enabled micro-processors that understand additional multimedia instructions.A consumer entertainmentvendor may think of multimedia as interactive cable TV with hundreds of digital channels,or a cable-TV-like service delivered over a high-speed Internet connection.A computer science student reading this book likely has a more application-orientedview of what multimedia consists of:applications that use multiple modalities to theiradvantage,includingtext,images,drawings(graphics),animation,video,sound (includingspeech), and, most likely, interactivity of some kind.The popular notion of"convergenceis one that inhabits the college campus as it does the culture at large.In this scenario,PCs,DVDs,games,digital TV,set-top web surfing,wireless,and so on are converging intechnology,presumably toarriveinthe nearfutureatafinal all-around,multimedia-enabledproduct.Whilehardware mayindeed involve such devices, the present is alreadyexcitingmultimedia is part of some of the most interesting projects underway in computer science.The convergence going on in this field is in fact a convergence of areas that have in the pastbeen separated but are now finding muchto share in this newapplication area.Graphics,visualization,HC,computervision,data compression,graph theory,networking,databasesysternsall have importantcontributions to makeinmultimedia at the present time.1.1.1ComponentsofMultimediaThemultiplemodalitiesoftext,audio,images,drawings,animation,andvideoinmultimediaare put to use in ways as diverse as.Videoteleconferencing·Distributed lecturesforhighereducation.Telemedicine.Cooperative work environments that allowbusiness peopleto edit a shared documentor schoolchildren to share a single game using two mice that pass control back andforth3
4Chapter1IntroductiontoMultimedia. Searching (very) large video and image databases for target visual objects。"Augmented"reality: placing real-appearing computer graphics and video objectsinto scenes so as to take the physics ofobjects and lights (e.g.,shadows)into accountAudio cues for where video-conference participants are seated, as well as taking intoaccountgazedirection and attention ofparticipantsBuilding searchablefeaturesintonew video andenabling veryhigh toverylow bitrateuseofnew, scalablemultimedia products Making multimedia components editable -- allowing the user side to decide whatcomponents, video,graphics,and so on are actually viewed and allowing the clienttomovecomponents around ordeletethem-making components distributedBuilding"inverse-Hollywood"applications that can re-create the process by which avideo was made,allowing storyboard pruning andconcise video summarizationUsing voice recognition to build an interactive environmentsayakitchen-wallwebbrowseFrom the computer science student's point of view, what makes multimedia interestingis that so much of the material covered intraditional computer science areas bears on themultimediaenterprise:networks,operating systems,real-timesystems,vision,informationretrieval.Likedatabases,multimedia touches on manytraditional areas.1.1.2Multimedia ResearchTopics and ProjectsTo the computer science researcher,multimedia consists of a wide variety of topics [1]:Multimedia processing and coding.This includes multimedia content analysis,content-basedmultimedia retrieval,multimedia security,audio/image/video process-ing,compression, and so on.Multimedia system support and networking.People look at such topics as networkprotocols, Internet, operating systems, servers and clients, quality of service (QoS)and databases.Multimedia tools, end systems, and applications. These include hypermedia sys-tems, user interfaces, authoring systems, multimodal interaction, and integration:*"ubiquity"--web-everywhere devices,multimedia education,including computersupportedcollaborativelearning anddesign,and applications ofvirtualenvironments.Theconcernsofmultimediaresearchers also impactresearchers in almosteveryotherbranchof computer science.For example, data mining is an important current research area,anda large database of multimedia data objects is a good example of just what we may beinterested in mining.Telemedicine applications, such as"telemedical patient consultativeencounters,are multimedia applications that place a heavy burden on existing networkarchitectures
Section1.2Multimedia and Hypermedia 5Current MultimediaProjectsMany exciting research projects are currently underwayin multimedia,andwe'd like to introduce afew of them hereForexample,researchers are interested incamera-basedobjecttrackingtechnology.Oneaimistodevelopcontrolsystemsforindustrialcontrol,gaming,andsoonthatrelyonmovingscale models (toys) around a real environment (a board game, say). Tracking the controlobjects (toys)provides user control of the process.3D motion capture can also be used for multiple actor capture, so that multiple realactors in a virtual studio can be used to automatically produce realistic animated modelswithnatural movement.Multiple views from severalcameras orfroma single camera underdifferinglighting canaccurately acquire data that gives both the shape and surface properties of materials, thusautomatically generating synthetic graphics models. This allows photo-realistic (video-quality)synthesisofvirtual actors.3Dcapturetechnologyis nextto fastenough nowto allow acquiringdynamic characteris-tics ofhumanfacialexpression during speech,to synthesize highlyrealistic facial animationfrom speech.Multimedia applications aimed at handicapped persons, particularly those with poorvision and the elderly, are a rich field of endeavor in current research."Digital fashion" aims to develop smart clothing that can communicate with other suchenhanced clothing using wireless communication, so as to artificially enhance human in-teraction in a social setting.The vision here is to use technology to allow individuals toallow certain thoughts andfeelings tobebroadcastautomatically,forexchange with othersequipped with similar technology.GeorgiaTech's Electronic Housecall system,an initiativeforproviding interactivehealthmonitoring services to patients in their homes, relies on networks for delivery, challengingcurrentcapabilities.Behavioral science models can be brought into play tomodel interaction between peo-ple,which can then be extended to enable natural interaction by virtual characters. Such"augmented interaction"applications can be used to develop interfaces between real andvirtual humans for tasks such as augmented storytelling.Each of these application areas pushes the development of computer science generallystimulates new applications,andfascinates practitioners.1.2MULTIMEDIAANDHYPERMEDIATo place multimedia in its proper context, in this section webriefly consider the history ofmultimedia, a recent part of which is the connection between multimedia and hypermedia.We go on toa quickoverview of multimedia software tools availableforcreationof multi-media content,which prepares us to examine,in Chapter 2,the larger issue of integratingthis content into full-blown multimedia productions.1.2.1 History of MultimediaAbriefhistoryoftheuse ofmultimediatocommunicateideasmightbegin with newspapers,whichwereperhaps thefirstmass communicationmedium,usingtext,graphics,and images
IntroductiontoMultimediaChapter1nMotion pictures were originally conceived of in the 1830s to observe motion too rapidfor perception by the human eye.Thomas Alva Edison commissioned the invention of amotion picture camera in 1887. Silent feature films appeared from 1910 to 1927; the silentera effectivelyended with the release of The Jazz Singer in 1927.In1895,GuglielmoMarconisenthisfirstwirelessradiotransmissionatPontecchio,ItalyAfew years later (1901), he detected radiowaves beamed across the Atlantic.Initiallyinvented for telegraph, radio is now a major medium for audio broadcasting.In 1909,Marconi shared theNobel Prize for physics.(ReginaldA.Fessenden,of Quebec,beatMarconi to human voice transmission by several years,but not all inventors receive duecredit.Nevertheless,Fessenden was paid $2.5 million in 1928 for his purloined patents.)Television was the new medium for the twentieth century.It established video as acommonly available medium and has since changed the world of mass communication.The connection between computers and ideas about multimedia covers what is actuallyonlya short period:1945As part of MIT's postwar deliberations on what to do with all those scientists em-ployed on the war effort, Vannevar Bush (1890-1974) wrote a landmark article [2]describing what amounts to a hypermedia system, called "Memex."Memex wasmeant to be a universally usefuland personalized memorydevicethat even includedthe concept ofassociative links-it really is theforerunner of the World Wide Web.After World War II, 6,o00 scientists who had been hard at work on the war effortsuddenly found themselves with time to consider other issues, and the Memex ideawas onefruitofthatnewfreedom.1960s Ted Nelson started the Xanadu project and coined the term"hypertext."Xanaduwas the first attempt at a hypertext system --- Nelson called it a "magic place ofliterarymemory."1967 Nicholas Negroponte formed the Architecture Machine Group at MIT1968 DouglasEngelbart,greatly influenced byVannevar Bush's"As We MayThinkdemonstrated the "On-Line System" (NLS), another early hypertext program. En-gelbart'sgroup at Stanford Research Institute aimed at"augmentation,not automation,to enhance human abilities through computer technology.NLS consisted ofsuch critical ideas as an outline editor for idea development, hypertext links, tele-conferencing, word processing,and e-mail, and made use of the mouse pointingdevice,windowing software, and helpsystems [3].1969 Nelson and van Dam at Brown University created an early hypertext editor calledFRESS[4].The present-day Intermedia project by the Institute for Research inInformation and Scholarship (IRIS)at Brown is the descendant of that early system1976 The MIT Architecture Machine Group proposed a project entitled"Multiple Media."This resulted in the Aspen Movie Map, the first hypermedia videodisc, in 1978.1985 Negroponte and Wiesner cofounded the MIT Media Lab, a leading research institu-tion investigating digital video and multimedia
Section 1.2Multimedia and Hypermedia71989Tim Berners-Leeproposedthe World Wide Web tothe European CouncilforNuclearResearch (CERN).1990 Kristina Hooper Woolsey headed the Apple Multimedia Lab, with a staff of 100.Education was a chief goal.1991 MPEG-1 was approved as an international standard for digital video. Its furtherdevelopment led to newer standards, MPEG-2, MPEG-4, and further MPEGs, in the1990s.1991 The introduction of PDAs in 1991 began a new period in the use of computers ingeneral and multimedia in particular.This development continued in 1996 with themarketing of the first PDA with no keyboard.1992JPEG was accepted as the international standardfor digital image compression.Itsfurtherdevelopmenthasnowled tothenewJPEG2000standard1992Thefirst MBoneaudiomulticast on theNet was made.1993 The University of Illinois National Center for Supercomputing Applications pro-duced NCSAMosaic, the first full-fledged browser,launching a new era in Internetinformationaccess.1994 Jim Clark and Marc Andreessen created the Netscape program1995The JAVA language was createdfor platform-independent application development.1996DVD video was introduced; high-quality,full-length movies were distributed on asingledisk.TheDVDformatpromisedto transformthemusic,gaming andcomputerindustries.1998XML1.0 was announced as a W3C Recommendation.1998 Handheld MP3 devices first made inroads into consumer tastes in thefall,with theintroduction of devices holding 32 MBof flash memory.2000 World Wide Web (WwW) size was estimated at over 1 billion pages.1.2.2HypermediaandMultimediaTed Nelson invented the term"HyperText"around 1965.Whereas we may think of a bookas a linearmedium,basicallymeanttobereadfrombeginningto end,a hypertext systemismeanttobe readnonlinearly,byfollowing links thatpointto other parts of thedocument,or indeed to other documents. Figure 1.1 illustrates this ideaHypermedia is not constrained to be text-based. It can include other media, such asgraphics,images,andespeciallythecontinuousmediasoundand video.ApparentlyTedNelson was also thefirst to use this term.The World Wide Web (WWW)is the bestexampleof ahypermediaapplication.As we have seen, multimedia fundamentally means that computer information can berepresented through audio,graphics, images,video,and animation in addition to traditional media (text andgraphics).Hypermedia can be consideredone particular multimediaapplication