PrefaceAcourseinmultimediaisrapidlybecominganecessityincomputerscienceandengineeringcurricula,especially now that multimedia touches most aspects of thesefields.Multimediawas originally seen as a vertical application area that is,a niche application with methodsthatbelong onlytoitself.However,likepervasivecomputing,multimediaisnowessentiallya horizontal application area and forms an important componentof the study of computergraphics,imageprocessing,databases,real-time systems,operatingsystemsinformationretrieval, computer networks, computer vision,and so on.Multimedia is no longer justa toy butforms part of thetechnological environment in which we work and think.Thisbook fills the need for a university-level text that examines a good deal of the core agendacomputer science sees as belonging to this subject area.Multimedia has become associatedwith a certain set of issues in computer science and engineering, and we address those here.The book is not an introduction to simple design issues--it serves a more advancedaudience than that. On the other hand, it is not a reference work -it is more a traditionaltextbook, While we perforce discuss multimedia tools, we would like to give a sense ofthe underlying principles in the tasks those tools carry out. Students who undertake andsucceed in a course based on this text can be said to really understandfundamental mattersin regard to this material; hence the title of the text.Inconjunctionwiththistext,afull-fledgedcourseshouldalso allow studentstomakeuseofthisknowledgetocarry outinterestingorevenwonderful practical projects inmultimedia,interactive projects that engage and sometimes amuse and, perhaps, even teach these sameconcepts.WhoShouldReadThisBook?This text aims at introducing the basic ideas in multimedia to an audience comfortable withtechnicalapplications—that is,computerscience and engineering students.It aims to coveran upper-level undergraduate multimedia course but could also be used in more advancedcourses and would beagood referencefor anyone, including those in industry,interested incurent multimedia technologies, Graduate students needing a solid grounding in materialstheymay not have seen before wouldundoubtedly benefit from reading it.The text mainly presents concepts,not applications.A multimedia course, on the otherhand, teaches these concepts and tests them but also allows students to use coding andpresentation skills they alreadyknow to address problems in multimedia.The accompanyingweb site shows some of the codeformultimedia applications,alongwith someof the betterprojects students have developed in such a course and other useful materials best presentedelectronically.The ideas in the text drive the results shown in student projects. We assume the readerknows how to program and is also completely comfortable learning yetanother tool.Insteadof concentrating on tools, however, we emphasize what students do not already know.xiv
PrefaceXVUsing the methods and ideas collected here, students are also able to learn more themselves,sometimes in a job setting.It is notunusual for students who take the type of multimediacoursethistext aims attogoon tojobs inamultimedia-relatedindustryimmediatelyaftertheirsenioryear, and sometimes before.The selection of material in the text addresses real issues these learners willface as soonas they showup in the workplace.Some topics are simple but new to the students; someare more complex but unavoidable in this emerging area.Have the Authors Used This Material in a Real Class?Since 1996, we have taught a third-year undergraduate course in multimedia systems basedon the introductory materials set out in this book.A one-semester course could very likelynot include all the material covered in this text, but we have usually managed to considera good many of the topics addressed and to mention a select number of issues in Part IIwithin that time frame.Over the same time period as an introduction to more advancedmaterials, wehave alsotaught a one-semester graduate-levelcourse using notes covering topics similarto the groundcovered by this text.A fourth-year or graduate course would do well to consider materialfrom Parts I and IIof the book and then somematerial from Part I,perhaps in conjunctionwith some of the originalresearchreferences included here and results presented at topicalconferences.We have attempted to fll both needs, concentrating on an undergraduate audience butincludingmore advanced material as well.Sections that can safelybe omitted on a firstreadingaremarkedwithanasterisk.What is Covered in This Text?In Part I, Multimedia Authoring and Data Representations, we introduce some of the no-tions included in theterm multimedia and look at its history as wellas its present.Practi-cally speaking,we carry outmultimedia projccts using softwaretools, so in addition to anoverviewof these tools, weget down to someof the nuts and bolts ofmultimedia authoring.Representing data is critical in multimedia,and we look at the most important data repre-sentationsformultimediaapplications,examiningimagedata,videodata,andaudiodataindetail.Since color is vitally important in multimedia programs,we see how this importantarea impacts multimedia issuesIn Part II, Multimedia Data Compression, we consider how we can make all this datafly onto the screen and speakers.Data compression turns out to be an important enablingtechnologythatmakesmodemmultimediasystemspossible,so welookatlossless and lossycompression methods. For the latter category, JPEG still-image compression standards,including JPEG2000,are arguably themost important, so weconsider these indetail.Butsince a picture is worth a thousand words and video is worth more than a million wordsper minute, we examine the ideas behind MPEG standards MPEG-1, MPEG-2, MPEG-4,MPEG-7andbeyond.Separately,weconsideromebasic audiocompressiontechniquand takealook atMPEGAudio,includingMP3.In Part II, Multimedia Communication and Retrieval, we consider the great demandsmultimedia places on networks and systems. We go on to consider network technologies
xviPrefaceand protocolsthatmakeinteractivemultimedia possible.Someoftheapplications discussedinclude multimediaon demand,multimedia overIP,multimedia over ATM,and multimediaover wireless networks. Content-based retrieval is a particularly important issue in digitallibraries and interactive multimedia,so we examine ideas and systems for this applicationin some detail.TextbookWebSiteThe book's web site is www.cs.sfu.ca/mmbook.There,you willfind copies offiguresfromthe book,an errata sheet updated regularly,programs that help demonstrate concepts in thetext, and a dynamic set of links for the Further Exploration section ofeach chapter.Sincethese links areregularly updated (and of course URLs change often) they are mostly onlineratherthaninthetext.Instructors'ResourcesThe main text web site has no ID and password, but access to sample student projects isat the instructor's discretion and is password-protected, Prentice Hall also hosts a web sitecontaining Course Instructor resources for adopters of the text. These include an extensivecollection of online course notes, a one-semester course syllabus and calendar of eventssolutionsfortheexercises inthe text,sampleassignments and solutions,sampleexams,andextraexamquestions,AcknowledgementsWe are mostgrateful to colleagues whogenerouslygave oftheirtime toreview thistext,andwewishtoepressourthankstoShuChingChendwardChangQianpingGu,RachelleHeller, Gongzhu Hu, S.N. Jayaram, Tiko Kameda,Xiaobo Li, Siwei Lu, Dennis Richards,and Jacques Vaisey.The writing of this texthas been greatly aided by a number of suggestions from presentandformer colleagues and students.We would like to thank James Au, Chad Ciavarro,Hao Jiang, Steven Kilthau, Michael King, Cheng Lu, Yi Sun, Dominic Szopa, ZinoviTauber,Malte von Ruden, Jian Wang, Jie Wei, Edward Yan, Yingchen Yang,Osmar Zaiane,Wenbiao Zhang,and William Zhongfor their assistance.As well, Mr.Ye Lu made greatcontributions to Chapters 8 and 9 and his valiant efforts are particularly appreciated.Weare also most gratefulfor the students who generously made their course projects availableforinstructional usefor this book
PARTONEMULTIMEDIAAUTHORING ANDDATAREPRESENTATIONSChapter1Introduction to Multimedia3Chapter 2MultimediaAuthoringand Tools20Chapter3Graphics and Image Data Representations60Chapter 4Color in Image and Video82Chapter5Fundamental Concepts inVideo112Chapter6Basics of Digital Audio126Introduction to MultimediaAs an introduction to multimedia, in Chapter1 we consider the question of just whatmultimedia is.We examine its history and the development of hypertext and hypermedia.Wethengetdownto practicalmatters with anoverview ofmultimedia softwaretools.Thesearethebasic meanswe useto developmultimedia content.Butamultimedia production ismuch more than the sum of its parts, so Chapter 2 looks at the nuts and bolts of multimediaauthoring design and a taxonomy of authoring metaphors.The chapter also sets out a listof important contemporarymultimedia authoringtools in currentuse.MultimediaDataRepresentationsAs in many fields, the issue of how to best represent the data is of crucial importance inthestudyof multimedia.Chapters3through6considerhowthisisaddressed inthisfield,setting out the most important data representations in multimedia applications.Because themain areas of concern are images, moving pictures, and audio, we begin investigating these1
2in Chapter 3,Graphics and ImageData Representations, then look at Basics of Video inChapter 5.Before going on to Chapter 6,Basics of Digital Audio, we take a side trip inChapter 4 to explore several issues on the use of color, since color is vitally important inmultimediaprograms