David B.Kirk Wen-mei W.Hwu Programming Massively Parallel Processors A Hands-on Approach nVIDIA. MK
In Praise of Programming Massively Parallel Processors: A Hands-on Approach Parallel programming is about performance,for otherwise you'd write a sequential program.For those interested in learning or teaching the topic, a problem is where to find truly parallel hardware that can be dedicated to the task,for it is difficult to see interesting speedups if its shared or only modestly parallel.One answer is graphical processing units(GPUs),which can have hundreds of cores and are found in millions of desktop and laptop computers.For those interested in the GPU path to parallel enlightenment, this new book from David Kirk and Wen-mei Hwu is a godsend,as it intro- duces CUDA,a C-like data parallel language,and Tesla,the architecture of the current generation of NVIDIA GPUs.In addition to explaining the language and the architecture,they define the nature of data parallel pro- blems that run well on heterogeneous CPU-GPU hardware.More con- cretely,two detailed case studies demonstrate speedups over CPU-only C programs of 10X to 15X for naive CUDA code and 45X to 105X for expertly tuned versions.They conclude with a glimpse of the future by describing the next generation of data parallel languages and architectures:OpenCL and the NVIDIA Fermi GPU.This book is a valuable addition to the recently reinvigorated parallel computing literature. David Patterson Director,The Parallel Computing Research Laboratory Pardee Professor of Computer Science,U.C.Berkeley Co-author of Computer Architecture:A Quantitative Approach Written by two teaching pioneers,this book is the definitive practical refer- ence on programming massively parallel processors-a true technological gold mine.The hands-on learning included is cutting-edge,yet very read- able.This is a most rewarding read for students,engineers and scientists interested in supercharging computational resources to solve today's and tomorrow's hardest problems. Nicolas Pinto MIT,NVIDIA Fellow 2009 I have always admired Wen-mei Hwu's and David Kirk's ability to turn complex problems into easy-to-comprehend concepts.They have done it again in this book.This joint venture of a passionate teacher and a GPU
In Praise of Programming Massively Parallel Processors: A Hands-on Approach Parallel programming is about performance, for otherwise you’d write a sequential program. For those interested in learning or teaching the topic, a problem is where to find truly parallel hardware that can be dedicated to the task, for it is difficult to see interesting speedups if its shared or only modestly parallel. One answer is graphical processing units (GPUs), which can have hundreds of cores and are found in millions of desktop and laptop computers. For those interested in the GPU path to parallel enlightenment, this new book from David Kirk and Wen-mei Hwu is a godsend, as it introduces CUDA, a C-like data parallel language, and Tesla, the architecture of the current generation of NVIDIA GPUs. In addition to explaining the language and the architecture, they define the nature of data parallel problems that run well on heterogeneous CPU-GPU hardware. More concretely, two detailed case studies demonstrate speedups over CPU-only C programs of 10X to 15X for naı¨ve CUDA code and 45X to 105X for expertly tuned versions. They conclude with a glimpse of the future by describing the next generation of data parallel languages and architectures: OpenCL and the NVIDIA Fermi GPU. This book is a valuable addition to the recently reinvigorated parallel computing literature. David Patterson Director, The Parallel Computing Research Laboratory Pardee Professor of Computer Science, U.C. Berkeley Co-author of Computer Architecture: A Quantitative Approach Written by two teaching pioneers, this book is the definitive practical reference on programming massively parallel processors—a true technological gold mine. The hands-on learning included is cutting-edge, yet very readable. This is a most rewarding read for students, engineers and scientists interested in supercharging computational resources to solve today’s and tomorrow’s hardest problems. Nicolas Pinto MIT, NVIDIA Fellow 2009 I have always admired Wen-mei Hwu’s and David Kirk’s ability to turn complex problems into easy-to-comprehend concepts. They have done it again in this book. This joint venture of a passionate teacher and a GPU
evangelizer tackles the trade-off between the simple explanation of the con- cepts and the depth analysis of the programming techniques.This is a great book to learn both massive parallel programming and CUDA. Mateo Valero Director,Barcelona Supercomputing Center The use of GPUs is having a big impact in scientific computing.David Kirk and Wen-mei Hwu's new book is an important contribution towards educat- ing our students on the ideas and techniques of programming for massively- parallel processors. Mike Giles Professor of Scientific Computing University of Oxford This book is the most comprehensive and authoritative introduction to GPU computing yet.David Kirk and Wen-mei Hwu are the pioneers in this increasingly important field,and their insights are invaluable and fascinating.This book will be the standard reference for years to come. Hanspeter Pfister Harvard University This is a vital and much needed text.GPU programming is growing by leaps and bounds.This new book will be very welcomed and highly useful across inter-disciplinary fields. Shannon Steinfadt Kent State University
evangelizer tackles the trade-off between the simple explanation of the concepts and the depth analysis of the programming techniques. This is a great book to learn both massive parallel programming and CUDA. Mateo Valero Director, Barcelona Supercomputing Center The use of GPUs is having a big impact in scientific computing. David Kirk and Wen-mei Hwu’s new book is an important contribution towards educating our students on the ideas and techniques of programming for massivelyparallel processors. Mike Giles Professor of Scientific Computing University of Oxford This book is the most comprehensive and authoritative introduction to GPU computing yet. David Kirk and Wen-mei Hwu are the pioneers in this increasingly important field, and their insights are invaluable and fascinating. This book will be the standard reference for years to come. Hanspeter Pfister Harvard University This is a vital and much needed text. GPU programming is growing by leaps and bounds. This new book will be very welcomed and highly useful across inter-disciplinary fields. Shannon Steinfadt Kent State University
Programming Massively Parallel Processors A Hands-on Approach David B.Kirk and Wen-mei W.Hwu AMSTERDAM·BOSTON·HEIDELBERG·LONDON NEW YORK·OXFORD·PARIS·SAN DIEGO MK SAN FRANCISCO·SINGAPORE·SYDNEY·TOKYO ELSEVIER Morgan Kaufmann Publishers is an imprint of Elsevier KAUFMANN
Programming Massively Parallel Processors A Hands-on Approach David B. Kirk and Wen-mei W. Hwu AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Morgan Kaufmann Publishers is an imprint of Elsevier
Morgan Kaufmann Publishers is an imprint of Elsevier. 30 Corporate Drive,Suite 400.Burlington,MA 01803,USA This book is printed on acid-free paper. 2010 David B.Kirk/NVIDIA Corporation and Wen-mei Hwu.Published by Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means.electronic or mechanical,including photocopying,recording.or any information storage and retrieval system,without permission in writing from the publisher.Details on how to seek permission,further information about the Publisher's permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency.can be found at our website:www.elsevier.com/permissions. This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein). NVIDIA,the NVIDIA logo.CUDA,GeForce,Quadro,and Tesla are trademarks or registered trademarks of NVIDIA Corporation in the U.S.and other countries. OpenCL is a trademark of Apple Inc. Notices Knowledge and best practice in this field are constantly changing.As new research and experience broaden our understanding.changes in research methods,professional practices.or medical treatment may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information,methods,compounds,or experiments described herein.In using such information or methods they should be mindful of their own safety and the safety of others,including parties for whom they have a professional responsibility. To the fullest extent of the law,neither the Publisher nor the authors.contributors.or editors.assume any liability for any injury and/or damage to persons or property as a matter of products liability. negligence or otherwise,or from any use or operation of any methods.products,instructions,or ideas contained in the material herein. Library of Congress Cataloging-in-Publication Data Application Submitted British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. ISBN:978-0-12-381472-2 For information on all Morgan Kaufmann publications. visit our Web site at www.mkp.com or www.elsevierdirect.com Printed in United States of America 101112131454321 Working together to grow libraries in developing countries www.clscvicr.com www.bookaid.org www.sabre.org ELSEVIER BOOK AID nternational Sabre Foundation
Morgan Kaufmann Publishers is an imprint of Elsevier. 30 Corporate Drive, Suite 400, Burlington, MA 01803, USA This book is printed on acid-free paper. # 2010 David B. Kirk/NVIDIA Corporation and Wen-mei Hwu. Published by Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions. This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein). NVIDIA, the NVIDIA logo, CUDA, GeForce, Quadro, and Tesla are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. OpenCL is a trademark of Apple Inc. Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility. To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein. Library of Congress Cataloging-in-Publication Data Application Submitted British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. ISBN: 978-0-12-381472-2 For information on all Morgan Kaufmann publications, visit our Web site at www.mkp.com or www.elsevierdirect.com Printed in United States of America 10 11 12 13 14 5 4 3 2 1