A.3.9One-Sided Communications Fortran Bindings..··.······. 563 A.3.l0 External Interfaces Fortran Bindings·················· 564 A.3.l1I/O Fortran Bindings....···.················ 564 A.3.12 Language Bindings Fortran Bindings 568 A.3.13 Profiling Interface Fortran Bindings 569 A.3.14 Deprecated Fortran Bindings 569 A.4C++Bindings(deprecated)...··.··.·· 571 A.4.1 Point-to-Point Communication C++Bindings 571 A.4.2 Datatypes C++Bindings.·.··..·.···. 574 A.4.3 Collective Communication C++Bindings.. 576 A.4.4 Groups,Contexts,Communicators,and Caching C++Bindings 577 A.4.5 Process Topologies C++Bindings··················· 580 A.4.6 MPI Environmenta Management C++Bindings 582 A.4.7 The Info Object C++Bindings........ 583 A.4.8 Process Creation and Management C++Bindings 584 A.4.9 One-Sided Communications C++Bindings.......... 585 A.4.10 External Interfaces C++Bindings 586 A.4.11I/0C++Bindings.......·...·..·····.· 586 A.4.12 Language Bindings C++Bindings 590 A.4.13 Profiling Interface C++Bindings.. 590 A.4.14 C++Bindings on all MPI Classes 591 A.4.15 Construction Destruction.... 591 A.4.16Copy/Assignment...·.·. 591 A.4.17 Comparison..·..··· 591 A.4.18 Inter-language Operability 591 B Change-Log 593 B.1 Changes from Version 2.1 to Version 2.2 ... 593 B.2 Changes from Version 2.0 to Version 2.1 ... 596 Bibliography 601 Examples Index 605 MPI Constant and Predefined Handle Index 608 MPI Declarations Index 615 MPI Callback Function Prototype Index 617 MPI Function Index 618 xiv
A.3.9 One-Sided Communications Fortran Bindings . . . . . . . . . . . . . 563 A.3.10 External Interfaces Fortran Bindings . . . . . . . . . . . . . . . . . . 564 A.3.11 I/O Fortran Bindings . . . . . . . . . . . . . . . . . . . . . . . . . . 564 A.3.12 Language Bindings Fortran Bindings . . . . . . . . . . . . . . . . . . 568 A.3.13 Profiling Interface Fortran Bindings . . . . . . . . . . . . . . . . . . 569 A.3.14 Deprecated Fortran Bindings . . . . . . . . . . . . . . . . . . . . . . 569 A.4 C++ Bindings (deprecated) . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 A.4.1 Point-to-Point Communication C++ Bindings . . . . . . . . . . . . 571 A.4.2 Datatypes C++ Bindings . . . . . . . . . . . . . . . . . . . . . . . . 574 A.4.3 Collective Communication C++ Bindings . . . . . . . . . . . . . . . 576 A.4.4 Groups, Contexts, Communicators, and Caching C++ Bindings . . 577 A.4.5 Process Topologies C++ Bindings . . . . . . . . . . . . . . . . . . . 580 A.4.6 MPI Environmenta Management C++ Bindings . . . . . . . . . . . 582 A.4.7 The Info Object C++ Bindings . . . . . . . . . . . . . . . . . . . . . 583 A.4.8 Process Creation and Management C++ Bindings . . . . . . . . . . 584 A.4.9 One-Sided Communications C++ Bindings . . . . . . . . . . . . . . 585 A.4.10 External Interfaces C++ Bindings . . . . . . . . . . . . . . . . . . . 586 A.4.11 I/O C++ Bindings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586 A.4.12 Language Bindings C++ Bindings . . . . . . . . . . . . . . . . . . . 590 A.4.13 Profiling Interface C++ Bindings . . . . . . . . . . . . . . . . . . . . 590 A.4.14 C++ Bindings on all MPI Classes . . . . . . . . . . . . . . . . . . . 591 A.4.15 Construction / Destruction . . . . . . . . . . . . . . . . . . . . . . . 591 A.4.16 Copy / Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591 A.4.17 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591 A.4.18 Inter-language Operability . . . . . . . . . . . . . . . . . . . . . . . . 591 B Change-Log 593 B.1 Changes from Version 2.1 to Version 2.2 . . . . . . . . . . . . . . . . . . . . 593 B.2 Changes from Version 2.0 to Version 2.1 . . . . . . . . . . . . . . . . . . . . 596 Bibliography 601 Examples Index 605 MPI Constant and Predefined Handle Index 608 MPI Declarations Index 615 MPI Callback Function Prototype Index 617 MPI Function Index 618 xiv
List of Figures 5.1 Collective comminucations,an overview······· 132 5.2 Intercommunicator allgather.,.,.···.· 136 5.3 Intercommunicator reduce-scatter..... 136 5.4 Gather example....·.········ 143 5.5 Gatherv example with strides .. 144 5.6 Gatherv example,2-dimensional 145 5.7 Gatherv example,2-dimensional,subarrays with different sizes 146 5.8 Gatherv example,2-dimensional,subarrays with different sizes and strides 147 5.9 Scatter example...············ 152 5.l0 Scatterv example with strides.....·.........··.。·...... 153 5.11 Scatterv example with different strides and counts.·.····,,····. 154 5.12 Race conditions with point-to-point and collective communications .... 185 6.1 Intercommunicator create using MPI_COMM_CREATE 204 6.2 Intercommunicator constructionwith MPI_COMM_SPLIT...... 207 6.3 Three-group pipeline.······················· 221 6.4 Three-group ring:·.···.· 223 7.1 Set-up of process structure for two-dimensional parallel Poisson solver.... 270 11.1 Active target communication 349 l1l.2 Active target communication,with weak synchronization....··.··.. 350 l1.3 Passive target communication................·..··.··.. 351 ll.4 Active target communication with several processes.············· 355 11.5 Schematic description of window 364 ll.6 Symmetric communication.·····.··.······.····· 370 1l.7 Deadlock situation.....···. 370 1l.8 No deadlock.....······· 370 l3.1 Etypes and filetypes··..· 390 l3.2 Partitioning a file among parallel processes.....·..·.·... 390 l3.3 Displacements.·..·····························… 402 l3.4 Example array file layout.·..························ 450 13.5 Example local array filetype for process 1 .............. 451 XV
List of Figures 5.1 Collective comminucations, an overview . . . . . . . . . . . . . . . . . . . . 132 5.2 Intercommunicator allgather . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 5.3 Intercommunicator reduce-scatter . . . . . . . . . . . . . . . . . . . . . . . . 136 5.4 Gather example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 5.5 Gatherv example with strides . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5.6 Gatherv example, 2-dimensional . . . . . . . . . . . . . . . . . . . . . . . . 145 5.7 Gatherv example, 2-dimensional, subarrays with different sizes . . . . . . . 146 5.8 Gatherv example, 2-dimensional, subarrays with different sizes and strides . 147 5.9 Scatter example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.10 Scatterv example with strides . . . . . . . . . . . . . . . . . . . . . . . . . . 153 5.11 Scatterv example with different strides and counts . . . . . . . . . . . . . . 154 5.12 Race conditions with point-to-point and collective communications . . . . . 185 6.1 Intercommunicator create using MPI_COMM_CREATE . . . . . . . . . . . . 204 6.2 Intercommunicator constructionwith MPI_COMM_SPLIT . . . . . . . . . . . 207 6.3 Three-group pipeline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 6.4 Three-group ring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 7.1 Set-up of process structure for two-dimensional parallel Poisson solver. . . . 270 11.1 Active target communication . . . . . . . . . . . . . . . . . . . . . . . . . . 349 11.2 Active target communication, with weak synchronization . . . . . . . . . . . 350 11.3 Passive target communication . . . . . . . . . . . . . . . . . . . . . . . . . . 351 11.4 Active target communication with several processes . . . . . . . . . . . . . . 355 11.5 Schematic description of window . . . . . . . . . . . . . . . . . . . . . . . . 364 11.6 Symmetric communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 11.7 Deadlock situation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 11.8 No deadlock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 13.1 Etypes and filetypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 13.2 Partitioning a file among parallel processes . . . . . . . . . . . . . . . . . . 390 13.3 Displacements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 13.4 Example array file layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 13.5 Example local array filetype for process 1 . . . . . . . . . . . . . . . . . . . 451 xv
1 2 3 4 5 67 List of Tables 9 0 2.1 Deprecated constructs...··..·.··········· 17 12 13 3.1 Predefined MPI datatypes corresponding to Fortran datatypes 27 14 3.2 Predefined MPI datatypes corresponding to C datatypes.......... 28 15 3.3 Predefined MPI datatypes corresponding to both C and Fortran datatypes 29 4.1 combiner values returned from MPI_TYPE_GET_ENVELOPE 106 6.1 MPI_COMM_*Function Behavior (in Inter-Communication Mode) 218 9 20 8.1 Error classes (Part 1) 284 21 8.2 Error classes (Part 2) 285 22 23 11.1 Error classes in one-sided communication routines....... 363 24 13.1 Data access routines 404 25 l3.2“external:32”sizes of predefined datatypes.··.·············· 433 26 13.3 I/O Error Classes 448 2 16.1 C++names for the MPI C and C++predefined datatypes....... 。。 472 29 l6.2C++names for the MPI Fortran predefined datatypes,····· 472 30 l6.3C++names for other MPl datatypes.·······. 473 31 32 33 34 6 3 38 40 41 42 43 44 45 4 xvi
List of Tables 2.1 Deprecated constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.1 Predefined MPI datatypes corresponding to Fortran datatypes . . . . . . . 27 3.2 Predefined MPI datatypes corresponding to C datatypes . . . . . . . . . . 28 3.3 Predefined MPI datatypes corresponding to both C and Fortran datatypes 29 4.1 combiner values returned from MPI_TYPE_GET_ENVELOPE . . . . . . . . 106 6.1 MPI_COMM_* Function Behavior (in Inter-Communication Mode) . . . . . 218 8.1 Error classes (Part 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 8.2 Error classes (Part 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 11.1 Error classes in one-sided communication routines . . . . . . . . . . . . . . 363 13.1 Data access routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 13.2 “external32” sizes of predefined datatypes . . . . . . . . . . . . . . . . . . . 433 13.3 I/O Error Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448 16.1 C++ names for the MPI C and C++ predefined datatypes . . . . . . . . . . 472 16.2 C++ names for the MPI Fortran predefined datatypes . . . . . . . . . . . . 472 16.3 C++ names for other MPI datatypes . . . . . . . . . . . . . . . . . . . . . . 473 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 xvi
Acknowledgments 3 4 This document represents the work of many people who have served on the MPI Forum. 5 The meetings have been attended by dozens of people from many parts of the world.It is 6 the hard and dedicated work of this group that has led to the MPI standard. The technical development was carried out by subgroups,whose work was reviewed by the full committee.During the period of development of the Message-Passing Interface (MPI),many people helped with this effort. 10 Those who served as primary coordinators in MPl-1.0 and MPl-1.1 are: 之 12 Jack Dongarra,David Walker,Conveners and Meeting Chairs 13 14 Ewing Lusk,Bob Knighten,Minutes 公 16 Marc Snir,William Gropp,Ewing Lusk,Point-to-Point Communication 么 .Al Geist,Marc Snir,Steve Otto,Collective Communication 19 ·Steve Otto,Editor 20 Rolf Hempel,Process Topologies 总 22 Ewing Lusk,Language Binding 23 24 William Gropp,Environmental Management 25 James Cownie,Profiling 36 Tony Skjellum,Lyndon Clarke,Marc Snir,Richard Littlefield,Mark Sears,Groups, 导 Contexts,and Communicators 喝 Steven Huss-Lederman,Initial Implementation Subset 31 32 The following list includes some of the active participants in the MPl-1.0 and MPl-1.1 33 process not mentioned above. 34 客 Ed Anderson Robert Babb Joe Baron Eric Barszcz 36 Scott Berryman Rob Bjornson Nathan Doss Anne Elster 37 Jim Feeney Vince Fernando Sam Fineberg Jon Flower Daniel Frye Ian Glendinning Adam Greenberg Robert Harrison Leslie Hart Tom Haupt Don Heller Tom Henderson % Alex Ho C.T.Howard Ho Gary Howell John Kapenga 41 James Kohl Susan Krauss Bob Leary Arthur Maccabe 42 Peter Madams Alan Mainwaring Oliver McBryan Phil McKinley 43 Charles Mosher Dan Nessett Peter Pacheco Howard Palmer 44 Paul Pierce Sanjay Ranka Peter Rigsbee Arch Robison Erich Schikuta Ambuj Singh Alan Sussman Robert Tomlinson 6 Robert G.Voigt Dennis Weeks Stephen Wheat Steve Zenith xvii
Acknowledgments This document represents the work of many people who have served on the MPI Forum. The meetings have been attended by dozens of people from many parts of the world. It is the hard and dedicated work of this group that has led to the MPI standard. The technical development was carried out by subgroups, whose work was reviewed by the full committee. During the period of development of the Message-Passing Interface (MPI), many people helped with this effort. Those who served as primary coordinators in MPI-1.0 and MPI-1.1 are: • Jack Dongarra, David Walker, Conveners and Meeting Chairs • Ewing Lusk, Bob Knighten, Minutes • Marc Snir, William Gropp, Ewing Lusk, Point-to-Point Communication • Al Geist, Marc Snir, Steve Otto, Collective Communication • Steve Otto, Editor • Rolf Hempel, Process Topologies • Ewing Lusk, Language Binding • William Gropp, Environmental Management • James Cownie, Profiling • Tony Skjellum, Lyndon Clarke, Marc Snir, Richard Littlefield, Mark Sears, Groups, Contexts, and Communicators • Steven Huss-Lederman, Initial Implementation Subset The following list includes some of the active participants in the MPI-1.0 and MPI-1.1 process not mentioned above. Ed Anderson Robert Babb Joe Baron Eric Barszcz Scott Berryman Rob Bjornson Nathan Doss Anne Elster Jim Feeney Vince Fernando Sam Fineberg Jon Flower Daniel Frye Ian Glendinning Adam Greenberg Robert Harrison Leslie Hart Tom Haupt Don Heller Tom Henderson Alex Ho C.T. Howard Ho Gary Howell John Kapenga James Kohl Susan Krauss Bob Leary Arthur Maccabe Peter Madams Alan Mainwaring Oliver McBryan Phil McKinley Charles Mosher Dan Nessett Peter Pacheco Howard Palmer Paul Pierce Sanjay Ranka Peter Rigsbee Arch Robison Erich Schikuta Ambuj Singh Alan Sussman Robert Tomlinson Robert G. Voigt Dennis Weeks Stephen Wheat Steve Zenith xvii 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
The University of Tennessee and Oak Ridge National Laboratory made the draft avail- able by anonymous FTP mail servers and were instrumental in distributing the document. The work on the MPI-1 standard was supported in part by ARPA and NSF under grant ASC-9310330,the National Science Foundation Science and Technology Center Cooperative 5 Agreement No.CCR-8809615,and by the Commission of the European Community through Esprit project P6643(PPPE) 9 MPI-1.2 and MPI-2.0: 10 11 Those who served as primary coordinators in MPl-1.2 and MPI-2.0 are: 13 Ewing Lusk,Convener and Meeting Chair 14 Steve Huss-Lederman,Editor 15 16 Ewing Lusk,Miscellany 18 Bill Saphir,Process Creation and Management 19 Marc Snir,One-Sided Communications 20 21 Bill Gropp and Anthony Skjellum,Extended Collective Operations 22 23 Steve Huss-Lederman,External Interfaces 24 ·Bill Nitzberg,I/O 26 Andrew Lumsdaine,Bill Saphir,and Jeff Squyres,Language Bindings 今 28 Anthony Skjellum and Arkady Kanevsky,Real-Time 29 30 The following list includes some of the active participants who attended MPl-2 Forum meetings and are not mentioned above. 31 32 Greg Astfalk Robert Babb Ed Benson Rajesh Bordawekar 33 Pete Bradley Peter Brennan Ron Brightwell Maciej Brodowicz 34 Eric Brunner Greg Burns Margaret Cahir Pang Chen Ying Chen Albert Cheng Yong Cho Joel Clark 36 Lyndon Clarke Laurie Costello Dennis Cottel Jim Cownie 37 Zhengian Cui Suresh Damodaran-Kamal Raja Daoud 38 Judith Devaney David DiNucci Doug Doefler Jack Dongarra 39 Terry Dontje Nathan Doss Anne Elster Mark Fallon 40 Karl Feind Sam Fineberg Craig Fischberg Stephen Fleischman 41 Ian Foster Hubertus Franke Richard Frost Al Geist 42 Robert George David Greenberg John Hagedorn Kei Harada 43 Leslie Hart Shane Hebert Rolf Hempel Tom Henderson 44 Alex Ho Hans-Christian Hoppe Joefon ,Jann Terry Jones 45 Karl Kesselman Koichi Konishi Susan Kraus Steve Kubica 46 Steve Landherr Mario Lauria Mark Law Juan Leon 47 Lloyd Lewins Ziyang Lu Bob Madahar Peter Madams 48 xviii
The University of Tennessee and Oak Ridge National Laboratory made the draft available by anonymous FTP mail servers and were instrumental in distributing the document. The work on the MPI-1 standard was supported in part by ARPA and NSF under grant ASC-9310330, the National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615, and by the Commission of the European Community through Esprit project P6643 (PPPE). MPI-1.2 and MPI-2.0: Those who served as primary coordinators in MPI-1.2 and MPI-2.0 are: • Ewing Lusk, Convener and Meeting Chair • Steve Huss-Lederman, Editor • Ewing Lusk, Miscellany • Bill Saphir, Process Creation and Management • Marc Snir, One-Sided Communications • Bill Gropp and Anthony Skjellum, Extended Collective Operations • Steve Huss-Lederman, External Interfaces • Bill Nitzberg, I/O • Andrew Lumsdaine, Bill Saphir, and Jeff Squyres, Language Bindings • Anthony Skjellum and Arkady Kanevsky, Real-Time The following list includes some of the active participants who attended MPI-2 Forum meetings and are not mentioned above. Greg Astfalk Robert Babb Ed Benson Rajesh Bordawekar Pete Bradley Peter Brennan Ron Brightwell Maciej Brodowicz Eric Brunner Greg Burns Margaret Cahir Pang Chen Ying Chen Albert Cheng Yong Cho Joel Clark Lyndon Clarke Laurie Costello Dennis Cottel Jim Cownie Zhenqian Cui Suresh Damodaran-Kamal Raja Daoud Judith Devaney David DiNucci Doug Doefler Jack Dongarra Terry Dontje Nathan Doss Anne Elster Mark Fallon Karl Feind Sam Fineberg Craig Fischberg Stephen Fleischman Ian Foster Hubertus Franke Richard Frost Al Geist Robert George David Greenberg John Hagedorn Kei Harada Leslie Hart Shane Hebert Rolf Hempel Tom Henderson Alex Ho Hans-Christian Hoppe Joefon Jann Terry Jones Karl Kesselman Koichi Konishi Susan Kraus Steve Kubica Steve Landherr Mario Lauria Mark Law Juan Leon Lloyd Lewins Ziyang Lu Bob Madahar Peter Madams 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 xviii