1 Argonne National Laboratory 2 Auburn University 3 Bull Cisco Systems,Inc. Cray Inc. 6 Forschungszentrum Jiilich Fujitsu The HDF Group Hewlett-Packard 10 International Business Machines 11 Indiana University Institut National de Recherche en Informatique et Automatique(INRIA) 13 Institute for Advanced Science Engineering Corporation 14 Lawrence Berkeley National Laboratory 15 Lawrence Livermore National Laboratory 16 Los Alamos National Laboratory 17 Mathworks 好 Mellanox Technologies 19 Microsoft 20 Myricom NEC Corporation 22 Oak Ridge National Laboratory 23 Ohio State University 24 Pacific Northwest National Laboratory 25 QLogic Corporation 26 RunTime Computing Solutions,LLC 27 Sandia National Laboratories 28 SiCortex,Inc. 29 Silicon Graphics Inc. 30 Sun Microsystems,Inc. 31 Tokyo Institute of Technology 32 University of Alabama at Birmingham 33 University of Houston 34 University of Illinois at Urbana-Champaign 35 University of Stuttgart,High Performance Computing Center Stuttgart (HLRS) 36 University of Tennessee,Knoxville 37 University of Tokyo University of Wisconsin 40 Funding for the MPI Forum meetings was partially supported by award #CCF-0816909 41 from the National Science Foundation. 42 In addition,the HDF Group provided travel support for one U.S.academic. 43 44 45 46 48 XXIV
Argonne National Laboratory Auburn University Bull Cisco Systems, Inc. Cray Inc. Forschungszentrum J¨ulich Fujitsu The HDF Group Hewlett-Packard International Business Machines Indiana University Institut National de Recherche en Informatique et Automatique (INRIA) Institute for Advanced Science & Engineering Corporation Lawrence Berkeley National Laboratory Lawrence Livermore National Laboratory Los Alamos National Laboratory Mathworks Mellanox Technologies Microsoft Myricom NEC Corporation Oak Ridge National Laboratory Ohio State University Pacific Northwest National Laboratory QLogic Corporation RunTime Computing Solutions, LLC Sandia National Laboratories SiCortex, Inc. Silicon Graphics Inc. Sun Microsystems, Inc. Tokyo Institute of Technology University of Alabama at Birmingham University of Houston University of Illinois at Urbana-Champaign University of Stuttgart, High Performance Computing Center Stuttgart (HLRS) University of Tennessee, Knoxville University of Tokyo University of Wisconsin Funding for the MPI Forum meetings was partially supported by award #CCF-0816909 from the National Science Foundation. In addition, the HDF Group provided travel support for one U.S. academic. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 xxiv
3 5 Chapter 1 6 Introduction to MPl 10 之 13 1.1 Overview and Goals 14 15 MPI(Message-Passing Interface)is a message-passing library interface specification.All parts of this definition are significant.MPI addresses primarily the message-passing parallel 17 18 programming model,in which data is moved from the address space of one process to that of another process through cooperative operations on each process.(Extensions to the 20 "classical"message-passing model are provided in collective operations,remote-memory access operations,dynamic process creation,and parallel I/O.)MPI is a specification,not 日 an implementation;there are multiple implementations of MPI.This specification is for a library interface;MPI is not a language,and all MPI operations are expressed as functions, 23 subroutines,or methods,according to the appropriate language bindings,which for C, 24 C++,Fortran-77,and Fortran-95,are part of the MPI standard.The standard has been 25 defined through an open process by a community of parallel computing vendors,computer 26 scientists,and application developers.The next few sections provide an overview of the 27 history of MPI's development. The main advantages of establishing a message-passing standard are portability and 多 ease of use.In a distributed memory communication environment in which the higher level 吃 routines and/or abstractions are built upon lower level message-passing routines the benefits 呢 of standardization are particularly apparent.Furthermore,the definition of a message- 32 passing standard,such as that proposed here,provides vendors with a clearly defined base 33 set of routines that they can implement efficiently,or in some cases provide hardware support 34 for,thereby enhancing scalability. The goal of the Message-Passing Interface simply stated is to develop a widely used 37 standard for writing message-passing programs.As such the interface should establish a practical,portable,efficient,and flexible standard for message passing. 38 A complete list of goals follows. 吃 S Design an application programming interface(not necessarily for compilers or a system 41 implementation library). 42 43 Allow efficient communication:Avoid memory-to-memory copying,allow overlap of 44 computation and communication,and offload to communication co-processor,where available. 46 47 Allow for implementations that can be used in a heterogeneous environment. 48 1
Chapter 1 Introduction to MPI 1.1 Overview and Goals MPI (Message-Passing Interface) is a message-passing library interface specification. All parts of this definition are significant. MPI addresses primarily the message-passing parallel programming model, in which data is moved from the address space of one process to that of another process through cooperative operations on each process. (Extensions to the “classical” message-passing model are provided in collective operations, remote-memory access operations, dynamic process creation, and parallel I/O.) MPI is a specification, not an implementation; there are multiple implementations of MPI. This specification is for a library interface; MPI is not a language, and all MPI operations are expressed as functions, subroutines, or methods, according to the appropriate language bindings, which for C, C++, Fortran-77, and Fortran-95, are part of the MPI standard. The standard has been defined through an open process by a community of parallel computing vendors, computer scientists, and application developers. The next few sections provide an overview of the history of MPI’s development. The main advantages of establishing a message-passing standard are portability and ease of use. In a distributed memory communication environment in which the higher level routines and/or abstractions are built upon lower level message-passing routines the benefits of standardization are particularly apparent. Furthermore, the definition of a messagepassing standard, such as that proposed here, provides vendors with a clearly defined base set of routines that they can implement efficiently, or in some cases provide hardware support for, thereby enhancing scalability. The goal of the Message-Passing Interface simply stated is to develop a widely used standard for writing message-passing programs. As such the interface should establish a practical, portable, efficient, and flexible standard for message passing. A complete list of goals follows. • Design an application programming interface (not necessarily for compilers or a system implementation library). • Allow efficient communication: Avoid memory-to-memory copying, allow overlap of computation and communication, and offload to communication co-processor, where available. • Allow for implementations that can be used in a heterogeneous environment. 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
CHAPTER 1.INTRODUCTION TO MPI 1 .Allow convenient C,C++,Fortran-77,and Fortran-95 bindings for the interface. 2 3 Assume a reliable communication interface:the user need not cope with communica- tion failures.Such failures are dealt with by the underlying communication subsystem. Define an interface that can be implemented on many vendor's platforms,with no significant changes in the underlying communication and system software. 8 Semantics of the interface should be language independent 9 g The interface should be designed to allow for thread safety. 1.2 Background of MPl-1.0 13 14 MPl sought to make use of the most attractive features of a number of existing message- 15 passing systems,rather than selecting one of them and adopting it as the standard.Thus, 16 MPI was strongly influenced by work at the IBM T.J.Watson Research Center [1,2],Intel's 17 NX/2 [38],Express [12],nCUBE's Vertex [34],p4 [7,8],and PARMACS [5,9].Other important contributions have come from Zipcode [40,41],Chimp [16,17],PVM [4,14], 19 Chameleon [25],and PICL [24]. The MPI standardization effort involved about 60 people from 40 organizations mainly 21 from the United States and Europe.Most of the major vendors of concurrent computers 22 were involved in MPl,along with researchers from universities,government laboratories,and 23 industry.The standardization process began with the Workshop on Standards for Message- 24 Passing in a Distributed Memory Environment,sponsored by the Center for Research on 25 Parallel Computing,held April 29-30,1992,in Williamsburg,Virginia [48].At this workshop 26 the basic features essential to a standard message-passing interface were discussed,and a 27 working group established to continue the standardization process. 28 A preliminary draft proposal,known as MPl1,was put forward by Dongarra,Hempel, Hey,and Walker in November 1992,and a revised version was completed in February 30 1993 [15].MPIl embodied the main features that were identified at the Williamsburg 31 workshop as being necessary in a message passing standard.Since MPIl was primarily 32 intended to promote discussion and "get the ball rolling,"it focused mainly on point-to-point 33 communications.MPIl brought to the forefront a number of important standardization 34 issues,but did not include any collective communication routines and was not thread-safe. 35 In November 1992,a meeting of the MPI working group was held in Minneapolis,at 36 which it was decided to place the standardization process on a more formal footing,and to 37 generally adopt the procedures and organization of the High Performance Fortran Forum. 38 Subcommittees were formed for the major component areas of the standard,and an email 39 discussion service established for each.In addition,the goal of producing a draft MPl 40 standard by the Fall of 1993 was set.To achieve this goal the MPl working group met every 41 6 weeks for two days throughout the first 9 months of 1993,and presented the draft MPI 42 standard at the Supercomputing 93 conference in November 1993.These meetings and the 43 email discussion together constituted the MPl Forum,membership of which has been open 44 to all members of the high performance computing community. 45 46 48
2 CHAPTER 1. INTRODUCTION TO MPI • Allow convenient C, C++, Fortran-77, and Fortran-95 bindings for the interface. • Assume a reliable communication interface: the user need not cope with communication failures. Such failures are dealt with by the underlying communication subsystem. • Define an interface that can be implemented on many vendor’s platforms, with no significant changes in the underlying communication and system software. • Semantics of the interface should be language independent. • The interface should be designed to allow for thread safety. 1.2 Background of MPI-1.0 MPI sought to make use of the most attractive features of a number of existing messagepassing systems, rather than selecting one of them and adopting it as the standard. Thus, MPI was strongly influenced by work at the IBM T. J. Watson Research Center [1, 2], Intel’s NX/2 [38], Express [12], nCUBE’s Vertex [34], p4 [7, 8], and PARMACS [5, 9]. Other important contributions have come from Zipcode [40, 41], Chimp [16, 17], PVM [4, 14], Chameleon [25], and PICL [24]. The MPI standardization effort involved about 60 people from 40 organizations mainly from the United States and Europe. Most of the major vendors of concurrent computers were involved in MPI, along with researchers from universities, government laboratories, and industry. The standardization process began with the Workshop on Standards for MessagePassing in a Distributed Memory Environment, sponsored by the Center for Research on Parallel Computing, held April 29-30, 1992, in Williamsburg, Virginia [48]. At this workshop the basic features essential to a standard message-passing interface were discussed, and a working group established to continue the standardization process. A preliminary draft proposal, known as MPI1, was put forward by Dongarra, Hempel, Hey, and Walker in November 1992, and a revised version was completed in February 1993 [15]. MPI1 embodied the main features that were identified at the Williamsburg workshop as being necessary in a message passing standard. Since MPI1 was primarily intended to promote discussion and “get the ball rolling,” it focused mainly on point-to-point communications. MPI1 brought to the forefront a number of important standardization issues, but did not include any collective communication routines and was not thread-safe. In November 1992, a meeting of the MPI working group was held in Minneapolis, at which it was decided to place the standardization process on a more formal footing, and to generally adopt the procedures and organization of the High Performance Fortran Forum. Subcommittees were formed for the major component areas of the standard, and an email discussion service established for each. In addition, the goal of producing a draft MPI standard by the Fall of 1993 was set. To achieve this goal the MPI working group met every 6 weeks for two days throughout the first 9 months of 1993, and presented the draft MPI standard at the Supercomputing 93 conference in November 1993. These meetings and the email discussion together constituted the MPI Forum, membership of which has been open to all members of the high performance computing community. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
1.3.BACKGROUND OF MPI-1.1,MPI-1.2,AND MPI-2.0 3 1.3 Background of MPI-1.1,MPI-1.2,and MPI-2.0 Beginning in March 1995,the MPl Forum began meeting to consider corrections and exten- 3 sions to the original MPI Standard document [21].The first product of these deliberations was Version 1.1 of the MPI specification,released in June of 1995 [22](see 5 http://www.mpi-forum.org for official MPI document releases). At that time.effort 6 focused in five areas. 1.Further corrections and clarifications for the MPl-1.1 document. 2.Additions to MPl-1.1 that do not significantly change its types of functionality (new 10 datatype constructors,language interoperability,etc.). 之 3.Completely new types of functionality (dynamic processes,one-sided communication, 13 parallel I/O,etc.)that are what everyone thinks of as "MPl-2 functionality." 14 4.Bindings for Fortran 90 and C++.MPI-2 specifies C++bindings for both MPl-1 and MPl-2 functions,and extensions to the Fortran 77 binding of MPl-1 and MPI-2 17 to handle Fortran 90 issues. 18 5.Discussions of areas in which the MPl process and framework seem likely to be useful, 19 but where more discussion and experience are needed before standardization (e.g. zero-copy semantics on shared-memory machines,real-time specifications). 品 Corrections and clarifications (items of type 1 in the above list)were collected in Chap- 23 ter 3 of the MPl-2 document:"Version 1.2 of MPI."That chapter also contains the function 24 for identifying the version number.Additions to MPl-1.1(items of types 2,3,and 4 in the 25 above list)are in the remaining chapters of the MPl-2 document,and constitute the specifi- 26 cation for MPl-2.Items of type 5 in the above list have been moved to a separate document, 27 the "MPI Journal of Development"(JOD),and are not part of the MPl-2 Standard. 导 This structure makes it easy for users and implementors to understand what level of 29 MPI compliance a given implementation has: 吃 MPI-1 compliance will mean compliance with MPl-1.3.This is a useful level of com- 9 32 pliance.It means that the implementation conforms to the clarifications of MPl-1.1 33 function behavior given in Chapter 3 of the MPl-2 document.Some implementations 34 may require changes to be MPl-1 compliant. 35 MPI-2 compliance will mean compliance with all of MPl-2.1. 36 37 The MPI Journal of Development is not part of the MPI Standard. It is to be emphasized that forward compatibility is preserved.That is,a valid MPl-1.1 % program is both a valid MPl-1.3 program and a valid MPl-2.1 program,and a valid MPl-1.3 41 program is a valid MPl-2.1 program. 42 43 1.4 Background of MPI-1.3 and MPI-2.1 44 飞 After the release of MPl-2.0,the MPI Forum kept working on errata and clarifications for 46 both standard documents(MPI-1.1 and MPI-2.0).The short document "Errata for MPI-1.1" 47 was released October 12,1998.On July 5,2001,a first ballot of errata and clarifications for
1.3. BACKGROUND OF MPI-1.1, MPI-1.2, AND MPI-2.0 3 1.3 Background of MPI-1.1, MPI-1.2, and MPI-2.0 Beginning in March 1995, the MPI Forum began meeting to consider corrections and extensions to the original MPI Standard document [21]. The first product of these deliberations was Version 1.1 of the MPI specification, released in June of 1995 [22] (see http://www.mpi-forum.org for official MPI document releases). At that time, effort focused in five areas. 1. Further corrections and clarifications for the MPI-1.1 document. 2. Additions to MPI-1.1 that do not significantly change its types of functionality (new datatype constructors, language interoperability, etc.). 3. Completely new types of functionality (dynamic processes, one-sided communication, parallel I/O, etc.) that are what everyone thinks of as “MPI-2 functionality.” 4. Bindings for Fortran 90 and C++. MPI-2 specifies C++ bindings for both MPI-1 and MPI-2 functions, and extensions to the Fortran 77 binding of MPI-1 and MPI-2 to handle Fortran 90 issues. 5. Discussions of areas in which the MPI process and framework seem likely to be useful, but where more discussion and experience are needed before standardization (e.g. zero-copy semantics on shared-memory machines, real-time specifications). Corrections and clarifications (items of type 1 in the above list) were collected in Chapter 3 of the MPI-2 document: “Version 1.2 of MPI.” That chapter also contains the function for identifying the version number. Additions to MPI-1.1 (items of types 2, 3, and 4 in the above list) are in the remaining chapters of the MPI-2 document, and constitute the specifi- cation for MPI-2. Items of type 5 in the above list have been moved to a separate document, the “MPI Journal of Development” (JOD), and are not part of the MPI-2 Standard. This structure makes it easy for users and implementors to understand what level of MPI compliance a given implementation has: • MPI-1 compliance will mean compliance with MPI-1.3. This is a useful level of compliance. It means that the implementation conforms to the clarifications of MPI-1.1 function behavior given in Chapter 3 of the MPI-2 document. Some implementations may require changes to be MPI-1 compliant. • MPI-2 compliance will mean compliance with all of MPI-2.1. • The MPI Journal of Development is not part of the MPI Standard. It is to be emphasized that forward compatibility is preserved. That is, a valid MPI-1.1 program is both a valid MPI-1.3 program and a valid MPI-2.1 program, and a valid MPI-1.3 program is a valid MPI-2.1 program. 1.4 Background of MPI-1.3 and MPI-2.1 After the release of MPI-2.0, the MPI Forum kept working on errata and clarifications for both standard documents (MPI-1.1 and MPI-2.0). The short document “Errata for MPI-1.1” was released October 12, 1998. On July 5, 2001, a first ballot of errata and clarifications for 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
4 CHAPTER 1.INTRODUCTION TO MPI 1 MPI-2.0 was released,and a second ballot was voted on May 22,2002.Both votes were done 2 electronically.Both ballots were combined into one document:"Errata for MPl-2",May 15,2002.This errata process was then interrupted,but the Forum and its e-mail reflectors 4 kept working on new requests for clarification. 5 Restarting regular work of the MPl Forum was initiated in three meetings,at Eu- roPVM/MPI'06 in Bonn,at EuroPVM/MPI'07 in Paris,and at SC'07 in Reno.In De- cember 2007,a steering committee started the organization of new MPI Forum meetings at regular 8-weeks intervals.At the January 14-16,2008 meeting in Chicago,the MPI Forum 9 decided to combine the existing and future MPl documents to one single document for each version of the MPl standard.For technical and historical reasons,this series was started 11 with MPl-1.3.Additional Ballots 3 and 4 solved old questions from the errata list started in 1995 up to new questions from the last years.After all documents (MPI-1.1,MPl-2, 13 Errata for MPl-1.1 (Oct.12,1998),and MPI-2.1 Ballots 1-4)were combined into one draft 14 document,for each chapter,a chapter author and review team were defined.They cleaned 15 up the document to achieve a consistent MPl-2.1 document.The final MPl-2.1 standard 16 document was finished in June 2008,and finally released with a second vote in September 17 2008 in the meeting at Dublin,just before EuroPVM/MPI'08.The major work of the 18 current MPI Forum is the preparation of MPl-3. 19 21 1.5 Background of MPl-2.2 22 MPl-2.2 is a minor update to the MPl-2.1 standard.This version addresses additional errors 23 and ambiguities that were not corrected in the MPl-2.1 standard as well as a small number 24 of extensions to MPl-2.1 that met the following criteria: 25 Any correct MPI-2.1 program is a correct MPI-2.2 program. 27 28 Any extension must have significant benefit for users. 9 Any extension must not require significant implementation effort.To that end,all 30 such changes are accompanied by an open source implementation. 31 32 The discussions of MPl-2.2 proceeded concurrently with the MPl-3 discussions;in some 33 cases,extensions were proposed for MPl-2.2 but were later moved to MPl-3. 34 35 1.6 Who Should Use This Standard? 36 37 This standard is intended for use by all those who want to write portable message-passing 38 programs in Fortran,C and C++.This includes individual application programmers,de- 39 velopers of software designed to run on parallel machines,and creators of environments 40 and tools.In order to be attractive to this wide audience,the standard must provide a 41 simple,easy-to-use interface for the basic user while not semantically precluding the high- 42 performance message-passing operations available on advanced machines. 43 44 45 1.7 What Platforms Are Targets For Implementation? 46 47 The attractiveness of the message-passing paradigm at least partially stems from its wide portability.Programs expressed this way may run on distributed-memory multiprocessors
4 CHAPTER 1. INTRODUCTION TO MPI MPI-2.0 was released, and a second ballot was voted on May 22, 2002. Both votes were done electronically. Both ballots were combined into one document: “Errata for MPI-2”, May 15, 2002. This errata process was then interrupted, but the Forum and its e-mail reflectors kept working on new requests for clarification. Restarting regular work of the MPI Forum was initiated in three meetings, at EuroPVM/MPI’06 in Bonn, at EuroPVM/MPI’07 in Paris, and at SC’07 in Reno. In December 2007, a steering committee started the organization of new MPI Forum meetings at regular 8-weeks intervals. At the January 14-16, 2008 meeting in Chicago, the MPI Forum decided to combine the existing and future MPI documents to one single document for each version of the MPI standard. For technical and historical reasons, this series was started with MPI-1.3. Additional Ballots 3 and 4 solved old questions from the errata list started in 1995 up to new questions from the last years. After all documents (MPI-1.1, MPI-2, Errata for MPI-1.1 (Oct. 12, 1998), and MPI-2.1 Ballots 1-4) were combined into one draft document, for each chapter, a chapter author and review team were defined. They cleaned up the document to achieve a consistent MPI-2.1 document. The final MPI-2.1 standard document was finished in June 2008, and finally released with a second vote in September 2008 in the meeting at Dublin, just before EuroPVM/MPI’08. The major work of the current MPI Forum is the preparation of MPI-3. 1.5 Background of MPI-2.2 MPI-2.2 is a minor update to the MPI-2.1 standard. This version addresses additional errors and ambiguities that were not corrected in the MPI-2.1 standard as well as a small number of extensions to MPI-2.1 that met the following criteria: • Any correct MPI-2.1 program is a correct MPI-2.2 program. • Any extension must have significant benefit for users. • Any extension must not require significant implementation effort. To that end, all such changes are accompanied by an open source implementation. The discussions of MPI-2.2 proceeded concurrently with the MPI-3 discussions; in some cases, extensions were proposed for MPI-2.2 but were later moved to MPI-3. 1.6 Who Should Use This Standard? This standard is intended for use by all those who want to write portable message-passing programs in Fortran, C and C++. This includes individual application programmers, developers of software designed to run on parallel machines, and creators of environments and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the highperformance message-passing operations available on advanced machines. 1.7 What Platforms Are Targets For Implementation? The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way may run on distributed-memory multiprocessors, 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48