MPI:A Message-Passing Interface Standard Version 2.2 Message Passing Interface Forum September 4,2009
MPI: A Message-Passing Interface Standard Version 2.2 Message Passing Interface Forum September 4, 2009
Contents Acknowledgments viii 1 Introduction to MPl 1 1.1 Overview and Goals 1 1.2 Background of MPI-1.0........... 2 1.3 Background of MP-1.1,MP1.2,and MPI--2.0.·······.·······- 3 1.4 Background of MP-1.3 and MP-2.1....·················- 3 1.5 Background of MPl-2.2............................. 4 1.6 Who Should Use This Standard?.......... 4 1.7 What Platforms Are Targets For Implementation? 4 1.8 What Is Included In The Standard? 5 1.9 What Is Not Included In The Standard? 6 l.l0 Organization of this Document···.··.················· 6 2 MPI Terms and Conventions 9 2.1 Document Notation 9 2.2 Naming Conventions.·· 9 2.3 Procedure Specification 10 2.4 Semantic Terms.··. 11 2.5 Data Types 12 2.5.1 Opaque Objects 12 2.5.2 Array Arguments 14 2.5.3 State.......。·············“ 14 2.5.4 Named Constants 14 2.5.5 Choice 15 2.5.6 Addresses 。 15 2.5.7 File Offsets 。。。 16 2.6 Language Binding 16 2.6.1 Deprecated Names and Functions.... 16 2.6.2 Fortran Binding Issues..·..··. 17 2.6.3 C Binding Issues 18 2.6.4 C++Binding Issues 18 2.6.5 Functions and Macros 2.7 Processes 22 2.8 Error Handling..·············· 2.9 Implementation Issues... 223 2.9.1 Independence of Basic Runtime Routines............... 23
Contents Acknowledgments viii 1 Introduction to MPI 1 1.1 Overview and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Background of MPI-1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Background of MPI-1.1, MPI-1.2, and MPI-2.0 . . . . . . . . . . . . . . . . . 3 1.4 Background of MPI-1.3 and MPI-2.1 . . . . . . . . . . . . . . . . . . . . . . 3 1.5 Background of MPI-2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.6 Who Should Use This Standard? . . . . . . . . . . . . . . . . . . . . . . . . 4 1.7 What Platforms Are Targets For Implementation? . . . . . . . . . . . . . . 4 1.8 What Is Included In The Standard? . . . . . . . . . . . . . . . . . . . . . . 5 1.9 What Is Not Included In The Standard? . . . . . . . . . . . . . . . . . . . . 6 1.10 Organization of this Document . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 MPI Terms and Conventions 9 2.1 Document Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Procedure Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.4 Semantic Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.5 Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1 Opaque Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.2 Array Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.3 State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.4 Named Constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.5 Choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5.6 Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5.7 File Offsets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6 Language Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6.1 Deprecated Names and Functions . . . . . . . . . . . . . . . . . . . . 16 2.6.2 Fortran Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.6.3 C Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.6.4 C++ Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.6.5 Functions and Macros . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.7 Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.8 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.9 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.9.1 Independence of Basic Runtime Routines . . . . . . . . . . . . . . . 23 v
2.9.2 nteraction with Signals.························ 24 2.l0 Examples··································· 24 3 Point-to-Point Communication 25 3.1 Introduction..·.........:......·..······· 25 3.2 Blocking Send and Receive Operations.............. 26 3.2.1 Blocking Send 26 3.2.2 Message Data.·.················ 27 3.2.3 Message Envelope 29 3.2.4 Blocking Receive 3 3.2.5 Return Status..·. 31 3.2.6 Passing MPI_STATUS_IGNORE for Status 3.3 Data Type Matching and Data Conversion 34 3.3.1 Type Matching Rules 34 Type MPI_CHARACTER........ 。 36 3.3.2 Data Conversion...·.···· 37 3.4 Communication Modes.. 38 3.5 Semantics of Point-to-Point Communication 42 3.6 Buffer Allocation and Usage... 45 3.6.1 Model Implementation of Buffered Mode... 47 3.7 Nonblocking Communication 48 3.7.1 Communication Request Objects ...... 西 3.7.2 Communication Initiation ......... 50 3.7.3 Communication Completion..·..··..···· 53 3.7.4 Semantics of Nonblocking Communications 56 3.7.5 Multiple Completions.................. 3.7.6 Non-destructive Test of status.....·.······, 。。。 64 3.8 Probe and Cancel... 3.9 Persistent Communication Requests 69 3.10 Send-Receive 73 3.11 Null Processes 75 4 Datatypes 77 4.1 Derived Datatypes ... 4.l.1 Type Constructors with Explicit Addresses·..··.··...... 79 4.l.2 Datatype Constructors.···················,····· 79 4.l.3 Subarray Datatype Constructor..····· 87 4.1.4 Distributed Array Datatype Constructor. 89 4.1.5 Address and Size Functions 94 4.1.6 Lower-Bound and Upper-Bound Markers 96 4.l.7 Extent and Bounds of Datatypes·..·.. 9 4.1.8 True Extent of Datatypes.......... 98 4.l.9 Commit and Free.···········- 99 4.1.10 Duplicating a Datatype.... 100 4.1.11 Use of General Datatypes in Communication 101 4.1.12 Correct Use of Addresses 104 4.l.l3 Decoding a Datatype...··················… 104 4.1.14 Examples......········ 112
2.9.2 Interaction with Signals . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.10 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3 Point-to-Point Communication 25 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2 Blocking Send and Receive Operations . . . . . . . . . . . . . . . . . . . . . 26 3.2.1 Blocking Send . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2.2 Message Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.3 Message Envelope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2.4 Blocking Receive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.2.5 Return Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2.6 Passing MPI_STATUS_IGNORE for Status . . . . . . . . . . . . . . . . 33 3.3 Data Type Matching and Data Conversion . . . . . . . . . . . . . . . . . . 34 3.3.1 Type Matching Rules . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Type MPI_CHARACTER . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.3.2 Data Conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.4 Communication Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.5 Semantics of Point-to-Point Communication . . . . . . . . . . . . . . . . . . 42 3.6 Buffer Allocation and Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.6.1 Model Implementation of Buffered Mode . . . . . . . . . . . . . . . . 47 3.7 Nonblocking Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.7.1 Communication Request Objects . . . . . . . . . . . . . . . . . . . . 49 3.7.2 Communication Initiation . . . . . . . . . . . . . . . . . . . . . . . . 50 3.7.3 Communication Completion . . . . . . . . . . . . . . . . . . . . . . . 53 3.7.4 Semantics of Nonblocking Communications . . . . . . . . . . . . . . 56 3.7.5 Multiple Completions . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.7.6 Non-destructive Test of status . . . . . . . . . . . . . . . . . . . . . . 64 3.8 Probe and Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.9 Persistent Communication Requests . . . . . . . . . . . . . . . . . . . . . . 69 3.10 Send-Receive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.11 Null Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4 Datatypes 77 4.1 Derived Datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.1.1 Type Constructors with Explicit Addresses . . . . . . . . . . . . . . 79 4.1.2 Datatype Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.1.3 Subarray Datatype Constructor . . . . . . . . . . . . . . . . . . . . . 87 4.1.4 Distributed Array Datatype Constructor . . . . . . . . . . . . . . . . 89 4.1.5 Address and Size Functions . . . . . . . . . . . . . . . . . . . . . . . 94 4.1.6 Lower-Bound and Upper-Bound Markers . . . . . . . . . . . . . . . 96 4.1.7 Extent and Bounds of Datatypes . . . . . . . . . . . . . . . . . . . . 97 4.1.8 True Extent of Datatypes . . . . . . . . . . . . . . . . . . . . . . . . 98 4.1.9 Commit and Free . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.1.10 Duplicating a Datatype . . . . . . . . . . . . . . . . . . . . . . . . . 100 4.1.11 Use of General Datatypes in Communication . . . . . . . . . . . . . 101 4.1.12 Correct Use of Addresses . . . . . . . . . . . . . . . . . . . . . . . . 104 4.1.13 Decoding a Datatype . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.1.14 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 vi
4.2 Pack and Unpack.. 121 4.3 Canonical MPI_PACK and MPI_UNPACK 127 5 Collective Communication 131 5.1 Introduction and Overview 。。。。 131 5.2 Communicator Argument.....·...·.··.······· 134 5.2.1 Specifics for Intracommunicator Collective Operations . 134 5.2.2 Applying Collective Operations to Intercommunicators... 134 5.2.3 Specifics for Intercommunicator Collective Operations 137 5.3 Barrier Synchronization..····················· 137 5.4 Broadcast·. 138 5.4.1 Example using MPI_BCAST.... 138 5.5 Gather.. 4 139 5.5.1 Examples using MPI_GATHER,MPI_GATHERV 142 5.6 Scatter..·..·· 149 5.6.1 Examples using MPI_SCATTER,MPI_SCATTERV 152 5.7 Gather-to-all 154 5.7.1 Exampleusing MPI_ALLGATHER 156 5.8 All-to-All Scatter/Gather...... 157 5.9 Global Reduction Operations ..... 162 5.9.1 Reduce·············· 163 5.9.2 Predefined Reduction Operations 164 5.9.3 Signed Characters and Reductions 167 5.9.4 MINLOC and MAXLOC..···· 167 5.9.5 User-Defined Reduction Operations 171 Example of User-defined Reduce 174 5.9.6 All-Reduce 175 5.9.7 Process-local reduction.······ 176 5.10 Reduce-Scatter .. 177 5.10.1 MPI_REDUCE_SCATTER_BLOCK 178 5.10.2 MPI_REDUCE_SCATTER.. 179 5.11Scan.. 180 5.11.1 Inclusive Scan........ 180 5.11.2 Exclusive Scan 181 5.11.3 Example using MPI_SCAN... 182 5.12 Correctness................. 183 6 Groups,Contexts,Communicators,and Caching 187 6.1 Introduction..··.·················· 187 6.1.1 Features Needed to Support Libraries 187 6.1.2 MPI's Support for Libraries 188 6.2 Basic Concepts 190 6.2.1 Groups 190 6.2.2 Contexts 190 6.2.3 Intra-Communicators.. 191 6.2.4 Predefined Intra-Communicators·.··········· 191 6.3 Group Management.························ 192 6.3.1 Group Accessors.·········· 192 i
4.2 Pack and Unpack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 4.3 Canonical MPI_PACK and MPI_UNPACK . . . . . . . . . . . . . . . . . . . 127 5 Collective Communication 131 5.1 Introduction and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 5.2 Communicator Argument . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 5.2.1 Specifics for Intracommunicator Collective Operations . . . . . . . . 134 5.2.2 Applying Collective Operations to Intercommunicators . . . . . . . . 134 5.2.3 Specifics for Intercommunicator Collective Operations . . . . . . . . 137 5.3 Barrier Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 5.4 Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 5.4.1 Example using MPI_BCAST . . . . . . . . . . . . . . . . . . . . . . . 138 5.5 Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 5.5.1 Examples using MPI_GATHER, MPI_GATHERV . . . . . . . . . . . . 142 5.6 Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 5.6.1 Examples using MPI_SCATTER, MPI_SCATTERV . . . . . . . . . . 152 5.7 Gather-to-all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 5.7.1 Exampleusing MPI_ALLGATHER . . . . . . . . . . . . . . . . . . . . 156 5.8 All-to-All Scatter/Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 5.9 Global Reduction Operations . . . . . . . . . . . . . . . . . . . . . . . . . . 162 5.9.1 Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.9.2 Predefined Reduction Operations . . . . . . . . . . . . . . . . . . . . 164 5.9.3 Signed Characters and Reductions . . . . . . . . . . . . . . . . . . . 167 5.9.4 MINLOC and MAXLOC . . . . . . . . . . . . . . . . . . . . . . . . 167 5.9.5 User-Defined Reduction Operations . . . . . . . . . . . . . . . . . . 171 Example of User-defined Reduce . . . . . . . . . . . . . . . . . . . . 174 5.9.6 All-Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 5.9.7 Process-local reduction . . . . . . . . . . . . . . . . . . . . . . . . . . 176 5.10 Reduce-Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 5.10.1 MPI_REDUCE_SCATTER_BLOCK . . . . . . . . . . . . . . . . . . . 178 5.10.2 MPI_REDUCE_SCATTER . . . . . . . . . . . . . . . . . . . . . . . . 179 5.11 Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 5.11.1 Inclusive Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 5.11.2 Exclusive Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 5.11.3 Example using MPI_SCAN . . . . . . . . . . . . . . . . . . . . . . . . 182 5.12 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 6 Groups, Contexts, Communicators, and Caching 187 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 6.1.1 Features Needed to Support Libraries . . . . . . . . . . . . . . . . . 187 6.1.2 MPI’s Support for Libraries . . . . . . . . . . . . . . . . . . . . . . . 188 6.2 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 6.2.1 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 6.2.2 Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 6.2.3 Intra-Communicators . . . . . . . . . . . . . . . . . . . . . . . . . . 191 6.2.4 Predefined Intra-Communicators . . . . . . . . . . . . . . . . . . . . 191 6.3 Group Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 6.3.1 Group Accessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 vii
6.3.2 Group Constructors······················· 193 6.3.3 Group Destructors······················· 198 6.4 Communicator Management·············· 199 6.4.1 Communicator Accessors 199 6.4.2 Communicator Constructors.... 200 6.4.3 Communicator Destructors 208 6.5 Motivating Examples......... 209 6.5.1 Current Practice #1 209 6.5.2 Current Practice #2... 210 6.5.3 (Approximate)Current Practice #3 210 6.5.4 Example#4.... 211 6.5.5 Library Example #1 212 6.5.6 Library Example #2 213 6.6 Inter-Communication...... 216 6.6.1 Inter-communicator Accessors... 217 6.6.2 Inter-communicator Operations 219 6.6.3 Inter-Communication Examples .. 221 Example 1:Three-Group“Pipeline'” 221 Example2:Three-Group“Ring”... 222 6.7 Caching. 224 6.7.1 Functionalitv 225 6.7.2 Communicators..····· 226 6.7.3 Windows 230 6.7.4 Datatypes 233 6.7.5 Error Class for Invalid Keyval 236 6.7.6 Attributes Example 236 6.8 Naming Objects 238 6.9 Formalizing the Loosely Synchronous Model.. 242 6.9.1 Basic Statements . 242 6.9.2 Models of Execution· 242 Static communicator allocation . 243 Dynamic communicator allocation.........····. 243 The General case..·························· 243 7 Process Topologies 245 7.1 Introduction..... 245 7.2 Virtual Topologies 246 7.3 Embedding in MPI 246 7.4 Overview of the Functions 246 7.5 Topology Constructors·..··.··········· 248 7.5.1 Cartesian Constructor ..... 248 7.5.2 Cartesian Convenience Function:MPI_DIMS_CREATE 248 7.5.3 General (Graph)Constructor ..................... 250 7.5.4 Distributed (Graph)Constructor ........... 252 7.5.5 Topology Inquiry Functions·············· 257 7.5.6 Cartesian Shift Coordinates .. 265 7.5.7 Partitioning of Cartesian structures.·...·.··.··. 266 7.5.8 Low-Level Topology Functions 267 viii
6.3.2 Group Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 6.3.3 Group Destructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 6.4 Communicator Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 6.4.1 Communicator Accessors . . . . . . . . . . . . . . . . . . . . . . . . 199 6.4.2 Communicator Constructors . . . . . . . . . . . . . . . . . . . . . . . 200 6.4.3 Communicator Destructors . . . . . . . . . . . . . . . . . . . . . . . 208 6.5 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 6.5.1 Current Practice #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 6.5.2 Current Practice #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 6.5.3 (Approximate) Current Practice #3 . . . . . . . . . . . . . . . . . . 210 6.5.4 Example #4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 6.5.5 Library Example #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 6.5.6 Library Example #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 6.6 Inter-Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 6.6.1 Inter-communicator Accessors . . . . . . . . . . . . . . . . . . . . . . 217 6.6.2 Inter-communicator Operations . . . . . . . . . . . . . . . . . . . . . 219 6.6.3 Inter-Communication Examples . . . . . . . . . . . . . . . . . . . . . 221 Example 1: Three-Group “Pipeline” . . . . . . . . . . . . . . . . . . 221 Example 2: Three-Group “Ring” . . . . . . . . . . . . . . . . . . . . 222 6.7 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 6.7.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 6.7.2 Communicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 6.7.3 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 6.7.4 Datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 6.7.5 Error Class for Invalid Keyval . . . . . . . . . . . . . . . . . . . . . . 236 6.7.6 Attributes Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 6.8 Naming Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 6.9 Formalizing the Loosely Synchronous Model . . . . . . . . . . . . . . . . . . 242 6.9.1 Basic Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 6.9.2 Models of Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Static communicator allocation . . . . . . . . . . . . . . . . . . . . . 243 Dynamic communicator allocation . . . . . . . . . . . . . . . . . . . 243 The General case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 7 Process Topologies 245 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 7.2 Virtual Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 7.3 Embedding in MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 7.4 Overview of the Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 7.5 Topology Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 7.5.1 Cartesian Constructor . . . . . . . . . . . . . . . . . . . . . . . . . . 248 7.5.2 Cartesian Convenience Function: MPI_DIMS_CREATE . . . . . . . . 248 7.5.3 General (Graph) Constructor . . . . . . . . . . . . . . . . . . . . . . 250 7.5.4 Distributed (Graph) Constructor . . . . . . . . . . . . . . . . . . . . 252 7.5.5 Topology Inquiry Functions . . . . . . . . . . . . . . . . . . . . . . . 257 7.5.6 Cartesian Shift Coordinates . . . . . . . . . . . . . . . . . . . . . . . 265 7.5.7 Partitioning of Cartesian structures . . . . . . . . . . . . . . . . . . 266 7.5.8 Low-Level Topology Functions . . . . . . . . . . . . . . . . . . . . . 267 viii