2.6.1 Deprecated and Removed Interfaces 2.6.9 Fortran Binding Issues. 2.6.3 C Binding Iss 26.4 tions and Mac 2.7 28 Error Handling 2.0 2.9.1 of Basic Runtime 2.9.2 with Si 2.10 gnals Examples. 3 Point-to-Point Communication 31 3.1 Introduction 3.2 Blocking Send and Receive Operations 3.21 Blocking Send 322 Message Data 3.2.3 Message envelope 324 Blocking Receive 325 Return Status 3.2.6 Passing MPI_STATUS_IGNORE for Status 327 Blocking Send Receive 42 3.3 Datatype Matching and Data Conversion 3.3.1 Type Matchi ng rules Type MPI_CHARACTER 3.3.2 Data Conv sion 3.4 Co mication modes 3.5 Semantics of Point-to-Point Communication 3.6 Buffer Allocation and Usage 947 3.6.1 Model Implementation of Buffered Mod 3.7 Nonblocking Co 271 Co nication re Obi 379 3.7.3 37 tics of Nonblocking C nicati 3.75 Multiple Co 3.7.6 Dest ctive Tes 3.8 and Cancel 3.82 D 3.8.3 ched Receives 38 7188880090 3.9 ent on Null Processes Requests 3.10 4 Partitioned Point-to-Point Communication 103 4.1 Introduction 103 4.2 Semantics of Partitioned point-to-Point communication 104 421 Communication Initialization and Starting with Partitioning 106
2.6.1 Deprecated and Removed Interfaces . . . . . . . . . . . . . . . . . 23 2.6.2 Fortran Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.6.3 C Binding Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.6.4 Functions and Macros . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.7 Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.8 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.9 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.9.1 Independence of Basic Runtime Routines . . . . . . . . . . . . . . 28 2.9.2 Interaction with Signals . . . . . . . . . . . . . . . . . . . . . . . . 29 2.10 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3 Point-to-Point Communication 31 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2 Blocking Send and Receive Operations . . . . . . . . . . . . . . . . . . . . 32 3.2.1 Blocking Send . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2.2 Message Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2.3 Message Envelope . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.2.4 Blocking Receive . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.2.5 Return Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.2.6 Passing MPI_STATUS_IGNORE for Status . . . . . . . . . . . . . . . 41 3.2.7 Blocking Send-Receive . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.3 Datatype Matching and Data Conversion . . . . . . . . . . . . . . . . . . . 45 3.3.1 Type Matching Rules . . . . . . . . . . . . . . . . . . . . . . . . . 45 Type MPI_CHARACTER . . . . . . . . . . . . . . . . . . . . . . . . 47 3.3.2 Data Conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.4 Communication Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.5 Semantics of Point-to-Point Communication . . . . . . . . . . . . . . . . . 54 3.6 Buffer Allocation and Usage . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.6.1 Model Implementation of Buffered Mode . . . . . . . . . . . . . . 60 3.7 Nonblocking Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.7.1 Communication Request Objects . . . . . . . . . . . . . . . . . . . 62 3.7.2 Communication Initiation . . . . . . . . . . . . . . . . . . . . . . . 62 3.7.3 Communication Completion . . . . . . . . . . . . . . . . . . . . . . 70 3.7.4 Semantics of Nonblocking Communications . . . . . . . . . . . . . 74 3.7.5 Multiple Completions . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.7.6 Non-Destructive Test of status . . . . . . . . . . . . . . . . . . . . 83 3.8 Probe and Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 3.8.1 Probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 3.8.2 Matching Probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 3.8.3 Matched Receives . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.8.4 Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3.9 Persistent Communication Requests . . . . . . . . . . . . . . . . . . . . . . 94 3.10 Null Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4 Partitioned Point-to-Point Communication 103 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 4.2 Semantics of Partitioned Point-to-Point Communication . . . . . . . . . . 104 4.2.1 Communication Initialization and Starting with Partitioning . . . 106 vi
4.2.2 Communication Completion under Partitioning 110 4.2.3 Semantics of Communications in Partitioned Mode 111 4.3 Partitioned Communication Examples. 112 4.3.1 Partition Communication with Threads/Tasks Using OpenMP 4.0 or later 119 4.3.2 Send-only Partitioning Example with Tasks and OpenMP version 4 0 or later 11 4.3.3 Send and receive partitioning example with openmP version 4.0 or later 115 5 Datatypes 119 5.1 Derived Datatypes. 119 5.1. Type Constructors with Explicit Addresses 121 5.1.2 Datatype Constructors 。。。。。。，。。，。。。，，。。 121 5.13 Subarray Datatype Constructor 133 51.4 Distributed array datatype constructor 135 5.1.5 Address and Size Functions 141 5.1.6 Lower-Bound and Upper-Bound markers 144 5.1.7 Extent and bounds of Datatypes 147 5.1.8 True Extent of Datatypes 149 5.1.9 ●ommit and Free 150 5.1.10 Duplicating a Datatype 5111 Use of General Datatypes in Communication 5.1.12 Correct Use of Addresses 5.1.13 Decoding a Datatype 157 5.1.14 Examples 165 5.2 Pack and Unpack 174 5.3 Canonical MPI_PACK and MPI_UNPACK 182 6 Collective Communication 187 6.1 Introduction and Overview 6.2 Commnunicator Argument 19d 6.2.1 Specifics for Intra-Communicator Collective Operations 。 1 6.2. Applying Collective Operations to Inter-Communicators 191 6.2.3 Specifics for Inter-Communicator Collective Operations 192 6.3 Barrier Synchronization............ 194 6.4 Broadcas 。。。。。。，。。。。。，。。 6.4.1 Example using MPI_BCAST..... 6.5 Gather ，。 196 6.5.1 Examples using MPI_GATHER,MPI_GATHERV... 200 6.6 Scatter 。 206 6.6.1 Examples using MPI_SCATTER,MPI_SCATTERV 210 6.7 Gather-to-all . 213 6.71 Example using mPl allGATHER 216 6.8 All-to-All Scatter/Gather... 217 6.9 Global Reduction Operations......... 223 6.9.1 Reduce 224 6.9.2 Predefined Reduction Operations 226 vii
4.2.2 Communication Completion under Partitioning . . . . . . . . . . . 110 4.2.3 Semantics of Communications in Partitioned Mode . . . . . . . . . 111 4.3 Partitioned Communication Examples . . . . . . . . . . . . . . . . . . . . . 112 4.3.1 Partition Communication with Threads/Tasks Using OpenMP 4.0 or later . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.3.2 Send-only Partitioning Example with Tasks and OpenMP version 4.0 or later . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 4.3.3 Send and Receive Partitioning Example with OpenMP version 4.0 or later . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5 Datatypes 119 5.1 Derived Datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.1.1 Type Constructors with Explicit Addresses . . . . . . . . . . . . . 121 5.1.2 Datatype Constructors . . . . . . . . . . . . . . . . . . . . . . . . 121 5.1.3 Subarray Datatype Constructor . . . . . . . . . . . . . . . . . . . 133 5.1.4 Distributed Array Datatype Constructor . . . . . . . . . . . . . . 135 5.1.5 Address and Size Functions . . . . . . . . . . . . . . . . . . . . . . 141 5.1.6 Lower-Bound and Upper-Bound Markers . . . . . . . . . . . . . . 144 5.1.7 Extent and Bounds of Datatypes . . . . . . . . . . . . . . . . . . . 147 5.1.8 True Extent of Datatypes . . . . . . . . . . . . . . . . . . . . . . . 149 5.1.9 Commit and Free . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 5.1.10 Duplicating a Datatype . . . . . . . . . . . . . . . . . . . . . . . . 152 5.1.11 Use of General Datatypes in Communication . . . . . . . . . . . . 153 5.1.12 Correct Use of Addresses . . . . . . . . . . . . . . . . . . . . . . . 156 5.1.13 Decoding a Datatype . . . . . . . . . . . . . . . . . . . . . . . . . 157 5.1.14 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.2 Pack and Unpack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 5.3 Canonical MPI_PACK and MPI_UNPACK . . . . . . . . . . . . . . . . . . . 182 6 Collective Communication 187 6.1 Introduction and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 6.2 Communicator Argument . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 6.2.1 Specifics for Intra-Communicator Collective Operations . . . . . . 190 6.2.2 Applying Collective Operations to Inter-Communicators . . . . . . 191 6.2.3 Specifics for Inter-Communicator Collective Operations . . . . . . 192 6.3 Barrier Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 6.4 Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 6.4.1 Example using MPI_BCAST . . . . . . . . . . . . . . . . . . . . . . 195 6.5 Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 6.5.1 Examples using MPI_GATHER, MPI_GATHERV . . . . . . . . . . . 200 6.6 Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 6.6.1 Examples using MPI_SCATTER, MPI_SCATTERV . . . . . . . . . 210 6.7 Gather-to-all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 6.7.1 Example using MPI_ALLGATHER . . . . . . . . . . . . . . . . . . 216 6.8 All-to-All Scatter/Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 6.9 Global Reduction Operations . . . . . . . . . . . . . . . . . . . . . . . . . . 223 6.9.1 Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 6.9.2 Predefined Reduction Operations . . . . . . . . . . . . . . . . . . . 226 vii
6.9.3 Signed characters and reductions 22g 6.9.4 MINLOC and MAXLOC 6.9.5 User-Defined Reduction Operation le of User-Defined Reduc 60e All-Redu 807 Pro s-Local Reduction 40 6 10 Redu 2 6.10.1 MPI REDUCE SCATTER BLOCK 610.2 MPI REDUCE SCATTER 244 6.11 24E sive Sca 240 6.112 247 6.11.3 g MPI_SCAN 248 6.12 50 25 6.12.2 Broad MPI_IBCAST ng Sca o1 ng( -t catter/Gather ing Re ng All-Reduce Scatter with E Equal Blocks 6.12.1 Nonbloc ocking Inclusive Scan 6.13 Pe tent Collectiv 6.13. Persistent Barrier Synchronization 6.13.2 Persistent Broadcast.······. 6.13. Persistent Gather 6.13.4 77 Persistent Scatter 6.13. Persistent Gather-to 6.13.0 Persistent All-to-All Scatter/Gather 29 6.13. Persistent Reduce 29 6.13. Persistent All-Reduce 295 6.13.9 Persistent Reduce-Scatter with Equal Blocks 296 6.13.10 Persistent Reduce-Scatter 298 6.13.11 Persistent Inclusive Scan 299 6.13.12 Persistent Exclusive Scan 6.14 Correctness............ 301 oduc upport Librarie MPI's Support for Libraries..... 7.2 7.2.1 Concepts Groups viii
6.9.3 Signed Characters and Reductions . . . . . . . . . . . . . . . . . . 229 6.9.4 MINLOC and MAXLOC . . . . . . . . . . . . . . . . . . . . . . . 229 6.9.5 User-Defined Reduction Operations . . . . . . . . . . . . . . . . . 233 Example of User-Defined Reduce . . . . . . . . . . . . . . . . . . . 237 6.9.6 All-Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 6.9.7 Process-Local Reduction . . . . . . . . . . . . . . . . . . . . . . . 240 6.10 Reduce-Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 6.10.1 MPI_REDUCE_SCATTER_BLOCK . . . . . . . . . . . . . . . . . . 242 6.10.2 MPI_REDUCE_SCATTER . . . . . . . . . . . . . . . . . . . . . . . 244 6.11 Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 6.11.1 Inclusive Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 6.11.2 Exclusive Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 6.11.3 Example using MPI_SCAN . . . . . . . . . . . . . . . . . . . . . . 248 6.12 Nonblocking Collective Operations . . . . . . . . . . . . . . . . . . . . . . . 250 6.12.1 Nonblocking Barrier Synchronization . . . . . . . . . . . . . . . . . 252 6.12.2 Nonblocking Broadcast . . . . . . . . . . . . . . . . . . . . . . . . 253 Example using MPI_IBCAST . . . . . . . . . . . . . . . . . . . . . 253 6.12.3 Nonblocking Gather . . . . . . . . . . . . . . . . . . . . . . . . . . 254 6.12.4 Nonblocking Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . 257 6.12.5 Nonblocking Gather-to-all . . . . . . . . . . . . . . . . . . . . . . . 260 6.12.6 Nonblocking All-to-All Scatter/Gather . . . . . . . . . . . . . . . . 263 6.12.7 Nonblocking Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . 269 6.12.8 Nonblocking All-Reduce . . . . . . . . . . . . . . . . . . . . . . . . 270 6.12.9 Nonblocking Reduce-Scatter with Equal Blocks . . . . . . . . . . . 271 6.12.10 Nonblocking Reduce-Scatter . . . . . . . . . . . . . . . . . . . . . 273 6.12.11 Nonblocking Inclusive Scan . . . . . . . . . . . . . . . . . . . . . . 274 6.12.12 Nonblocking Exclusive Scan . . . . . . . . . . . . . . . . . . . . . . 275 6.13 Persistent Collective Operations . . . . . . . . . . . . . . . . . . . . . . . . 276 6.13.1 Persistent Barrier Synchronization . . . . . . . . . . . . . . . . . . 277 6.13.2 Persistent Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . 278 6.13.3 Persistent Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 6.13.4 Persistent Scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 6.13.5 Persistent Gather-to-all . . . . . . . . . . . . . . . . . . . . . . . . 286 6.13.6 Persistent All-to-All Scatter/Gather . . . . . . . . . . . . . . . . . 289 6.13.7 Persistent Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 6.13.8 Persistent All-Reduce . . . . . . . . . . . . . . . . . . . . . . . . . 295 6.13.9 Persistent Reduce-Scatter with Equal Blocks . . . . . . . . . . . . 296 6.13.10 Persistent Reduce-Scatter . . . . . . . . . . . . . . . . . . . . . . . 298 6.13.11 Persistent Inclusive Scan . . . . . . . . . . . . . . . . . . . . . . . 299 6.13.12 Persistent Exclusive Scan . . . . . . . . . . . . . . . . . . . . . . . 300 6.14 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 7 Groups, Contexts, Communicators, and Caching 311 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 7.1.1 Features Needed to Support Libraries . . . . . . . . . . . . . . . . 311 7.1.2 MPI’s Support for Libraries . . . . . . . . . . . . . . . . . . . . . . 312 7.2 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 7.2.1 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 viii
7.2.2 Contexts 314 7.2.3 Intra-Communicators 7.2.4 Predefined Intra-Communicators 73 Gro up Management 316 731 Group Accessors 732 Group Constructors 7.3.3 Groun Destructors 324 74 Communicator Management 741 Communicator Ac 7.4.2 Communicator Constructors 7.4.3 345 74 Comm Motivating Ex 34 7.5.1 C ctice 4 7.5.2 rent Practice #2 4 7.5.3 (App roximate)Cu rent Practice 349 7.5.4 ication Safety Example 35 7.5.5 Library Example #1 351 75.6 Library Exa mple #2 353 10 atio 355 76.1 Inter-Cor 357 7.6.2 icator o 358 763 nication E 362 D eline 362 mple 2:Three-Gr "Ri 7.7 Cachin 65 7.1 3 77 s for Invalid Keyva 77.d ibutes 78 Obj ali oosely Synchronous 7.9.2 or E ecut1o】 Static C catio Dynami municator Allocatior The General Case 8 Pro cess Topologies 389 389 82 Virtual Topologie 390 Embeddin in MPI of the Functions 8.5 truc 392 8.5.1 Carte 392 8.5.2 nstructor ian co nce fuinction:mPl dIms CrEAte 393 ix
7.2.2 Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 7.2.3 Intra-Communicators . . . . . . . . . . . . . . . . . . . . . . . . . 315 7.2.4 Predefined Intra-Communicators . . . . . . . . . . . . . . . . . . . 315 7.3 Group Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 7.3.1 Group Accessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 7.3.2 Group Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . 318 7.3.3 Group Destructors . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 7.4 Communicator Management . . . . . . . . . . . . . . . . . . . . . . . . . . 325 7.4.1 Communicator Accessors . . . . . . . . . . . . . . . . . . . . . . . 325 7.4.2 Communicator Constructors . . . . . . . . . . . . . . . . . . . . . 327 7.4.3 Communicator Destructors . . . . . . . . . . . . . . . . . . . . . . 345 7.4.4 Communicator Info . . . . . . . . . . . . . . . . . . . . . . . . . . 345 7.5 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348 7.5.1 Current Practice #1 . . . . . . . . . . . . . . . . . . . . . . . . . . 348 7.5.2 Current Practice #2 . . . . . . . . . . . . . . . . . . . . . . . . . . 349 7.5.3 (Approximate) Current Practice #3 . . . . . . . . . . . . . . . . . 349 7.5.4 Communication Safety Example . . . . . . . . . . . . . . . . . . . 350 7.5.5 Library Example #1 . . . . . . . . . . . . . . . . . . . . . . . . . . 351 7.5.6 Library Example #2 . . . . . . . . . . . . . . . . . . . . . . . . . . 353 7.6 Inter-Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355 7.6.1 Inter-Communicator Accessors . . . . . . . . . . . . . . . . . . . . 357 7.6.2 Inter-Communicator Operations . . . . . . . . . . . . . . . . . . . 358 7.6.3 Inter-Communication Examples . . . . . . . . . . . . . . . . . . . 362 Example 1: Three-Group “Pipeline” . . . . . . . . . . . . . . . . . 362 Example 2: Three-Group “Ring” . . . . . . . . . . . . . . . . . . . 363 7.7 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 7.7.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 7.7.2 Communicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 7.7.3 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 7.7.4 Datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 7.7.5 Error Class for Invalid Keyval . . . . . . . . . . . . . . . . . . . . 379 7.7.6 Attributes Example . . . . . . . . . . . . . . . . . . . . . . . . . . 379 7.8 Naming Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 7.9 Formalizing the Loosely Synchronous Model . . . . . . . . . . . . . . . . . 386 7.9.1 Basic Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 7.9.2 Models of Execution . . . . . . . . . . . . . . . . . . . . . . . . . . 386 Static Communicator Allocation . . . . . . . . . . . . . . . . . . . 387 Dynamic Communicator Allocation . . . . . . . . . . . . . . . . . 387 The General Case . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 8 Process Topologies 389 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389 8.2 Virtual Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 8.3 Embedding in MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 8.4 Overview of the Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 8.5 Topology Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 8.5.1 Cartesian Constructor . . . . . . . . . . . . . . . . . . . . . . . . . 392 8.5.2 Cartesian Convenience Function: MPI_DIMS_CREATE . . . . . . . 393 ix
8.5.3 Graph constructor 8.5.4 8.5.5 op Function 8.5.6 logy Ing rdinate 8.5.7 of Cart Functions 8 6 Neighborhood Collec 8.6.i Neighborhood Gather 8.6.2 Neighbo 8> 87.1 Nonblocking 8.7.2 ersistent Ne 8.9 An Application 9 MPI Environmental Management 451 9.1 Implementation Information 4 9.1.1 Version Inquiries 012 ental Inquiries 45 Tag values 453 Host Rank 453 IO Rank 454 Clock Synchronization 454 sor Nam 9.2 455 9.3 Error Handling 458 9.3.1 Error 461 9.3.2 Error Handlers for Windows 463 9.3.3 Handlers for Files 465 9.3.4 Handler fo 466 9.3.5 and Retrievi ing Er ror Strings 463 04 Erro d Classe 469 9.5 and Error Hand 9.6 10 The Info Object 479 11 Process Initialization,Creation,and Management 11.2 The World Mode 11.2.1 Starting MPI Processes 11.2.2 Finalizing MPI . 11.2.3 Determining Whether MPI Has Been Initialized When Using the World Model.. 11.2.4 Allowing User Functions at MPI Finalization 48 ll.3 The Sessions Model..················ 4 4
8.5.3 Graph Constructor . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 8.5.4 Distributed Graph Constructor . . . . . . . . . . . . . . . . . . . . 396 8.5.5 Topology Inquiry Functions . . . . . . . . . . . . . . . . . . . . . . 403 8.5.6 Cartesian Shift Coordinates . . . . . . . . . . . . . . . . . . . . . . 412 8.5.7 Partitioning of Cartesian Structures . . . . . . . . . . . . . . . . . 413 8.5.8 Low-Level Topology Functions . . . . . . . . . . . . . . . . . . . . 414 8.6 Neighborhood Collective Communication . . . . . . . . . . . . . . . . . . . 416 8.6.1 Neighborhood Gather . . . . . . . . . . . . . . . . . . . . . . . . . 417 8.6.2 Neighbor Alltoall . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422 8.7 Nonblocking Neighborhood Communication . . . . . . . . . . . . . . . . . 429 8.7.1 Nonblocking Neighborhood Gather . . . . . . . . . . . . . . . . . . 429 8.7.2 Nonblocking Neighborhood Alltoall . . . . . . . . . . . . . . . . . 432 8.8 Persistent Neighborhood Communication . . . . . . . . . . . . . . . . . . . 437 8.8.1 Persistent Neighborhood Gather . . . . . . . . . . . . . . . . . . . 438 8.8.2 Persistent Neighborhood Alltoall . . . . . . . . . . . . . . . . . . . 441 8.9 An Application Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 9 MPI Environmental Management 451 9.1 Implementation Information . . . . . . . . . . . . . . . . . . . . . . . . . . 451 9.1.1 Version Inquiries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 9.1.2 Environmental Inquiries . . . . . . . . . . . . . . . . . . . . . . . . 453 Tag Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 Host Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 IO Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454 Clock Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . 454 Inquire Processor Name . . . . . . . . . . . . . . . . . . . . . . . . 454 9.2 Memory Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455 9.3 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458 9.3.1 Error Handlers for Communicators . . . . . . . . . . . . . . . . . . 461 9.3.2 Error Handlers for Windows . . . . . . . . . . . . . . . . . . . . . 463 9.3.3 Error Handlers for Files . . . . . . . . . . . . . . . . . . . . . . . . 465 9.3.4 Error Handlers for Sessions . . . . . . . . . . . . . . . . . . . . . . 466 9.3.5 Freeing Errorhandlers and Retrieving Error Strings . . . . . . . . 468 9.4 Error Codes and Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 9.5 Error Classes, Error Codes, and Error Handlers . . . . . . . . . . . . . . . 473 9.6 Timers and Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . 477 10 The Info Object 479 11 Process Initialization, Creation, and Management 487 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487 11.2 The World Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488 11.2.1 Starting MPI Processes . . . . . . . . . . . . . . . . . . . . . . . . 488 11.2.2 Finalizing MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494 11.2.3 Determining Whether MPI Has Been Initialized When Using the World Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 11.2.4 Allowing User Functions at MPI Finalization . . . . . . . . . . . . 498 11.3 The Sessions Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499 x