《高等数值分析（高性能计算/并行计算）》课程教学资源（参考资料）OpenMP Application Programming Interface Version 5.0

OpenMP OpenMP Application Programming Interface Version 5.0 November 2018 Copyright 1997-2018 OpenMP Architecture Review Board. Permission to copy without fee all or part of this material is granted,provided the OpenMP Architecture Review Board copyright notice and the title of this document appear.Notice is given that copying is by permission of the OpenMP Architecture Review Board

OpenMP Application Programming Interface Version 5.0 November 2018 Copyright c 1997-2018 OpenMP Architecture Review Board. Permission to copy without fee all or part of this material is granted, provided the OpenMP Architecture Review Board copyright notice and the title of this document appear. Notice is given that copying is by permission of the OpenMP Architecture Review Board

Contents 1 Introduction 1 1.1 Scope. 1.2 Glossary 。·。。。·。。。。·。·。。4。·。·。。。。·。· 2 1.2.1 Threading Concepts 1.2.2 OpenMP Language Terminology 123 Loop Terminology.....................·..... 1.2.4 Synchronization Terminology ........................ 9 1.2.5 Tasking Terminology 。。。。。。。。。。，。。，。，。。。。。。。。。。。。 10 1.2.6 Data Terminology 12 1.2.7 Implementation Terminology 17 128 Tool Terminology 17 1.3 Execution Model.... 20 1.4 Memory Model.... 23 1.4.1 Structure of the OpenMP Memory Model. 23 1.4.2 Device Data Environments 24 1.4.3 Memory Management 3 14.4 The Flush Operation......,..··· 子 1.4.5 Flush Synchronization and Happens Before.················· 27 1.4.6 OpenMP Memory Consistency 28 l5 Tool Interfaces.········ 29 1.5.1 OMPT 1.5.20MPD 30

Contents 1 Introduction 1 1.1 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2.1 Threading Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2.2 OpenMP Language Terminology . . . . . . . . . . . . . . . . . . . . . . . 2 1.2.3 Loop Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2.4 Synchronization Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.2.5 Tasking Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.6 Data Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.2.7 Implementation Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.2.8 Tool Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.3 Execution Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.4 Memory Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 1.4.1 Structure of the OpenMP Memory Model . . . . . . . . . . . . . . . . . . . 23 1.4.2 Device Data Environments . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.4.3 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4.4 The Flush Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4.5 Flush Synchronization and Happens Before . . . . . . . . . . . . . . . . . . 27 1.4.6 OpenMP Memory Consistency . . . . . . . . . . . . . . . . . . . . . . . . 28 1.5 Tool Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 1.5.1 OMPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 1.5.2 OMPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 i

l.6 OpenMP Compliance............................... 31 l7 Normative References...·····.················· 31 l.8 Organization of this Document······················· 34 2 Directives 37 2.1 Directive Format.......·....·...····.··.···. 38 2.1.1 Fixed Source Form Directives 41 2.1.2 Free Source Form Directives 41 2.1.3 Stand-Alone Directives... 42 2.l.4 Array Shaping...·········· 3 2.1.5 Array Sections..........。..。.。.........。..。.。 44 2.1.6 47 2.2 Conditional Compilation 49 2.2.1 Fixed Source Form Conditional Compilation Sentinels.. 50 2.2.2 Free Source Form Conditional Compilation Sentinel.···..,,..·,· 50 23 Variant Directives.,......,. 51 2.3.1 51 2.3.2 Context Selectors 53 2.33 Matching and Scoring Context Selectors.········ 55 2.3.4 Metadirectives.............................. 56 2.3.5 declare variant Directive ....................... 58 2.4 requires Directive.,.,...,.,.,.,..,.............,. 60 2.5 Internal Control Variables 63 2.5.1 ICV Descriptions 64 2.5.2 ICV Initialization 66 2.5.3 Modifying and Retrieving ICV Values.................... 68 2.5.4 How ICVs are Scoped........,...........,....,... 70 2.5.4.1 How the Per-Data Environment ICVs Work 72 2.5.5 ICV Override Relationships 72 2.6 parallel Construct.············ 2.6.1 Determining the Number of Threads for a parallel Region........ 中 2.6.2 Controlling OpenMP Thread Affinity.··········· 。。。。。。。 80 2.7 teams Construct 82 OpenMP API-Version 5.0 November 2018

1.6 OpenMP Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.7 Normative References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.8 Organization of this Document . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2 Directives 37 2.1 Directive Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.1.1 Fixed Source Form Directives . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.1.2 Free Source Form Directives . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.1.3 Stand-Alone Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.1.4 Array Shaping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.1.5 Array Sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.1.6 Iterators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.2 Conditional Compilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 2.2.1 Fixed Source Form Conditional Compilation Sentinels . . . . . . . . . . . . 50 2.2.2 Free Source Form Conditional Compilation Sentinel . . . . . . . . . . . . . 50 2.3 Variant Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.3.1 OpenMP Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.3.2 Context Selectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 2.3.3 Matching and Scoring Context Selectors . . . . . . . . . . . . . . . . . . . 55 2.3.4 Metadirectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.3.5 declare variant Directive . . . . . . . . . . . . . . . . . . . . . . . . 58 2.4 requires Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.5 Internal Control Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 2.5.1 ICV Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 2.5.2 ICV Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 2.5.3 Modifying and Retrieving ICV Values . . . . . . . . . . . . . . . . . . . . 68 2.5.4 How ICVs are Scoped . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 2.5.4.1 How the Per-Data Environment ICVs Work . . . . . . . . . . . . . . . 72 2.5.5 ICV Override Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . 72 2.6 parallel Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 2.6.1 Determining the Number of Threads for a parallel Region . . . . . . . . 78 2.6.2 Controlling OpenMP Thread Affinity . . . . . . . . . . . . . . . . . . . . . 80 2.7 teams Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 ii OpenMP API – Version 5.0 November 2018

2.8 Worksharing Constructs. 86 2.8.1 sections Construct.:........................... 86 2.8.2 single Construct 89 283 workshare Construct 2.9 Loop-Related Directives 8 2.9 1 Canonical Loop Form,·,················ 号 2.9.2 Worksharing-Loop Construct......................... 101 2.9.2.1 Determining the Schedule of a Worksharing-Loop............ 109 2.9.3 SIMD Directives 110 2.9.3.1 simd Construct 110 2.9.3.2 Worksharing-Loop SIMD Construct...... 114 2.9.3.3 declare simd Directive 116 2.9.4 distribute Loop Constructs 120 2.9.4.1 distribute Construct 120 29.42 distribute simd Construct 123 2.9.4.3 Distribute Parallel Worksharing-Loop Construct 125 2.9.4.4 Distribute Parallel Worksharing-Loop SIMD Construct......... 126 2.9.5 loop Construct 4。。。。。。。。。。。 128 2.9.6 scan Directive 132 2.10 Tasking Constructs. 135 2.10.1 task Construct 135 2.10.2 taskloop Construct 140 2.10.3 taskloop simd Construct 146 2.10.4 taskyield Construct 147 148 2.10.6 Task Scheduling 149 2.11 Memory Management Directives.......................... 152 2.11.1 Memory Spaces 。。。。。。。，。，。。。。。。。。。。。。。。。。 152 2.11.2 Memory Allocators 152 2.11.3 allocate Directiv 。。。。。。。。。。。。。。。。。。。。。。。。 156 2.11.4 allocate Clause 158 2.12 Device Directives 160 2.l2.1 Device Initialization...........·,.········· 160 Contents 道

2.8 Worksharing Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 2.8.1 sections Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 2.8.2 single Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 2.8.3 workshare Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.9 Loop-Related Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 2.9.1 Canonical Loop Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 2.9.2 Worksharing-Loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . 101 2.9.2.1 Determining the Schedule of a Worksharing-Loop . . . . . . . . . . . . 109 2.9.3 SIMD Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 2.9.3.1 simd Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 2.9.3.2 Worksharing-Loop SIMD Construct . . . . . . . . . . . . . . . . . . . 114 2.9.3.3 declare simd Directive . . . . . . . . . . . . . . . . . . . . . . . . 116 2.9.4 distribute Loop Constructs . . . . . . . . . . . . . . . . . . . . . . . . 120 2.9.4.1 distribute Construct . . . . . . . . . . . . . . . . . . . . . . . . . 120 2.9.4.2 distribute simd Construct . . . . . . . . . . . . . . . . . . . . . . 123 2.9.4.3 Distribute Parallel Worksharing-Loop Construct . . . . . . . . . . . . . 125 2.9.4.4 Distribute Parallel Worksharing-Loop SIMD Construct . . . . . . . . . 126 2.9.5 loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 2.9.6 scan Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 2.10 Tasking Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 2.10.1 task Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 2.10.2 taskloop Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 2.10.3 taskloop simd Construct . . . . . . . . . . . . . . . . . . . . . . . . . 146 2.10.4 taskyield Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 2.10.5 Initial Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 2.10.6 Task Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 2.11 Memory Management Directives . . . . . . . . . . . . . . . . . . . . . . . . . . 152 2.11.1 Memory Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 2.11.2 Memory Allocators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 2.11.3 allocate Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 2.11.4 allocate Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 2.12 Device Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 2.12.1 Device Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 Contents iii

2.12.2 target data Construct........................... 161 2.l2.3 target enter data Construct..··· 164 2.12.4 target exit data Construct 2.12.5 target Construct...... 170 2.12.6 target update Construct 176 2.12.7 declare target Directive 。。。4。。。。。tg。g。+。。。。4.74。。。 180 2.13 Combined Constructs 185 2.13.1 Parallel Worksharing-Loop Construct 185 2.13.2 parallel loop Construct 186 2.l3.3 paral1 el sections Construct··················· 188 2.13.4 parallel workshare Construct 189 2.13.5 Parallel Worksharing-Loop SIMD Construct 190 2.13.6 parallel master Construct 191 2.l3.7 master taskloop Construct.··:.·············· 192 2.13.8 master taskloop simd Construct 194 2.13.9 parallelmaster taskloop Construct 。。。。。。。。。。。。。 195 2.13.10 parallel master taskloop simd Construct...... 196 2.l3.11 teams distribute Construct.··,···,···· 197 2.13.12 teams distribute simd Construct 198 2.13.13 Teams Distribute Parallel Worksharing-Loop Construct 200 2.l3.14 Teams Distribute Parallel Worksharing-Loop SIMD Construct.·····.. 201 2.13.15 teams loop Construct 202 2.l3.l6 target paral1e1 Construct.···· 203 2.13.17 Target Parallel Worksharing-Loop Construct 205 2.13.18 Target Parallel Worksharing-Loop SIMD Construct 206 2.13.19 target parallel loop Construct 208 2.l3.20 target simd Construct........................... 209 2.l3.2 I target teams Construct················ 210 2.13.22 target teams distribute Construct 211 2.13.23 target teams distribute simd Construct 213 2.13.24 target teams loop Construct...... 214 2.l3.25 Target Teams Distribute Parallel Worksharing--Loop Construct.······· 215 2.13.26 Target Teams Distribute Parallel Worksharing-Loop SIMD Construct 216 iv OpenMP API-Version 5.0 November 2018

2.12.2 target data Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 2.12.3 target enter data Construct . . . . . . . . . . . . . . . . . . . . . . . 164 2.12.4 target exit data Construct . . . . . . . . . . . . . . . . . . . . . . . 166 2.12.5 target Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 2.12.6 target update Construct . . . . . . . . . . . . . . . . . . . . . . . . . 176 2.12.7 declare target Directive . . . . . . . . . . . . . . . . . . . . . . . . . 180 2.13 Combined Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 2.13.1 Parallel Worksharing-Loop Construct . . . . . . . . . . . . . . . . . . . . . 185 2.13.2 parallel loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . 186 2.13.3 parallel sections Construct . . . . . . . . . . . . . . . . . . . . . . 188 2.13.4 parallel workshare Construct . . . . . . . . . . . . . . . . . . . . . 189 2.13.5 Parallel Worksharing-Loop SIMD Construct . . . . . . . . . . . . . . . . . 190 2.13.6 parallel master Construct . . . . . . . . . . . . . . . . . . . . . . . . 191 2.13.7 master taskloop Construct . . . . . . . . . . . . . . . . . . . . . . . . 192 2.13.8 master taskloop simd Construct . . . . . . . . . . . . . . . . . . . . 194 2.13.9 parallel master taskloop Construct . . . . . . . . . . . . . . . . . 195 2.13.10 parallel master taskloop simd Construct . . . . . . . . . . . . . . 196 2.13.11 teams distribute Construct . . . . . . . . . . . . . . . . . . . . . . . 197 2.13.12 teams distribute simd Construct . . . . . . . . . . . . . . . . . . . 198 2.13.13 Teams Distribute Parallel Worksharing-Loop Construct . . . . . . . . . . . 200 2.13.14 Teams Distribute Parallel Worksharing-Loop SIMD Construct . . . . . . . . 201 2.13.15 teams loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 2.13.16 target parallel Construct . . . . . . . . . . . . . . . . . . . . . . . . 203 2.13.17 Target Parallel Worksharing-Loop Construct . . . . . . . . . . . . . . . . . 205 2.13.18 Target Parallel Worksharing-Loop SIMD Construct . . . . . . . . . . . . . 206 2.13.19 target parallel loop Construct . . . . . . . . . . . . . . . . . . . . 208 2.13.20 target simd Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 2.13.21 target teams Construct . . . . . . . . . . . . . . . . . . . . . . . . . . 210 2.13.22 target teams distribute Construct . . . . . . . . . . . . . . . . . . 211 2.13.23 target teams distribute simd Construct . . . . . . . . . . . . . . 213 2.13.24 target teams loop Construct . . . . . . . . . . . . . . . . . . . . . . . 214 2.13.25 Target Teams Distribute Parallel Worksharing-Loop Construct . . . . . . . . 215 2.13.26 Target Teams Distribute Parallel Worksharing-Loop SIMD Construct . . . . 216 iv OpenMP API – Version 5.0 November 2018