Writing Parallel software 花5 Overall Strategy Data or task based parallelism o Definitely both concurrently o Use as much parallelism as you can o allows to hide dependencies/contention o and to keep the hardware busy what find empect 11/46 S.Ponce-CERN
Writing Parallel software 11 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Overall Strategy Data or task based parallelism ? Definitely both concurrently Use as much parallelism as you can allows to hide dependencies/contention and to keep the hardware busy
Writing Parallel software Intro Is it worth it Amdahl's law Applies to a fixed size problem, 1 Parallel part speedup in terms of time 50% 14 75% 90% Maximum achievable speedup 95% 10 Speedupmax=(1-p)+月 7 8 6 4 op the parallel portion, 2 p∈[0,1刂 0 2 48163264128256512 o n the number of workers level of parallelisation "..the effort expended on achieving high parallel processing rates is wasted unless it is accompanied by achievements in sequential processing rates of very nearly the same magnitude."-Gene Amdahl-1967 what find expect 12/46 S.Ponce-CERN
Writing Parallel software 12 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Is it worth it ? Amdahl’s law Applies to a fixed size problem, speedup in terms of time Maximum achievable speedup Speedupmax = 1 (1−p)+ p n p the parallel portion, p ∈ [0, 1] n the number of workers 1 2 4 8 16 32 64 128 256 512 0 2 4 6 8 10 12 14 16 18 20 level of parallelisation Speedup Parallel part 50% 75% 90% 95% “. . . the effort expended on achieving high parallel processing rates is wasted unless it is accompanied by achievements in sequential processing rates of very nearly the same magnitude.” - Gene Amdahl - 1967
Writing Parallel software Intro Is it worth it Gustafson's law 120 Parallel part Applies to a fixed time interval, 100 50% with more or less work done 75% 80 90% 95% Maximum achievable speedup 60 Speedupmax =(1-p)+np 0 o p the parallel portion, 20 p∈[0,1刂 0 2040. 60 80100120 o n the number of workers level of parallelisation Conclusion o there is no limit to the achievable speedup o you only need enough work items providing they can run in parallel what find expect 13/46 S.Ponce-CERN
Writing Parallel software 13 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Is it worth it ? Gustafson’s law Applies to a fixed time interval, with more or less work done Maximum achievable speedup Speedupmax = (1 − p) + np p the parallel portion, p ∈ [0, 1] n the number of workers 20 40 60 80 100 120 0 20 40 60 80 100 120 level of parallelisation Speedup Parallel part 50% 75% 90% 95% Conclusion there is no limit to the achievable speedup ! you only need enough work items providing they can run in parallel
Writing Parallel software Consequences Amdahl's limitations o the speedup of a given task has a limit o in other terms,latency improvements are constrained the limit being inversely proportional to the size of the serial part o so it is vital to reduce the non parallel part o and even more with large parallelism (manycores) Gustafson's promises doing more processing at once will drop limitations by reducing the percentage of serial work overall as it's usually a fixed amount of time o in other terms,throughput is not limited providing you have enough independent items o think throughput rather than latency what find expect 14/46 S.Ponce-CERN
Writing Parallel software 14 / 46 S. Ponce - CERN Intro Threading Safety Solutions what find expect Consequences Amdahl’s limitations the speedup of a given task has a limit in other terms, latency improvements are constrained the limit being inversely proportional to the size of the serial part so it is vital to reduce the non parallel part and even more with large parallelism (manycores) Gustafson’s promises doing more processing at once will drop limitations by reducing the percentage of serial work overall as it’s usually a fixed amount of time in other terms, throughput is not limited providing you have enough independent items think throughput rather than latency
Writing Parallel software Intro Threading Safety Solutioas Threading,theory and practice o Python threading Introduction and expectations o C++threading and async Threading,theory and practice Thread-safety issues Processes and threads Thread-safety solutions process pthon C+ 15/46 S.Ponce-CERN
Writing Parallel software 15 / 46 S. Ponce - CERN Intro Threading Safety Solutions process python C ++ Threading, theory and practice 1 Introduction and expectations 2 Threading, theory and practice Processes and threads Python threading C ++threading and async 3 Thread-safety issues 4 Thread-safety solutions