Lecture 8: Shared Memory Parallel Programming

Background Material

• Material from lectures 5, , and 7
• Timing and performance analysis of sequential programs

Reading

• Online documentation for Pthreads and OpenMP.
• K. Chen and J. Johnson, A Prototypical Self-Optimizing Package for Parallel Implementation of Fast Signal Transforms, IPDPS 2002, International Parallel and Distributed Processing Symposium.

Topics

• Multi-core processors
• Introduction to shared memory parallel programming with Pthreads
1. Threads
2. fork/join
3. race conditions
4. synchronization
5. performance issues - synchronization overhead, contention and granularity, load balance, cache coherency and false sharing.
• Introduction parallel program design paradigms
  1. Data parallelism (static scheduling)
  2. Task parallelism with workers
  3. Divide and conquer parallelism (fork/join)
• Introduction to shared memory parallel programming with OpenMP
1. Parallel loops
2. Parallel regions
3. Synchronization

Lecture Slides

• ipdps.ppt - slides on parallel WHT (from IPDPS 02 paper)

Programs

• Example programs with UNIX processes
• parchi.c - simple program withfork
• tfork.c and pproc.c - simple program with fork/wait and execl (compile both executables and make sure to name the executable for pproc.c pproc)
• timefj.c, dummy.c timeproc.c dummyproc.c wtime.c (programs to time fork/wait vs. procedure call. wtime.c is used for timing)
• Pthreads programs
• thr.c - sample program with fork/join
• badcount.c - shared memory program with race condition.
• goodcount.c - shared memory program with mutex.
• bettercount.c - improved shared memory program with mutex.
• timefj.c - program to time fork/join programs to time fork/wait vs. procedure call. ( wtime.c is used for timing)
• OpenMP programs
  1. hello.c
  2. hellob.c - hello with barrier
  3. badcount.c
  4. goodcount.c
  5. bettercount.c
  6. gorycount.c
• Parallel programming examples
  1. icount.c - iterative program to add the elements of an array.
  2. rcount.c - recursive program to add the elements of an array.
  3. pcount.c - Parallel divide and conquer program to add the elements of an array.
  4. dcount.c - Parallel dynamic worker program to add the elements of an array.
  5. scount.c - Parallel static task program to add the elements of an array.
  6. cfft.c - divide and conquer radix two FFT. Relies on complex.h and complex.c (don't forget to link with the math library -lm).
  7. testcfft.c - test program for cfft. Input, n = log of size (N = 2^n) and index 0 &le i < N. The output should be the ith column of the N-point DFT matrix.
  8. timecfft.c - timing program for cfft using wtime.c.
  9. pcfft.c - parallel divide and conquer FFT
  10. pcffts.c - parallel divide and conquer FFT version 2 (with stride parameter)
  11. timepcfft.c - timing program for pcfft using wtime.c.
  12. timepcffts.c - timing program for pcffts using wtime.c.

Tasks

1. Implement and time a Parallel radix 2 divide and conquer WHT using Pthreads. Compute speedup compared to your sequential code. What was the smallest input size for which you obtained speedup?
2. Implement and time parallel multiple WHTs, i.e. I_M tensor W_N using openMP.
3. Implement and time a Parallel WHT, i.e. W_MN = (W_M tensor I_N)(I_M tensor W_N), using OpenMP. Both the divide and conquer parts should be parallelized. To improve performance (remove false sharing) loop interleaving should be used for (W_M tensor I_N).

Related Links and Info

1. Multi-core page from wikipedia
2. Multi-core processors
3. xeon 5000 series - Intel quad core (kodiak.cs.drexel.edu) information.
4. Intel Nehalem microarchitecture
5. Hyper-threading page from wikipedia
6. Pthreads page from wikipedia
7. Pthreads tutorial.
8. openmp.org
9. openMP specifications
10. OpenMP tutorial.