# CS 5220: Applications of Parallel Computers
## Memory matters
## 01 Sep 2015
## [My laptop](http://ark.intel.com/products/75991/Intel-Core-i5-4288U-Processor-3M-Cache-up-to-3_10-GHz)
- Theoretical peak: roughly 100 GFlop/s
- Peak memory bandwidth: 25.6 GigaBytes/s
- Arithmetic intensity = flops/memory access
- Low arithmetic intensity is bad news...
How long will it take?
Consider my machine (peak 100 GFlop/s, peak memory bandwidth 25.6 GB/s).
I have a code with double precision arithmetic intensity one.
What is the max flop rate?
A: 100 GFlop/s
=: I appreciate your optimism, but no.
A: 25.6 GFlop/s
=: Right idea, but each double is eight bytes.
A: 3.2 GFlop/s
=: Yup.
A: 320 MFlop/s
=: That's a bit too pessimistic.
## Memory basics
- Memory *latency* = how long to get requested item
- Memory *bandwidth* = rate memory can provide data
- Bandwidth improving faster than latency
- Processor demand growing faster than either!
## Cache basics
- Programs usually have *locality*
- *Spatial*: nearby items accessed consecutively
- *Temporal*: use a small "working set" repeatedly
- Cache hierarchy built to use locality
- Cache = small, fast memory
- Several types of cache on modern chips
## Caches help...
- Hide memory cost by reusing data
- Exploits temporal locality
- Use bandwidth to fetch *cache line* all at once
- Exploits spatial locality
- Use bandwidth to support multiple outstanding reads
- Overlap computation + communication with memory
- aka prefetching
This is (mostly) automatic and implicit.
## Cache organization
- Store cache *lines* of several bytes
- Cache *hit* = copy of needed data in cache
- Cache *miss* otherwise. Three types:
- Compulsory: data never used before
- Capacity: working set too big, discarded data
- Conflict: insufficient *associativity* for access pattern
- Cache *hit rate* = cache hits / memory accesses attempted
## Cache associativity
- Where can data for a given main memory address go?
- Direct-mapped: only one cache location
- n-way set associative: n possible cache locations
- Fully associative: anywhere in cache
- Ex: 8-bit addresses $10011101_2$
- Cache location based on low-order bits of address
- Direct mapped (16 entries): only store in $1101_2$
- 4-way associative (64 entries): four possible locations
- In either case, address $10111101_2$ would conflict
- High associativity is more expensive
## Caches on my laptop
Multiple *levels* of cache with different sizes and latencies.
Cache lines are 64B in all cases, I think.
- Data caches:
- L1 cache: 64 KB/core, 8-way (4 clocks)
- L2 cache: 256 KB/core, 8-way (12 clocks)
- L3 cache: 3 MB (shared), direct mapped (21 clocks?)
- Also have instruction caches for code (less worry)
Miss in lower level cache may still hit in higher level cache.
## Modeling question
Consider my machine (100 Gigaflop/s peak, 25.6 GB/s bandwidth).
Suppose a workload of mostly double precision fused multiply-adds.
What is the minimum cache hit rate needed to maintain half peak?
(We'll talk more about this in class)
## Modeling question
We have $N = 10^6$ two-dimensional coordinates, and want the centroid.
Which of these is faster and why?
1. Store an array of $(x_i, y_i)$ coordinates. Loop $i$ and
simultaneously sum the $x_i$ and $y_i$.
2. Store an array of $(x_i, y_i)$ coordinates. Loop $i$ and
sum the $x_i$, then sum the $y_i$ in a separate loop.
3. Store the $x_i$ n one array, the $y_i$ in a second array.
Sum the $x_i$, then sum the $y_i$.
4. Other methods?
Try it out and see!
## A memory benchmark (membench)
for array A of length L from 4KB to 8MB by 2x
for stride s from 4 bytes to L/2 by 2x
time the following loop
for i = 0 to L by s
load A[i]
Membench on my laptop
Membench on my laptop
