Reiner Pope – Chip design from the bottom up
Working up from basic logic gates to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.
TL;DR
- Chips are built from fundamental logic gates like AND, OR, and NOT.
- The multiply-accumulate operation is a core primitive for AI chips, essential for matrix multiplication.
- Systolic arrays are designed to bake matrix multiplication loops into hardware, significantly reducing data movement costs compared to traditional CPU/GPU cores.
- Clock cycles synchronize parallel operations in chips; inserting pipeline registers can increase clock speed but also area.
- FPGAs offer flexibility and deterministic latency for frequently changing workloads, while ASICs are more cost-effective and performant for fixed designs.
- CPU cores are larger than GPU cores primarily due to extensive cache systems and branch predictors, which are largely absent in GPUs.
- The human brain operates with unstructured sparsity and co-located memory and compute, differing from the structured parallelism and separate memory hierarchies in chips.
- GPUs can be seen as tiled arrangements of many smaller, similar units, akin to many tiny TPUs, each with local memory and compute capabilities.