GPUs, CUDA and memory arithmetic

Deep learning is economically possible because one hardware family does dense linear algebra ~100× cheaper than CPUs — every matmul in the Deep Learning module’s lessons lands on it. But the GPU stack is also the toolkit’s most fragile tower, and its memory its hardest wall. This lesson gives you the three things practitioners actually need: a mental model of the software stack (so is_available() == False is a five-minute fix, not a day), a model of GPU performance (so you know when you’re leaving 10× on the table), and the memory arithmetic — which, unlike most of this module, is exact math you can do on a napkin and should do before every sizable run.

The stack: three versions that must agree

Bottom to top: the driver (kernel module; ships with the OS/GPU installer; talks to silicon), the CUDA runtime (libraries implementing the compute API), and PyTorch built against some CUDA version. The modern simplification most tutorials haven’t caught up with: torch wheels bundle their own CUDA runtime — you do NOT install the CUDA toolkit to use PyTorch. Only two version constraints remain: the driver must be at least as new as the bundled runtime needs (drivers are backward- compatible with older runtimes), and the wheel must match your platform/ Python. Hence the entire diagnosis tree for torch.cuda.is_available() == False: (1) nvidia-smi fails → no/broken driver — that’s the layer to fix, nothing Python-side will help; (2) nvidia-smi works but torch says no → you installed a CPU-only wheel (check torch.version.cuda — None is the tell) or a wheel whose CUDA needs a newer driver than nvidia-smi reports; (3) both fine but wrong GPU/none visible → CUDA_VISIBLE_DEVICES or a container missing --gpus. Ninety-odd percent of incidents are case (2): reinstall from the correct index URL, done.

import torch
print(torch.__version__, torch.version.cuda)   # None ⇒ CPU-only wheel — case (2)
print(torch.cuda.is_available(), torch.cuda.get_device_name(0))
print(torch.cuda.get_device_properties(0).total_memory / 2**30)  # the budget

What GPUs are fast at — and the two regimes

A GPU is thousands of simple cores plus very high-bandwidth memory, designed for throughput on regular parallel work. Two numbers govern everything: peak FLOP/s and memory bandwidth, and their ratio sets the arithmetic intensity (FLOPs per byte moved) a kernel needs to be compute-bound. Big matmuls (the DL module’s workloads) clear the bar — that’s the 100×. Elementwise ops, small batches, and short sequences do not: they are memory-bound, running at bandwidth speed no matter the TFLOPs on the spec sheet. Practical corollaries: batch small ops together (the vectorization lesson’s law, GPU edition), prefer a few big matmuls to many small ones, and know that inference of LLMs is memory-bandwidth- bound (the LLM track’s serving lesson builds on exactly this fact). Also budget honestly for transfers: PCIe is ~10–60 GB/s against ~1–3 TB/s on-device — a model that ping-pongs tensors to CPU each step can be slower than not using the GPU at all. Mixed precision (bf16 — the DL module’s training lesson has the fine print) roughly doubles both effective compute and effective memory, which is why it is on by default everywhere.

The memory arithmetic

VRAM during training is four ledgers, three of them exact:

The canonical case — full fine-tuning with AdamW in mixed precision: bf16 weights (2 B) + bf16 grads (2 B) + fp32 Adam m and v (4+4 B) + fp32 master weights (4 B) = 16 bytes per parameter before a single activation: a 7B model wants ~112 GB just to exist in training. The two levers that make the impossible fit: quantize the frozen weights and shrink the trainable set (LoRA — the LLM track derives it; its memory win is visible below), and offload/shard what remains (ZeRO/FSDP, the DL module’s closing map). Inference is the cheap ledger: weights at chosen precision plus a KV-cache the LLM track prices exactly.

Operational habits that close the loop: measure, don’t guess — torch.cuda.max_memory_allocated() after a probe step is ground truth; fragmentation makes the usable budget ~10% less than the total; OOM first-response is (in order) halve the batch + gradient-accumulate (free, exact — DL module proved it), turn on activation checkpointing (~30% compute for a big activation cut), then shrink/shard the optimizer ledger (8-bit Adam, LoRA, FSDP). And del + torch.cuda.empty_cache() is for notebooks, not a strategy.

Exercises