Training Models Too Big for One GPU | ZeRO & FSDP

How do you train a model that does not fit on one GPU? In this video, we cover how GPUs represent and store real numbers, mixed-precision training, and the Zero Redundancy Optimizer (ZeRO) and FSDP to reduce memory redundancy in data parallelism. Chapters 0:00 How a model learns 0:55 Adam optimizer 1:56 How a computer stores real numbers 11:40 The memory constraint 12:40 Data parallelism 13:21 How GPUs sync their gradients 16:07 ZeRO stage 1 and 2: shard the optimizer states, then the gradients 19:27 ZeRO stage 3: shard the weights 21:28 FSDP: the same schedule one unit at a time Referenc