Your Softmax Is Broken. Here's the Fix.

Standard implementations of softmax, log-softmax, and cross-entropy in every major framework have numerical BUGS! Where does the gradient go, and why does a rounding error make a model's loss spike? Check out the video to find out! Chapters 0:00 The paradox: 99.999997% sure, yet zero gradient 1:46 Softmax and cross-entropy from scratch 5:40 Floating point: why 1 + something small = 1 10:06 Softmax Collapse: the gradient falls off a cliff 13:38 The fix: avoid adding one to a tiny number 17:23 Adam optimizer, grokking, and the Slingshot spikes 20:53 Why margins stop at log(1/ε) 24:07 Softmax Can