KLD is misunderstood

I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't. So what does KLD actually measure? If you feed the same text to both the base model and the quantized model, you can compare the probability distributions obtained from their logits at each token. KLD basically measures how different these distributions are, and then we take the average across all tokens. The first thing that can go very wrong here is that people just throw some random text at the model, like a bunch of