PBT doesn’t make grokking faster, but it can escape a hyperparameter range where nothing groks [P]

I ran Population Based Training over learning rate and weight decay on grokking (modular addition mod 97, 35% train split, small MLP, 20 members, 20k steps, full batch). Results vs a grid (equal compute, 3 seeds): the b…