Next-Latent Prediction Transformers Learn World Models feat. Jayden Teoh from MIT

it's very interesting to see where the next-token transformer still fall short in surprisingly ways while crushing problems that look even more daunting. good exemple is drawing manhattan map from directions or the little path-star graphs problem. I had the pleasure of chatting with jayden teoh from MIT who walks us through next-latent prediction (NextLat), a simple fix in the training objective he worked on when he was still an undergrad! the trick is super tiny: bolt a small MLP onto a GPT and make it predict the transformer's own next hidden state. to pull that off, the internal representat