Exploring “Next-Latent Prediction Transformers Learn Compact World Models” with Jayden Teoh from MIT

today we're chatting with jayden teoh, MIT CSAIL PhD student about Next-Latent Prediction Transformers and what they means for pre-training. The technique developed during his time at Microsoft Research under John Langford's essentially force a transformer towards learning a belief state. Something that isn’t explicitly enforced during normal next token prediction training. NextLat achieves this by adding one small MLP that has to predict the model's own next hidden state given the next token. If the current state can regenerate the next one, it must already contain everything that matters