Position Embeddings: Learned, Sinusoidal, Relative | The attention family series - Lecture 9

This is Lecture 9 of The Attention Family. Attention by itself does not know the order of words: "All dogs are animals" and "All animals are dogs" give every word the same output. In this lecture we see four ways to give attention the position of each word, and where each one enters the attention computation. In this lecture I cover: - The problem: nothing in Q, K, V or the softmax uses the position of a token, so reordering the words only reorders the outputs - Does adding a position vector make "dogs" look like a different word? Measured on the real GPT-2 weights: in 99.999% of cases the cl