The world's smallest attention mechanism

This is the third and last video of a trilogy of videos titled "The world's smallest transformer model". In this video, we add an attention mechanism to the tiny transformer we created in the second video, that is trained in a dataset of 2 sentences and 4 words, and with a 4-2-4 architecture. This attention mechanism allows the network to carry memory of the previous words in the context. The attention mechanism is analogous to a gravitational force that pulls words together. When ambiguities arise, multi-head attention comes to the rescue. Video 1: How neural networks bend space https://www