The GELU, SiLU and SwiGLU activation functions, clearly explained!!!

There's a new activation function in town. Actually, there are a bunch of new activation functions in town. This StatQuest gives you the essential information you need at the very start, followed by a surprisingly interesting derivation of each of these new activation functions. It turns out that Dropout is all you need. Who knew? If you'd like to support StatQuest, please consider... Patreon: https://www.patreon.com/statquest ...or... YouTube Membership: https://www.youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw/join ...buying a book, a study guide, a t-shirt or hoodie, or a song from the Sta