Live coding Jev from Scratch | Understanding Qwen architecture
Deep dive into Pytorch and HF Transformers, to adapt a Qwen model into JEV. Main YT video: https://youtu.be/sF3CNPbWA8o
Git: https://github.com/avbiswas/bev-train
We made it choice-order invariant by hacking directly into the attention mask and positional-ids!
To join our Patreon, visit: https://www.patreon.com/NeuralBreakdownwithAVB
0:00 Introduction/Advice for new devs
10:05 Jev project
13:21 Decision data set creation
17:16 Understanding model inputs
23:54 Language model architecture
35:29 Data structure and training
51:57 Live coding: Model setup
59:43 Input ids and attention masks
1:1