Anupam Nayak - Late-Stage Patching Full-Attention Transformers to Gated DeltaNets

Description: This talk explores how model architecture can be changed late in training rather than remaining fixed after pretraining. In particular, pretrained full-attention transformers can be converted into more efficient hybrid architectures without retraining from scratch. Prior conversion approaches largely recover short-context performance but struggle to recover long-context reasoning and retrieval. We show that a pipeline of representation alignment, off-policy distillation, and prolonged on-policy distillation can recover these capabilities using only a small fraction of the original