How Vision-Language Models Actually See Images: Inside ViT, CLIP, SigLIP & 3D M-RoPE

Why an “iPod” sticker fools OpenAI CLIP (64.5%) while Qwen2.5-VL reads it cleanly — measured layer by layer on a 96 GB Blackwell GPU. By Latent AI Labs (@LatentAILabs) TL;DR — How Do Vision-Language Models (VLMs) Actually See Images? Modern Vision-Language Models (such as Qwen2.5-VL , GPT-4o , and Claude 3.5 Sonnet ) process images in three stages : 1. Vision Transformer (ViT) Patch Slicer & 2×2 Spatial Patch Merger: A 560 × 560 RGB photograph ( 940,800 floats) is sliced by a 14 × 14 pixel grid into 40 × 40 = 1,600 patches , projected into 1,280-dimensional vectors , refined through 28 local-w