AI Evals: Everything You Need to Know

This document curates the most common questions Shreya and I received while teaching 5,000+ engineers and PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment. How to use this FAQ Browse the questions that interest you, or choose a guide below for a curated reading path through the FAQs and related articles. Where are you? This sounds like me I’m new to evals I’ve heard the term, but I’m not sure what evals involve or whether I need them. I don’t know what to test I’m building an AI product, but I haven’t figured out w