How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI

A hands-on workshop from SallyAnn DeLucia, head of product at Arize AI, on moving from vibe checks to real evals. She covers what an eval is and how it differs from a unit test, the main types (LLM as a judge, decision models, code evals, annotations, agent as a judge), how to read a trace, how to write concrete pass/fail criteria, and how to build and iterate on an LLM judge in Arize's prompt playground. Chapters 00:00 Intro and agenda 02:06 Why vibe checks don't scale 03:08 Is an eval just a unit test? 04:13 Why agents make evals harder 05:15 Types of evals 07:29 Workshop setup 10:16 Explori