Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

A one-line prompt change made a support agent 17 points safer and 17 points worse at finishing tasks. Sofia Jakovcevic, forward deployed engineer at Arize AI, explains why agents fail at runtime rather than in code, and walks through a five-step regression loop built from production traces: curate datasets, score with evaluators, run baseline experiments, compare changes, and gate merges in CI. Then she shows how to close the loop at agent speed, with coding agents that read traces and fix themselves. Chapters 00:00 Intro: observability for agents 01:08 The real problem: fixes that break other