Regression Testing in Agile for RAG Applications

Here’s a sprint story that’s getting really common. The team building the company’s AI assistant tweaks the chunking strategy, because answers about long policy documents felt a bit thin. Demo day comes, the new answers look great, everyone’s happy. Then a week later support pings: the assistant now gets refund questions wrong. Questions it used to nail. Nobody touched the refund docs. Nobody touched the refund prompt. But retrieval-augmented generation has a lot of moving parts, and changing one of them can quietly shift behavior somewhere else entirely. That’s exactly the kind of thing regre