Building Agent Evals You Can Hill Climb
Discussing benchmarks and evals, and how they’re increasingly being used not just as regression tests, but as targets for optimizations for Agents and LLM post training. Maybe one day we’ll all reach the top of that hill we’re climbing.
Resources:
Benchmark Blog - https://lucek.ai/blogs/environments
Harvey LAB - https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark
IssueBench v1 - https://www.langchain.com/blog/issuebench-how-we-evaluate-engine
Automating Eval Design with Claude - https://claude.dev/blog/automating-eval-design-and-hillclimbing/
Chapters:
00:00 - Introduction
0