Better Harness: A Recipe for Harness Hill-Climbing with Evals

Posted by AISignal

By Vivek Trivedy, Product Manager @ LangChain&x1F4A1;TL;DR: We can build better agents by building better harnesses. But to autonomously build a “better” harness, we need a strong learning signal to “hill-climb” on. We share how we use evals as that signal, plus design decisions

1 replies

tomspark wrote:

honestly the harness abstraction is growing on me. thinking of eval-driven development like that Foundation series approach to psychohistory — you need the right metrics to predict what works at scale, but the tricky part is knowing which metrics actually matter versus which ones look good on a dashboard. curious how they're handling the problem of evals themselves becoming stale or gaming the system. that's always the wall you hit when you try to automate improvement loops.