Harrison Chase (LangChain): How to Build Agents That Improve Over Time

At Interrupt London, LangChain's agent conference, co-founder and CEO Harrison Chase explains how LangChain thinks about agents that improve over time. He splits an agent into three parts (the model, the harness and the context), shows how each can be improved, and argues that two things sit underneath all of it: traces and evals. He calls a fully self-improving agent a North Star rather than something that exists today.

This article is our own summary, with short quotes, from the video's automatic transcript. Watch the full video above for everything that was said.

The short version

• An agent is a model, a harness and context. The model does the reasoning. The harness (Deep Agents, Claude Code or Codex, for example) decides what the model sees, takes its actions, checks results and saves state. The context is the text, files, knowledge and tools it is connected to.
• Each part improves differently. Improvement can be manual, human-in-the-loop, or fully automated; Chase says nobody is at fully automated "by any stretch of the imagination."
• The model: update the weights, almost always started by a person because it costs time and money. Supervised fine-tuning works well for narrow tasks; reinforcement learning (RL) has a higher ceiling but no good managed offering yet.
• The harness: read traces, find issues, change the code, prompts or tools, with a person reviewing because changes reach every user. LangChain's LangSmith Engine automates the first steps.
• The context: the easiest to update because it is just text, and it can be updated per user or per organization as memory. He points to Hex and Profound as companies doing this well.
• Evals are "the new PRD." In AI there are so many edge cases that evals are often the best way to define what good looks like.

What he said, by topic

Three parts of an agent

Chase says you can't improve an agent until you define it. The core model is the workhorse; other models may sit at the edges for guardrails or evaluation. LangChain's open-source Deep Agents harness includes an execution environment (a file system and sandbox), built-in context management (skills, memory, summarization), human in the loop and subagents.

Improving the model

With supervised fine-tuning, you take traces, keep the ones where the agent did well and train on them. He sees it used for classification, extraction and summarization. LangChain recently launched LangSmith Fine-Tuning with smithtune, a command-line tool that pulls trajectories from LangSmith, filters and formats them, and hands them to providers such as Fireworks or Baseten, which return a tuned model. For the core model on complex tasks, a few teams use RL: give it tasks, run rollouts, score them, update the model and repeat. It is more complicated, but has a higher ceiling.

Improving the harness

The harness is code, prompts and tools. The process is to look at traces, identify issues and make changes, with a person in the loop. LangSmith Engine is a background agent that monitors traces, clusters them into issues, and proposes fixes: a pull request, new tests, new eval data points. He says Engine is "nothing more than a deep agent in a sandbox" with the LangSmith CLI and specialized subagents, skills and prompts.

Improving the context

Because context is text, it can change at the agent, user or organization level. Hex, a data analytics agent, has a unified context layer improved by a Review Agent that finds context gaps and suggests fixes. Profound builds company-level Brand Records and Personal Records from each user's conversations with its AI Marketer. LangChain's open-source OpenWiki builds an updatable "brain" for coding agents from your repository and from Codex or Claude Code traces. He says the hardest part is deciding what to remember, and that is domain-specific.

Traces as a new type of data

He defines a run (one model or tool call), a trace (a sequence of runs), a thread (a conversation of traces) and a trajectory (a standardized view of a thread). What matters is the exact inputs, outputs and order of events. Agent failures are semantic and logic failures that don't show up in traditional logs, which is why LangChain built SmithDB, its own database for agent observability, and launched AI Search, which scores each trajectory against a plain-language filter.

Why evals are hard

There is no silver bullet, he says, because evals must be domain-specific. Three things are hard:

1. Realistic inputs. You can't predict everything users will ask; tracing shows you.
2. Defining good. Exact match works for simple cases. For complex agents, teams write positive or negative criteria that can judge the final answer, the end state of the environment (a database or files) or the sequence of tool calls. This is best done by domain experts.
3. Simulation environments. You don't want evals writing to production systems, so teams use mock databases or dev environments, which is harder when the agent touches a system you don't own, like Salesforce.

Red teaming

Waiting for failures in production means rare problems are found too late, and domain experts testing by hand is costly. LangChain's recently launched red teaming looks for mistakes proactively. Since there is no universal judge of correctness, one technique sends two semantically similar questions and flags the agent when its answers differ, because one of them must be wrong. It also checks security and fairness, with the goal of a menu of checks that produce red-teaming reports.

The improvement loop

Run the agent, get feedback from a person or an online judge, inspect the trace, add an eval data point so you don't regress, then change the model weights, the harness code or the context. The North Star is an agent that notices its own mistake, creates its own eval data point and suggests its own fix.

What it means for your business

1. Log full traces from day one. Every way of improving an agent starts with exact inputs, outputs and steps.
2. Write evals before you scale. Treat them as the spec: what the agent must say, must not say and what state it should leave behind.
3. Start with context, not the model. Updating instructions and memory is cheapest; save fine-tuning for narrow, high-volume tasks.
4. Keep a person in the loop on harness changes. They reach every user at once.
5. Build a safe test environment. Agents that write to real systems need a sandbox for evals.
6. Bring in domain experts. Defining what "good" looks like is the hardest part, and they know it best.

Video details

• Title: How to Build Agents That Improve Over Time: Model, Harness, Context | Interrupt London
• Speaker: Harrison Chase, co-founder and CEO of LangChain
• Published by: LangChain on YouTube (recorded at Interrupt London)
• Watch: youtube.com/watch?v=xy_7SzwsIjM
• Note: quotes come from the video's automatic transcript and are lightly cleaned. Check the video for exact wording.

Want to put this to work?

We help owners turn AI trends into a plan, a product and a launch. Book a free call or see our services.