In-context reinforcement learning for coding
A stripped-down agent that reuses its own successful runs matched a hand-engineered harness on Terminal-Bench, but only with careful retrieval.
June 10, 2026
Most of what makes a coding agent good is engineering. Someone writes the prompts, builds the tools, tunes when it stops, and adds whatever scaffolding the benchmark rewards, then redoes most of it for the next domain. I spent a few months checking whether an agent could skip some of that by reusing what it had already done. On Terminal-Bench it can match a hand-built harness, but only if you are careful about what it remembers and when.
A good agent is mostly engineering
Building a strong coding agent today is mostly human labor. You write the system prompt, pick the tools, tune the logic that decides when the agent is done, and add scaffolding for whatever you are scored on. A plain plan-reason-act loop is a long way from a real leaderboard harness, and almost none of that distance closes on its own. None of it transfers either. A harness tuned for one kind of task is usually brittle on the next, so you rebuild it, and performance keeps scaling with expert hours rather than anything cheaper.
People do not work that way. An engineer who hits an ugly build failure rarely starts from scratch. They half-remember the last time they saw it, dig up an old PR or some internal doc, and adapt the fix. The experience is the asset. Most agents throw it away after every run: each task starts cold, with a long instruction sheet and no memory of the hundred similar tasks before it. Giving some of that memory back is the obvious thing to try.
What if the agent learned from its own runs?
In-context reinforcement learning (ICRL) is the cheap version of that idea. The agent saves the trajectories where it succeeded, and on a later task that looks similar it pulls a few of them back into context as worked examples. Nothing is retrained, there is no extra prompt to write, and no second model in the loop. The only difference between a weak run and a strong one is what is in the context window when the agent picks its next action. The idea is not mine and is not new: self-generated in-context examples lift decision-making agents, and growing archives of verified programs push code generation further.
Coding is where I would expect this to matter most. Agents tend to live in one codebase for a long time, with a fixed build system, house conventions, and the same handful of failure modes over and over. The next task usually rhymes with the last, and the commands that finally unstuck a flaky test or built a stubborn C extension are often exactly what you reach for a week later. If reused experience is ever going to stand in for engineering, this is the friendly case.
The catch is that the same experience can sink a run. A finished terminal task carries a reusable recipe, which is the appealing part, but that recipe can be quietly wrong somewhere else: a different package manager, the wrong working directory, a verifier checking something subtly different. Surface the wrong memory at the wrong moment and the agent will happily burn its whole budget on a command that stopped applying two tasks ago. So the question I cared about was timing and selection, not whether memory is a good idea in the abstract.
pip install -e . && pytest -qThe editable install resolves, the tests run, the verifier passes. The memory was the answer.
No setup.py, wrong working directory, a missing extra. The same line wastes steps. The memory was a trap.
The whole algorithm
It is worth seeing the method before the numbers, because there is so little of it. You keep an archive of steps from runs that passed. During a task, the agent reaches into that archive as it works. After a task, you write its trajectory back only if it succeeded.
db = [] # archive of steps from verified-correct runs
for task in tasks:
trajectory = []
while not done:
examples = retrieve(db, context) # in-context "experience"
action = agent(context, examples) # one frozen model, no weight updates
observation = execute(action)
trajectory.append((action, observation))
if verify(trajectory): # keep only runs that passed the verifier
db.extend(trajectory) # memory grows only from success
That is the whole thing. The agent improves only because retrieve puts relevant past steps in front
of it at the right moment, so every result below is really a question about that one function: what it
hands back, how often it runs, and where its archive came from. Get those wrong and the same machinery
that helps will hurt.
The setup
I ran everything on Terminal-Bench 2.089 containerized tasks built on the Harbor framework, spanning DevOps, scientific computing, cryptography, biology, and more. Scored as the fraction of tasks whose verifier passes., a benchmark of 89 tasks that look like the work coding agents actually get asked to do: building packages, scientific computing, data processing, cryptography, Git recovery, media processing, environment setup, and database repair. It is a good stress test for the trap above. The tasks are procedural enough that a past solution often carries a reusable command sequence, but varied enough that the same sequence can be exactly wrong somewhere else.
I compared a ladder of agents that differ only in interface and memory. All runs use Gemini 3.1 Pro Preview on the same 89 tasks, scored as the fraction solved.
- React: the classic plan, reason, act loop with text-parsed actions.
- Simple: a minimal native-tool agent. A short system prompt plus two tools,
execute_commandandfinish, and essentially no task-specific logic. -
Kira
A strong engineered harness in the Terminus-KIRA family: terminal-discipline prompting, explicit completion behavior, and a curated tool set. My stand-in for "manual engineering."
: a strong, hand-engineered Terminal-Bench harness with terminal-discipline prompting, explicit completion behavior, and a curated tool set. This is my stand-in for "manual engineering."
For external calibration, the public Terminal-Bench 2.0 leaderboard puts Gemini 3.1 Pro agents in a tight band on the same 89 tasks: Terminus-KIRA at 74.8%, Forge Code at 78.4%, and TongAgents at 80.2%. My Kira reproduction lands at 74.2% (66/89), right on top of the public Terminus-KIRA entry. That is reassuring, since the engineered baseline here is a real, independently built harness, not a strawman I propped up to knock down. The whole story hangs on a simple agent hitting a number a serious harness also hits, so it is worth knowing the number is legitimate.
One thing had to hold before any of this was a fair test of memory: the action interface. Parsing actions out of free text adds failures that have nothing to do with the task, so I gave the simple agent native tool calls. That single, standard change takes the baseline from 37/89 to 62/89, most of the way to Kira's 66/89, before any memory is involved. I am not claiming credit for it. It is exactly the confound I wanted out of the way, so any later gain comes from what the agent remembers, not how it types its actions.
Here is the headline comparison. The denominator is always 89 tasks.
| System | Solved | Score |
|---|---|---|
| React | 37/89 | 0.416 |
| Simple (native tools) | 62/89 | 0.697 |
| Kira pass@1 | 66/89 | 0.742 |
| Simple + sparse ICRL | 66/89 | 0.742 |
| Kira pass@4 | 75/89 | 0.843 |
| Simple + sparse ICRL loop | 75/89 | 0.843 |
The last four rows are the ones I care about. The simple agent with the right memory matches the engineered one at a single try (66 tasks), and again after a few retries at four tries (75 tasks), with none of Kira's hand-built machinery.
Naive retrieval backfires
Getting there took a few dead ends. The natural first move, and the one I made, is to drop retrieved trajectories into the context at every step. It backfires. I ended up sweeping eight retrieval designs, changing what I queried on, how I represented each example, and how often retrieval fired, and most of them landed under the no-memory baseline of 0.697.
The transcripts make the failure obvious. Retrieve too often or too literally and the example starts dragging in operational detail that fights with the task in front of the agent. Raw command histories are the worst: something that looks almost right is often exactly wrong once the package or directory has changed. Careless memory is just noise that costs extra steps.
What actually worked
Exactly one design beat the baseline, and it won by holding back everywhere. It is sparse, late, compact, and seeded.
- Compact, not verbatim. A raw trajectory is mostly logs, dead-end commands, and one-off
environment fixes. Instead of dumping a transcript, I
Nearest-neighbor retrieval
A FAISS cosine search (k = 1) over L2-normalized embeddings of each step's observation and reasoning. Only steps from verified-correct runs are in the index.
and render it as one short hint that the model is told to adapt rather than copy. - Late and sparse. The agent acts with no examples at first, and retrieval only turns on after it has local context, then fires at most once every third step. Examples are attached to the current model call only, so they never pile up in the conversation.
- Seeded from verified successes. The database is grown only from prior simple-agent runs that passed the verifier with reward 1.0.
Put together, the loop is unglamorous. The archive is built once from verified successes, and during a task the agent mostly works alone, dipping into memory on a fixed cadence for a single short hint.
The hint format is deliberately plain:
Relevant verified hints. Use as strategy, not commands to copy blindly:
- Similar task: <goal of nearest verified step>
Context: <observation at that step>
Useful next command: <action taken at that step>
And the per-task loop, with a single nearest neighbor and the every-third-step gate:
db = seed_from_verified(source_job) # keep only reward == 1.0, dedup by (job, task)
steps, last_text = [], ""
for t in range(max_steps):
hint = ""
if t >= 2 and t % 3 == 0:
query = build_query(instruction, last_text, steps[-4:])
step = db.nearest_step(query, k=1) # FAISS cosine over obs + reasoning
hint = compact_hint(step) # attached to this call only
action, last_text = llm(messages + hint)
obs = execute(action)
steps.append((action, obs))
if action == "finish":
break
if verify(steps):
db.add(trajectory) # successes feed future retrieval
The most telling number is what seeding alone buys. The same sparse, compact policy scores 0.640 when it grows its archive online from nothing, and 0.742 when it starts from verified simple-agent successes. A restrained schedule is necessary but not sufficient: the agent also needs an archive it can trust, and only then does it pull level with Kira.
| Variant | Representation | Schedule | Seeded | Score |
|---|---|---|---|---|
| Simple (no ICRL) | -- | -- | -- | 0.697 |
| Reasoning | full step | every step | no | 0.629 |
| History | full step | every step | no | 0.596 |
| Combined | full step | every step | no | 0.551 |
| Trajectory window | local window | on reasoning | no | 0.663 |
| Sparse compact | compact hint | every 3rd step | no | 0.640 |
| Sparse compact | compact hint | every 3rd step | yes | 0.742 |
Retries compound the gain
Memory is not the only lever. Letting the agent take another pass at the tasks it has not solved yet (the standard
pass@kRetry each unsolved task up to k times; the denominator stays fixed at 89. We report the cumulative fraction solved across passes.
setting) recovers a lot of capability the agent already had, as long as the bookkeeping stays honest: each pass only re-attempts the still-open tasks, and the denominator never leaves 89. Kira goes from 0.742 at one try to 0.843 by the fourth, the simple agent from 0.697 to 0.809, and the sparse ICRL loop also lands at 0.843, level with the engineered harness.
When does memory transfer?
The number I keep coming back to is that match: a stripped-down agent with nothing but retrieved experience hit the same Terminal-Bench score as a hand-built harness, at one try and again at four. The conditions behind it are the part worth keeping. Hold the action interface fixed, retrieve late and rarely, keep each example short, and only seed from runs you have already verified. Drop any one of those and the result quietly goes away.
It also explains why memory looks magical in some papers and useless here. When the domain is uniform, like generating GPU kernels, a solved program is a clean idiom you can paste forward, so hoarding successes just works. Terminal tasks are not like that. A solved one is usually a local repair bound to its environment, and it only helps once the current task has shown enough of itself to make the analogy safe. What gets punished here is not memory, it is reaching for it too soon.
There are caveats I will be upfront about. This is one model on one benchmark, and every configuration is a single run, so I trust the big swings (0.640 to 0.742 from seeding, 0.697 to 0.843 from baseline to full loop) far more than any single-row gap. The sparse schedule is a reasonable setting, not a tuned optimum. And I only ever retrieve from successes, which sidesteps the harder, more interesting problem of learning from failures.
The takeaway
A hand-built harness pays for performance up front, in human time: someone learns the domain, writes the prompts, designs the tools, and encodes when to stop. A retrieval loop pays for it gradually, and lets the agent do the paying, by banking its own successes and drawing on them later. On Terminal-Bench both routes end up in the same place, 75 of 89 tasks: one through careful engineering, the other through a clean interface plus a disciplined way to reuse what worked. The capability we normally buy with effort can also be accrued from experience.
The catch matters as much as the result. Experience only stands in for engineering when it is fed back carefully. Retrieve too often, too literally, or from an archive you have not checked, and the same mechanism that matched the harness will pull the agent under its own baseline instead. So the craft does not vanish, it moves: less of it goes into the harness, more into deciding what the agent should remember and when it is allowed to look. For coding agents that keep returning to the same codebase, that is a trade worth making, and the direction I am most curious to push on next.