Steepish Descent

“Notes for your future self” is an effective zero-shot pattern for continual learning. Can it be improved with training?

Massively parallel CLIs (“agentic swarms”) are pushing the envelope of automation of coding, both along the multi-agent axis, but also along the task-length (horizon) axis. Between this and Moltbot, the case for continual learning has never been better: the myriad of deployment scenarios that models experience is extremely hard to anticipate up front and it’d be much better if they learned on the job.

For agentic swarms, practitioners have been learning to manage this complexity and a useful pattern has emerged: continual self-managed note taking. There are many many examples of this, but here’s a few for reference: a KarelDoostrinck post, a YoungSatoshi post, and a giffmana post.

I find the zero-shot effectiveness of this technique remarkable. Kareldoostrinck states “Codex consistently gets better and faster at tasks I use it for, just because I have the habit of asking it to take notes and improve. While working, codex commits notes and helpers to my personal folder in our monorepo. After a few interactions with a new part of the codebase, these helpers tend to stabilize. I’ve never actually read these notes, their utility to me is purely the effect on codex’s performance.“

This is analogous to chain of thought, which exhibited a clear zero-shot benefit from saying “Let’s think step-by-step”. When researchers leaned into this, the result was reasoning models. So it makes sense to ask: how can we learn to do better continual note-taking?

Challenge: storage reward is not episode-local

Policy gradient has been a workhorse lately, so let’s envision training a continual note-taker with it, using the standard ToolRL-style interleaved episode.

The retrieval tool call is causal for the episode reward, so policy gradient can presumably optimize the retrieval (i.e., contextual construction of the key). Maybe there’s better ways to do it, but first pass this seems reasonable.

However the storage tool call is not causal for the episode reward, since it is literally “a note for your future self”. Training it on the current episode reward at best would do nothing and at worst would reinforce incorrect spurious associations. Thus the Memory-R1 and Mem-$\alpha$ papers both assess memory quality via downstream task performance, respecting causality. I’ll call this approach prospective evaluation.

An alternative approach would be retrospective evaluation. This would look backward over previously completed tasks and counterfactually assess whether a particular memory update would have induced better historical performance, had it existed in the past. Online learning is built upon in-hindsight analysis, so maybe this could work, especially in simulation / offline settings where previously encountered problems can be “replayed”.¹ I haven’t found any papers using this approach, however, so maybe it’s a bad idea.

My gut says prospective evaluation is easier to get working, but retrospective evaluation might be more sample efficient.

Making prospective evaluation work

Superficially this seems straightforward: you just chain together problems in a temporal sequence to prospectively evaluate the memory storage operations.

Unfortunately the reward for the storage tool is very sparse: if a stored (key, value) pair is never retrieved, there is no learning signal. Said differently, if the number of stores is equal to the number of retrievals, the average signal per memory is low. The classical solution to sparse reward is a value function, but why learn a value function tabula rasa? We’re all about leveraging pretrained priors to reduce sample complexity nowadays, so using an LLM as a memory judge is sensible.²

Agentic Context Engineering

In the ACE paper, they leverage an LLM memory judge and a semi-structured memory format (“dynamic cheatsheet”) to get zero-shot dense prospective evaluation. Essentially, an LLM adds items to the memory, and in future tasks these items are scored for their usefulness to the current task. There’s no retrieval step per se, the entire memory is retrieved each time, but it is actively curated, with poorly judged entries removed. They get great results from this setup zero-shot (no fine-tuning); perhaps it can be further optimized with RL.

The “retrieve everything all the time” strategy makes the reward dense, but does it scale? If the notes are localized (e.g., per repository, per developer) then perhaps yes, since $10^4$ tokens is 10-15 pages of text and not much overhead. At $10^6$ tokens of notes, retrieval would be necessary, but an explore subagent could treat this another large artifact like the repository itself, in exchange for sparser updates on the memory.

Parenthetically, the effectiveness of properly architected zero-shot actor-critic architectures, where everything is “software 3.0” and nothing is fine-tuned, makes you wonder if RL will be less important going forward. It’s true that “Let’s think step by step” was amplified radically by RL, but that was a general technique with broad applicability. For something to be “worth RL-ing”, it will need similarly broad impact. Continual note-taking is imho general enough to qualify, but many patterns are not: in practice most solutions will get most of the upside with limited effort (prompt/context/architecture tweaking) and call it a day.

Engrams

The Engrams paper takes a different approach: the retrieval is fixed, the stored values are continuous-valued, and the gradient with respect to the current task performance is used to update the continuous value. Although the Engrams paper is operating at a much lower level (pretraining), the ideas could be applied to continual note-taking. For example:

• Fixed retrieval: an explore agent which is given the prompt can retrieve a subset of the notes at the beginning of the episode.
  • Technically, since retrieval is causal, we could relax this and let the agent self-schedule retrieval, but without bothering to tune the retriever.
• “Continuous” stored values: instead of continuous, we use text. Instead of imposing structure like the ACE paper, one could just bitter pill it and not specify the structure.
• Update based upon “gradient”: Here we want to compute a “text gradient”, which can be done with frameworks like Trace, or just by prompting a reflector to suggest a change in the notes. If a more general text gradient is computed, it needs to be applied back to the memory, so the retrieval system needs to return enough information to localize the source of the retrieved notes and apply the change (without corrupting other notes).

In this lens, The Reflector-Curator pair in the ACE paper was a software-3.0 text gradient update leveraging their particular structure (bulleted lists): but it also took “small steps” akin to a small learning rate in gradient descent, because it accumulated evidence over several trials and made incremental changes to memory. In no-regret algorithms, large changes from round-to-round can cause large regret, so conservatism in changes is good.

1. The terminology of prospective and retrospective is also used in Reflective Memory Management paper, but differently: in that paper “prospective” = “storage” and “retrospective” = “retrieval”, and only the retriever is optimized via RL. ↩︎
2. An LLM memory judge could also make retrospective evaluation viable, even in settings where past episodes cannot be replayed, and perhaps this retrospective LLM memory judge could be itself trained prospectively (i.e., to make the retrospective evaluations of memories align with prospective evaluations). ↩︎