Volume VII · Memory and Many Minds

RAM and Disk, for a Language Model

A context window is fast and immediate but small, and everything a model was trained on sits frozen inside its parameters — hard to inspect, harder to update, impossible to verify against a source. Both problems are old ones in computing, and both have the same shape as a program whose working set outgrows its RAM. The mental model for this lecture is that literal: a language model's context is RAM, and everything outside it is disk.

The mental model

RAG is a program calling out to disk for a page it doesn't have loaded — a fixed retrieve-then-generate routine the system runs on the model's behalf. MemGPT is more radical: it gives the model itself the function calls to decide what gets paged in and out, the way an OS scheduler manages memory, rather than a fixed external heuristic making that call for it.

RAG: retrieval as a fixed routine

Retrieval-Augmented Generation (Lewis et al., arXiv:2005.11401, NeurIPS 2020) combines two kinds of memory in one system. The first is parametric memory — everything baked into the weights of a pretrained sequence-to-sequence generator, fast to consult but opaque: there is no way to point at which weights hold a given fact, no way to correct one fact without retraining, no way to check a generated claim against its source. The second is non-parametric memory — a dense vector index built over a large text corpus (Wikipedia, in the original paper) and searched by a pretrained neural retriever. The two are fine-tuned together, end to end: retriever and generator are trained as one system rather than bolted together after the fact.

The mechanism at inference time is retrieve, then generate: given a query, the retriever pulls the passages its index scores as most relevant, and the generator conditions its output on those retrieved passages in addition to the query itself. The paper reports state-of-the-art results, at the time of publication, on knowledge- intensive NLP tasks, and output that is more specific, more factual, and more diverse than a purely parametric baseline produces on its own — because the generator now has real text in front of it to draw from, rather than only whatever the weights happened to absorb during training.

Figure 15a The query goes to a retriever searching a fixed external index; the retrieved passages are joined with the original query and handed to the generator. The routine itself never changes — retrieval always happens the same way, for every query.

MemGPT: paging as a function the model calls itself

MemGPT (Packer et al., arXiv:2310.08560, "MemGPT: Towards LLMs as Operating Systems") takes the RAM/disk metaphor and makes it the literal design, calling the result "virtual context management" and modeling it explicitly on OS-style virtual memory paging. In this scheme, the LLM's context window plays the role of RAM: fast to use, strictly limited in size. Two external stores play the role of disk. A recall store holds message history, and an archival store holds a searchable long-term text database. Between them and the context window sits a system managing when data moves — but the part that makes MemGPT more than a scripted retrieval loop is who decides: the LLM itself issues the function calls that move data between main context and external storage, and the system relies on interrupts to manage when that paging happens, rather than a fixed external heuristic making the call on the model's behalf the way RAG's retrieval step does.

Figure 15b Context window as RAM; recall store and archival store as disk. Both arrows into and out of the context window are function calls the LLM itself issues — the system manages the interrupts that govern when paging happens, but the decision of what to page is the model's.

Where the name went next

MemGPT became the basis for the Letta framework and company — a case where a research paper's architecture became the founding design of a piece of production infrastructure, rather than remaining a benchmark result.

Short-term and long-term, as a convention, not a citation

It is common to hear an agent's memory described as "short-term" versus "long-term" — the context window playing short-term or working memory, bounded strictly by context length, and retrieval systems such as a vector store or memory buffer playing long-term memory, queried only as needed. Worth stating plainly: no single canonical paper defines this dichotomy for agents specifically. It is a widely-used convention that both RAG and MemGPT can be described through, not a result either paper claims credit for. Read it as a useful frame borrowed from human memory research and popular usage, not as an attributable finding.

Read the primary source

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al. (arXiv:2005.11401, NeurIPS 2020). The parametric-plus-non-parametric memory design, fine-tuned end to end.
MemGPT: Towards LLMs as Operating Systems, Packer et al. (arXiv:2310.08560). Virtual context management: context window as RAM, recall/archival stores as disk, paged by the LLM's own function calls.

❦