MemGPT introduces virtual context management to provide LLMs with the illusion of extended context by paging data between main and external storage, enabling long-term conversations and document analysis.
- MemGPT uses an OS-inspired hierarchical memory system with main context and external context to manage data movement.
- The queue manager handles context overflow by evicting messages to recall storage and generating summaries, mimicking OS paging.
- MemGPT leverages function calls to allow the LLM to self-direct memory edits and retrieval, improving efficiency.
- In document analysis, MemGPT outperforms fixed-context baselines by querying archival storage iteratively, unaffected by document length.
- In conversational agents, MemGPT enhances consistency and engagement by retrieving long-term memories, outperforming baselines on deep memory retrieval and conversation openers.
Introduction
Large language models (LLMs) are limited by fixed-length context windows, hindering their use in long conversations and document analysis. Directly scaling context is computationally expensive and often ineffective.
MemGPT introduces virtual context management, inspired by OS virtual memory paging, to provide the illusion of infinite context. It uses function calls to let the LLM manage its own memory, paging data between main context and external context.
MemGPT Architecture
MemGPT's memory hierarchy consists of main context (the prompt tokens) and external context (archival storage and recall storage). The main context contains system instructions, working context, and a FIFO queue of messages.
The queue manager manages the FIFO queue and recall storage. When prompt tokens exceed a threshold, it inserts a memory pressure warning, allowing the LLM to store important information. On overflow, it evicts messages and generates summaries.
The function executor interprets LLM outputs as function calls, enabling self-directed memory edits and retrieval. This feedback loop allows MemGPT to learn and adjust. Function chaining enables multi-step retrieval by allowing immediate follow-up inference.
Evaluation
MemGPT was evaluated on document analysis and conversational agents. For document QA, MemGPT queries archival storage via function calls, allowing it to scale to large documents, outperforming fixed-context baselines that degrade with truncation.
In conversational agents, MemGPT was tested on deep memory retrieval and conversation openers. MemGPT significantly outperformed fixed-context baselines in retrieving specific facts from past sessions and crafting engaging openers that referenced persona information.
MemGPT also succeeded in nested key-value retrieval, a multi-hop lookup task, where baseline models failed at higher nesting levels, demonstrating its ability to combine multiple queries.
Read this at any depth.
Install Depth and pick your level — Glance for a sentence, Summary for the gist, Read for the full take. Free daily quota, no signup needed.
Add to Chrome