Memory Scaling: how Databricks proves AI agents get better as they accumulate memory
Memory Scaling is one of the most interesting concepts to emerge from the AI agent ecosystem lately. Databricks just published experimental results showing how AI agents can consistently improve as they accumulate memory, and the numbers are pretty convincing. 📊
But here is the point that makes all of this even more fascinating: more memory, on its own, guarantees nothing. Storing a lot of stuff does not mean you are using what was stored effectively, and that is exactly the challenge Databricks decided to tackle head-on with their latest experiments.
For a long time, the debate around AI Agents revolved almost entirely around more powerful models, smarter prompts, or more reasoning capacity during inference. What Databricks is proposing now is a different axis entirely: what if the agent just got better with use? Not through retraining, not through manual tuning, but because it learns from real conversations, user feedback, and organizational context accumulated over time. 🤖
This is a major shift, especially for anyone working with AI in enterprise settings where tacit knowledge is abundant and a single agent might serve hundreds of people with completely different needs. In the sections ahead, we dive into the experiments, the types of memory that actually matter, the infrastructure required, and what all of this means for the future of the agents we already use every day.
What is Memory Scaling and why it matters so much
Memory Scaling is the property that an AI agent’s performance improves as its external memory grows. Here, memory does not refer to the model’s weights or the current context window, but to a persistent store of information the agent can query during inference. Think of it as the difference between someone who has to memorize everything off the top of their head and someone who has an organized notebook and knows exactly where to look for each piece of information.
This concept sits alongside parametric scaling, which is basically training ever-larger models, and inference-time scaling, which involves giving the model more time and resources to reason during response generation. Memory Scaling addresses domain knowledge and grounding gaps that neither model size nor reasoning capacity can fill on their own.
And the benefits go beyond response quality. When an agent has already memorized the relevant schemas, domain rules, or successful actions from past interactions, it can skip redundant exploration and resolve queries faster. In Databricks’ experiments, gains showed up in both accuracy and efficiency, which is particularly relevant for anyone concerned about operational AI costs.
How it differs from continual learning
A natural question is: is this not the same thing as continual learning? Not really. Continual learning focuses on updating the model’s parameters over time, which works well in limited scenarios but becomes computationally expensive and fragile when there are many simultaneous users, multiple agents, and projects that change quickly.
Memory Scaling asks a different question: does an agent with thousands of users perform better than one with a single user? By expanding the agent’s shared external state while keeping the LLM weights frozen, the answer can be yes. A workflow pattern learned from one user can be retrieved and applied for another immediately, with no retraining whatsoever. This is a property that continual learning, focused on parameter updates for a single user, simply was not designed to offer.
How it differs from long context
Large context windows might seem like a substitute for memory, but they solve different problems. Stuffing millions of raw tokens into a prompt increases latency, drives up computational costs, and degrades reasoning quality since irrelevant tokens compete for attention. Memory Scaling uses selective retrieval, deciding not just how much context to include, but what to include, surfacing only the high-signal information relevant to the current task.
The types of memory that actually make a difference
Not all memory carries the same weight within an AI Agents system. Databricks identified two fundamental distinctions that matter in practice:
Receive the best innovation content in your email.
All the news, tips, trends, and resources you're looking for, delivered to your inbox.
By subscribing to the newsletter, you agree to receive communications from Método Viral. We are committed to always protecting and respecting your privacy.
Episodic memory versus semantic memory
Episodic memory consists of raw records of past interactions: conversation logs, tool-call trajectories, and user feedback. Semantic memory represents generalized skills and facts distilled from those interactions. For example, a semantic memory might record that users in a certain workspace always mean fiscal quarter when they just say quarter. Each type requires different storage, processing, and retrieval strategies: episodic memories for direct retrieval and semantic memories distilled by an LLM for broader pattern matching.
Personal memory versus organizational memory
Some memories are specific to a single user’s preferences and workflows. Others represent shared organizational knowledge, like naming conventions, common queries, and business rules. The memory system needs to route retrieval and updates appropriately: sharing organizational knowledge broadly while keeping individual context private, respecting permissions and access controls. This balance between personalization and privacy is one of the most delicate challenges in the architecture.
What the Databricks experiments show in practice
Databricks used MemAlign, a memory framework for AI agents, to run their experiments. MemAlign stores past interactions as episodic memories, uses an LLM to distill them into generalized rules and patterns (semantic memories), and retrieves the most relevant entries during inference to guide the agent.
The tests were conducted on Databricks Genie Spaces, a natural language interface where business users ask questions about data in plain English and receive SQL-based answers. The goal was to measure how agent performance scales as it receives more memory, using two data sources: curated examples with labels and raw user conversation logs.
Scaling with labeled data
MemAlign was evaluated on unseen questions distributed across 10 Genie Spaces, incrementally adding chunks of annotated training examples to the agent’s memory. The baseline was an agent using expert-curated Genie instructions, with hand-written table schemas, domain rules, and few-shot examples.
The results showed consistent scaling along two dimensions:
- Accuracy: Test scores increased steadily with each additional chunk of memory, climbing from near zero to 70%, surpassing the expert-curated baseline by roughly 5%. In the analysis, human-labeled data proved more comprehensive and therefore more useful than hand-written table schemas and domain rules.
- Efficiency: The average number of reasoning steps per example dropped from approximately 20 to around 5 as memory grew. The agent learned to retrieve relevant context directly instead of exploring the database from scratch, approaching the efficiency of hardcoded instructions, which hovered around 3.8 steps.
The effect is cumulative: since the memorized samples span 10 different Genie Spaces, each chunk contributes cross-domain information that builds on the knowledge already accumulated.
Scaling with unlabeled user logs
The next question was: can memory scale with noisy real-world data? To find out, the team ran MemAlign on a live Genie Space and fed it historical user conversation logs with no reference answers. An LLM judge filtered those logs by usefulness, and only the high-quality ones were memorized.
The scaling curve followed a similar pattern, with an even steeper start:
- Accuracy: The agent showed a sharp initial gain. After the first chunk of logs, it extracted key information about relevant tables and implicit user preferences. Performance jumped from 2.5% to over 50%, surpassing the expert-curated baseline (33.0%) after just 62 log entries.
- Efficiency: Reasoning steps dropped from approximately 19 to about 4.3 after the first chunk and remained stable. The agent internalized the space’s schema early and avoided redundant exploration on subsequent queries.
The takeaway is clear: uncurated user interactions, filtered only by an automated referenceless judge, can replace manually crafted domain instructions, which are costly and time-consuming to produce. This points toward agents that continuously improve from normal usage and can scale beyond the limitations of human annotation. 🔄
Organizational Knowledge Store: knowledge that existed before the first user
The experiments above show how Memory Scaling happens through user interactions. But companies also have knowledge that predates any interaction: table schemas, dashboard queries, business glossaries, and internal documentation. Databricks tested whether precomputing this organizational knowledge into a structured memory store could improve agent performance.
The pipeline processes raw database metadata into retrievable knowledge in three stages: extracting information about assets, enriching assets via additional transformations, and indexing the enriched content. At query time, the agent can fetch business context through keyword search or hierarchical navigation. This bridges the gap between how business users phrase questions, for example AI consumption, and how the data is actually stored in specific column names within specific tables.
Adding the knowledge store improved accuracy by approximately 10% across both benchmarks evaluated. The gains were concentrated on questions that required vocabulary bridging, table joins, and column-level knowledge, meaning information the agent could not have discovered through schema exploration alone.
From a competitive standpoint, this changes the equation quite a bit. Companies that invest early in this kind of memory infrastructure are not just buying a smarter agent today. They are building a cumulative advantage that grows over time. The more the agent is used, the more it learns. The more it learns, the more useful it becomes. And the more useful it becomes, the more people use it, creating a virtuous cycle that is hard to replicate.
The infrastructure required to make it all work in production
Memory Scaling in enterprise deployments demands robust infrastructure that goes well beyond a simple vector store. Databricks identified three core challenges that this infrastructure needs to address.
Scalable storage
The simplest memory store is the file system: markdown files in hierarchical folders, browsed and searched with standard terminal tools. This works fine at small scale and for individual users, but lacks indexing, structured queries, and efficient similarity search. As memory grows to thousands of entries across multiple users, retrieval degrades and governance becomes difficult to enforce.
Dedicated databases are the natural next step. Standalone vector databases handle semantic search well but lack relational capabilities like joins and filtering. Modern systems built on PostgreSQL offer a more unified alternative, natively supporting structured queries, full-text search, and vector similarity search within a single engine.
Databricks has been using Lakebase, built on Neon’s serverless PostgreSQL engine, thanks to its scale-to-zero cost model and support for both vector and exact search. Native database branching also simplifies the development cycle, allowing engineers to fork the agent’s memory state for testing without affecting production.
Memory management
Scalable storage alone is not enough. A memory system also needs to actively manage its contents:
- Bootstrapping: New agents suffer from cold-start problems. Ingesting existing enterprise assets like wikis, documentation, and internal guides through parsing and extraction provides an initial memory base that alleviates these issues.
- Distillation: Raw episodic memories are useful for direct retrieval but become expensive to store and search at scale. Periodically distilling them into semantic memories, which are compressed rules and patterns, keeps storage manageable and provides the agent with generalizable insights.
- Consolidation: As memory grows, keeping the system consistent, compact, and up to date is critical. This requires pipelines that remove duplicates, prune outdated information, and resolve conflicts between old and new entries.
Security and governance
Memory introduces governance requirements that simply do not exist for stateless agents. As agents accumulate deeply contextual knowledge, including user preferences, proprietary workflows, and internal data patterns, the same governance principles that apply to corporate data need to extend to the agent’s memory.
Access controls need to be identity-aware: individual memories should remain private, while organizational knowledge can be shared within access-controlled boundaries. Beyond access control, data lineage and auditability matter. When an agent’s behavior is shaped by its memory, teams need to trace which memories influenced a specific response and when those memories were created or updated. Compliance and regulatory requirements, particularly in regulated industries, demand that memory stores support the same observability guarantees as the underlying data.
The obstacles that still need to be overcome
Every scaling axis eventually hits its own bottleneck. Parametric scaling is constrained by the supply of high-quality training data. Inference-time scaling can degenerate into overthinking, where longer reasoning chains add cost without adding signal. Memory Scaling has analogous limits, centered on quality, scope, and access.
Memory quality is hard to maintain. Some memories are wrong from the start; others become wrong over time. A stateless agent makes isolated mistakes, but an agent with memory can turn a mistake into a recurring mistake by storing it and later retrieving it as evidence. Databricks reported cases of agents that cited notebooks from previous runs that were already wrong, and reused those results with even more confidence. Staleness is more subtle: an agent that learned last quarter’s schema might keep querying tables that have already been renamed or deleted.
Governance needs to extend to distillation. Scaling memory within an organization requires distilling repeated interactions into reusable semantic memories. But abstraction does not remove sensitivity. A memory like for company Y, join the CRM, market intelligence, and partnerships tables might look harmless while still revealing a confidential acquisition interest. Access controls and sensitivity labels need to survive distillation, not just ingestion.
Useful memories can remain inaccessible. Even if memory is accurate and current, the agent still needs to discover that it exists. Retrieval is inherently metacognitive: the agent has to decide what to ask its memory store before knowing what is in there. When it does not anticipate that a relevant memory could help, it never formulates the right query and falls back to slow, redundant exploration. In practice, the gap between stored knowledge and accessible knowledge may be the primary limiter of Memory Scaling.
The agent as memory: Databricks’ vision for the future
The experiments and infrastructure described above point to a natural design pattern: an agent whose identity lives in its memory, not in its model weights.
In this design, the agent’s context is built from a persistent store hosted on a serverless database like Lakebase. The store contains three components: system prompts and agent capabilities (skills), structured and unstructured enterprise assets (knowledge), and episodic and semantic memories scoped at both the organization and user levels. Together, these components form the agent’s state: instructions, retrieved documents, relevant memories, SQL query execution results, API calls and other tool outputs, plus conversation history. This state is fed to the LLM at each step and updated after each interaction.
The LLM itself is a swappable reasoning engine: upgrading to a newer model is straightforward since the new model reads from the same persistent store and immediately benefits from all the accumulated context.
As foundation models converge in capability, the differentiator for enterprise agents will increasingly be what memory they have accumulated rather than which model they call. Hypothetically, a smaller model with a rich memory store could outperform a larger model with less memory. If that holds true, investing in memory infrastructure may yield greater returns than scaling model parameters. Domain knowledge, user preferences, and operational patterns specific to your organization are not in any foundation model. They can only be built through usage, and unlike model capabilities, they are unique to each deployment.
What changes in practice for everyday AI users
For anyone already working with AI Agent-based tools on a daily basis, the Databricks experiments point to an important shift in perspective: the agent should not be treated as a static tool that delivers the same result regardless of context, but as a system that evolves and needs to be fed in order to improve. This changes how teams interact with it, how feedback is collected, and how organizations think about governance for these tools.
On the performance side, the documented gains are encouraging but depend on careful implementation. Simply flipping on a memory mechanism is not enough. The quality of the memory, how it is retrieved, and how well the model can coherently integrate it into responses are the factors that determine whether Memory Scaling will actually deliver on its promise.
At the end of the day, what Databricks is signaling with these experiments is that the next big leap in quality for AI Agents may not come from bigger models or more computational power during inference, but from systems that accumulate context intelligently and learn from their own usage over time. It is an approach that looks more like how humans develop expertise than the traditional AI model most people have in mind, and that alone is a paradigm shift worth watching closely. 🚀
