The AI Diary: AI Memory

Apr 22, 2026

Today’s article will be a bit technical and will help you better understand how AI integration in your business should operate and what it should look like.

Each LLM (Large Language Model), commonly known as an AI model, has a specific parameter called the context window.

A context window is the maximum amount of text an LLM can “hold in mind” at a given moment and process at once.

More formally, it is the model’s short-term working memory for a single request. This includes the chat history, uploaded documents and your latest prompt. All of this must fit within the token limit defined by the context window.

The larger the context window, the more data the LLM can process at once.

The Problem

When working with LLMs (AI models), if I provide too much context, the LLM may exhibit the following behaviors:

• Forgetting things (failing to retain or reference key information)
• Giving wrong answers (providing inaccurate or unsupported responses)

Excessive context can confuse LLMs through multiple mechanisms:

• Attention mechanism limitations. The attention mechanism has quadratic complexity (O(n²)), meaning computational demands grow exponentially as context length increases. This causes severe performance degradation when context approaches model limits — typically dropping below 50% accuracy at approximately half the training context length. Source: https://arxiv.org/abs/2209.04881
• “Lost in the Middle” problem. LLMs exhibit a characteristic U-shaped performance curve where they perform best when relevant information appears at the beginning or end of long contexts, but accuracy degrades significantly when critical information is positioned in the middle sections. Source: https://arxiv.org/abs/2307.03172
• Context distraction. As context grows, models struggle to properly attend to all information, becoming “distracted” by excessive content and losing focus on the most relevant details. Source: https://arxiv.org/html/2503.23306v1

First Though Experiment

Imagine me walking down the street.

While I’m walking, I don’t keep in my working memory the poem I read 10 months ago or the conversation I had with a friend two years ago — that would be overwhelming. Even if I had access to all that knowledge at once, I’d feel “crazy”. If someone stopped me and asked a question, I might give a strange, possibly incoherent answer.

Instead, the brain works differently. When someone asks a question, it’s like calling an internal LLM that checks an index of records. That index has two columns (very similar to an Excel sheet): the first contains keywords, and the second contains references to knowledge stored in my brain.

For example, if the question is about how to cook potatoes, I scan the left column for keywords like “cook”, “potatoes”, or “cook potatoes”. If I find them, I know which knowledge sections to load into my working memory so I can answer correctly. I can further refine the lookup to find not just all sources about cooking potatoes, but the particular one that will help me answer in the best possible way.

This index gives me a crucial advantage: I don’t need access to the entirety of my knowledge at once. I only need to know where to find it. Once I know where it is, I can read it and load it into memory. This is the moment we describe as “let me try to remember” — we’re searching our internal index.

Once my brain finds the relevant knowledge and loads it as context, that context becomes the input to a second “LLM call” that produces the answer.

First Statement

Rather than loading the entire context into the LLM, we should build an index that points to distinct knowledge sections. Different AI tasks will require different knowledge, so we don’t need everything at once.

Second Though Experiment

Imagine asking an LLM to build a car by providing every step and detail at once. We might expect it to figure out everything from the supplied knowledge, but as we know, the chances of success diminish as the context grows.

An alternative approach is to prepare multiple prompts, each with its own focused knowledge base. All of these small prompt chunks align with the ultimate goal — building a car — but instead of giving the entire picture at once, we instruct the LLM to complete one small step at a time. For example: “Design a bolt with these parameters”.

The LLM doesn’t need to know why we need the bolt; it only needs the parameters to design it. That output then becomes input for the next task.

Second Statement

We should avoid thinking of AI as a monolithic solution (a single AI model) that solves a complex problem by ingesting a large context all at once.

Instead, we should break our work into multiple AI agents, each performing small, well‑defined tasks that, when combined, produce a refined final result. The user experience should feel like a single monolithic AI agent; however, behind the scenes, multiple agents may be working together.

Example tasks:

• Read a brief set of instructions about the goal.
• Perform research on the Internet (if necessary).
• Combine the knowledge gathered so far and brainstorm potential solutions (internally, possibly between multiple AI models).
• Refine the result into a single, concise output.
• Execute further actions.

Memory is Everything

So, it turns out that simply giving a large amount of data to an AI model and asking questions is not the most efficient way to get the best results.

Designing a proper memory solution — a way to store data and efficiently retrieve it when needed — allows us to work with datasets far beyond the model’s context window and, as a result, solve more complex problems.

Additionally, when implementing such a solution in a business context, as you’ve seen, building a memory system inevitably leads to the need for multiple AI agents working together as a team.

My conclusion is that, due to the nature of AI models, effectively integrating AI into your business requires three key components:

• A robust memory solution;
• A team of AI agents that delegate and handle subtasks collaboratively;
• An AI agent acting as an orchestrator for all other agents, serving as a middle layer between the human and the AI system.

Conclusion

Integrating AI solutions into your business is not as simple as using a single tool like ChatGPT to handle complex tasks.

Effective implementation requires a well-designed AI infrastructure tailored to your specific business needs that involves a memory solution and a team of AI agents.