In my previous article, I shared my research on how RAG is positioned as a layer that helps companies use information in a more up-to-date, controlled, and explainable way. In this article, I wanted to focus on a slightly different point: Is RAG enough for this on its own? Can RAG really build relationships between pieces of information?
Retrieval-Augmented Generation, or RAG, is a significant architectural approach. It gives large language models a way to reach internal company documents, up-to-date information, and private data sources instead of relying only on what they learned during training. Yet when we examine how RAG actually operates, some limits of this structure start to show themselves.
How RAG works: the chunk–embed–retrieve cycle
At the center of RAG, there is a cycle that we can call chunk – embed – retrieve.
Let’s imagine that we give the system a report with hundreds of pages. The system does not try to handle this report as a whole and understand all of it. First, it divides the report into smaller parts. Then it matches these parts in a high-dimensional space with a mathematical method called cosine similarity and turns these parts into vectors. In other words, it represents texts mathematically in a high-dimensional space. When we ask a question, the system tries to find the parts that are geometrically closest to this question among the parts it created before.
This approach really works in many scenarios. Because it prevents the model from depending only on its own training data and allows it to access current information inside the company. But an important problem appears here.
Geometric closeness does not always mean logical relevance
Geometric closeness does not always mean logical relevance.
A study connected to Google DeepMind, called “On the Theoretical Limitations of Embedding-Based Retrieval” shows that single-vector embedding-based retrieval structures have theoretical limits. According to the research, the relevant document combinations that a query can return are limited by the embedding size. This means that vector search can struggle, especially when more than one concept, condition, or relationship must be evaluated together. So the problem is not only finding the right word, but also representing the possible combinations and relationships between different pieces of information.
To test this limit, the study builds a simple dataset based on “things a person likes.” For example, sentences like “Jon likes Hawaiian pizza, sports cars…” are created, and very simple questions like “Who likes Hawaiian pizza?” are asked to the system. Normally, this task looks quite easy. However, the study shows that single-vector embedding-based retrieval structures can reach their limits even in these simple relationship matching tasks.
This example looks simple, but this problem becomes much more complex in company data. We probably do not even want to imagine this from any manager’s point of view.
Because inside a company, information rarely lives in a single document, a single sentence, or a single piece. One decision influences another project. An incident may be the outcome of a technical choice made earlier. A customer request can reshape roadmap priorities. A rise in costs may be tied not only to budget, but also to maintenance load, team capacity, or technical debt.
RAG may be able to find these pieces. But it cannot always build the relationship between these pieces by itself.
The retrieval-generation gap: finding the right information is not enough
Another important topic is the retrieval-generation gap.
This means that even if the system finds the correct information, it is not guaranteed that this information will be produced as the correct answer. Even when it finds the truth, it can still hallucinate. This happens because of the structural mismatch between the search part and the language generation model.
Sometimes, for a footnote to make sense, you need to read the three pages before it. Sometimes, to understand what a decision means, you need more than the decision sentence. You also need the meeting where the decision was made, the previous problem, the later effect, and the historical context.
For example, let’s say we ask a question about the profit rate in the first quarter. RAG may bring similar parts from different places in the report and put them in front of the LLM. But the real answer may actually be in a small footnote on the third page. And that footnote may mean nothing on its own without reading the previous three pages. In this case, the system looks like it is getting closer to the information. But it may not build the real answer together with its context.
For this reason, I think RAG on its own can sometimes stay like a word or meaning similarity matching game. Of course, I am not talking about a structure as simple as classical keyword search. It can build semantic closeness. But still, this does not mean that it builds a real relationship between events.
The detective board: RAG finds the photos, not the red strings
As a result, I think the example of a homicide detective board is very useful for explaining classical RAG.
Imagine a board with photos of suspects, crime scenes, dates, and documents. These photos are fixed to the board with pins. The relationships between them are shown with red strings. RAG can find and bring the related photos on this board. But it cannot always draw the red strings between those photos by itself.
Those red strings, meaning the relationships between events, decisions, people, projects, dates, and results, must be modeled separately in the background of the system. Because company knowledge is usually not made only of documents.
- Which project did a decision affect?
- Which operational cost did a technical debt turn into?
- Which architectural choice was an incident connected to?
- Which roadmap change did a customer request cause?
- Which process problem was behind a drop in a team metric?
These questions may not be answered only by finding similar text pieces.
So is RAG enough on its own?
It would be wrong to call RAG useless. RAG is a very valuable starting point, especially when it comes to connecting company knowledge to artificial intelligence systems. It is a strong layer for generating answers that are more current, more controlled, and more explainable.
But seeing RAG as the final solution on its own feels incomplete to me. Because RAG makes access to information easier, but it does not always make information meaningful.
So I think the main question in enterprise AI systems should not only be:
“Are we using RAG?”
A better question may be:
Can we turn the information brought by RAG into a truly meaningful, connected, and reliable structure?
Because on the enterprise AI side, the real value is not only in finding the right document. The real value is in understanding which decisions, processes, teams, and results that document is connected to.
RAG is an important layer in this journey. But it does not complete the whole picture on its own.
In my next article, I want to pick up from this point and discuss how we can hand these red strings to the system. I will look at how data modeling approaches, Modern RAG structures that move beyond classical RAG, GraphRAG, knowledge graph, and hybrid retrieval approaches address this problem.