Beyond RAG: Context Engineering and the Compilation-Stage Knowledge Layer

TL;DR — I'm not saying RAG is dead; but from what I see in the field, the golden age of flat vector RAG is over. What saves projects in enterprise settings today isn't picking a bigger model; it's building knowledge in the right layer. In this piece I trace the evolution from flat vector RAG to agentic RAG, and onward to "context engineering" and the compilation-stage knowledge layer. My core thesis: the enterprises pulling ahead in 2026 are moving reasoning from query time to compile time; that is, before the model ever sees your question, the knowledge is already interpreted, structured, and wrapped in governance, waiting. On the KVKK (Turkey's data protection law) and EU AI Act side, this turns "which data you put in which layer" from a technical question into a legal one.

In most companies I consult for, the first RAG project starts with the same story. Someone stands up a vector database over a weekend, chunks the documents, generates embeddings, and adds a chat interface. The first demo enchants everyone. Then that system meets real users, real questions, and real documents, and everything goes off the rails.

Why? Because from the start we thought of RAG as a simple function: "Take a question, fetch relevant chunks, hand them to the model, get an answer." But retrieval was never a deterministic, single-step function. This is the most common misconception I see in the field, and it lies at the root of nearly every failed project.

Let me give you a concrete example. At a manufacturing company, we built an assistant for maintenance teams. When a user asked "How do I fix the pressure drop on the pump on line 3?", flat vector RAG returned chunks semantically close to "pressure" and "pump." But the correct answer lived in a fault-log table and a PLC log file; neither is capturable by free-text embedding. The system produced confident, fluent, and wrong answers. This "confident but unreliable" behavior is the most dangerous disease of enterprise RAG.

Over dozens of projects across the years, we hit the same three walls. I want to lay them out clearly, because the logic of the solution runs through understanding these limits.

1. A fixed pipeline isn't flexible. In traditional RAG, the flow is always the same: embed → search → rerank → generate. But real questions come in every shape. Some want a single fact, some a comparison, some a change over time, some a calculation. A single-type pipeline answers non-uniform questions with uniform errors.

2. Semantic similarity is not correctness. Vector similarity means "close in meaning"; it does not mean "correct source." Two paragraphs can be very close in meaning while one is current and the other is a procedure canceled two years ago. A similarity score doesn't tell them apart.

3. An ungoverned vector store is a black box. When you dump everything into a flat vector store, you can't trace where each piece of data came from, who may access it, or when it was updated. This is an outright minefield under KVKK, which I'll detail shortly.

The first serious answer to these three limits was agentic RAG. The idea is simple but transformative: you stop making retrieval a fixed pipeline and give the agent decision-making autonomy. The agent now answers these questions itself:

• Which tool should I fetch with? Keyword search, semantic search, an API call, or a direct SQL query? On the pressure question, the agent should be able to choose SQL against the fault table instead of free text.
• When should I fetch? Some questions need no retrieval (the model already knows); others need multiple rounds. The agent must tell the difference.
• Is what I fetched enough? The agent should look at the results and say "this is insufficient, let me check that source too." We usually call this reflection.

The difference agentic RAG brings in the field is clear: the system is no longer one-shot, but iterative. When we moved the maintenance assistant above to an agentic architecture, the agent first extracts context via semantic search, then verifies the identified equipment code with a SQL query against the fault table, and calls the PLC log API when needed. Accuracy rose dramatically in the scenarios I measured.

But to be honest, agentic RAG is no magic wand. It comes with costs:

Dimension	Flat Vector RAG	Agentic RAG
Flexibility	Low (fixed flow)	High (dynamic decisions)
Latency	Low, predictable	High, variable
Cost	Low	High (multi-turn calls)
Debugging	Easy	Hard (long decision chain)
Accuracy potential	Limited	High (can pick sources)

This table is the thing I show clients most in consulting. Because giving the agent autonomy also means unpredictability and cost. Not every question needs the full agentic flow every time; sometimes a system that makes five tool calls for a simple question is slow, expensive, and unnecessary.

Agentic RAG improved decisions, but another problem remained: a flat vector store loses the relationships between pieces of knowledge. Multi-step relational questions like "Which parts does this supplier provide, which products use those parts, and which customers received those products?" can't be solved with embedding similarity.

From this, graph-augmented architectures were born. A knowledge graph models entities and the relationships between them explicitly. Retrieval can now run not just over "similar text" but over "related nodes." The strongest enterprise RAG systems I've seen in the field were hybrids blending vector search with graph traversal. Vector answers the "which topic" question, the graph answers the "which relationship" question.

At an energy company in Turkey, after we modeled the dependencies between field equipment in a graph, we could answer questions like "if this transformer fails, which feeders are affected" within seconds and in a traceable way. This would never have been possible with flat vectors, because the answer wasn't in a single document but in a web of relationships.

Now I come to the heart of the matter. The concept that has matured most in the industry over the past year is "context engineering," and I'd ask you not to see it as a buzzword. Context engineering means designing the context sent to the model as an engineering discipline. It's far broader than writing prompts.

The scope of context engineering is this:

1. What to fetch? The right chunks from the right sources.
2. How to chunk? A chunking strategy that preserves semantic integrity, neither too small nor too large.
3. How to rank and prune? The context window is limited; placing the most relevant information in the most efficient way.
4. How to structure? Giving the model not a raw text pile but structured, tagged, source-attributed context.
5. How to govern? Access control, source quality, freshness, auditability.

Notice: most of these items are work to be done before the model runs, not while it runs. This observation led me, and the industry, to the next big idea.

In traditional RAG, most of the reasoning happens at query time. The moment the user asks, the system runs, fetches, interprets, structures, and answers. This means redoing the same interpretation work from scratch on every query. It's slow, expensive, and inconsistent.

The emerging idea is this (some teams reference it with a "Nexus"-like approach): move the work of interpreting and structuring knowledge from query time to compile time. That is, before any agent question arrives, run a "compilation" stage that takes the raw knowledge source, interprets it, extracts its relationships, cleans up its contradictions, normalizes its entities, and builds a pre-structured knowledge layer.

Let me explain with an analogy from software, because this analogy causes a lightning flash in my clients: it's like the difference between an interpreted language and a compiled one. Flat RAG is an interpreter that re-interprets every query. The knowledge-layer approach is a compiler that pre-compiles and optimizes knowledge in advance. The agent no longer reasons from scratch each time; it sits on a pre-compiled, cleaned, related knowledge foundation.

What does this change?

• Consistency: Because interpretation is done once, centrally, every query sits on the same clean foundation.
• Speed: Less work at query time; the heavy interpretation is finished at compile time.
• Governance: The knowledge layer becomes a single auditable asset; you centrally manage what goes in, who accesses it, and when it was updated.
• Quality control: Contradictions, stale records, and duplicate information are caught at the compilation stage; no garbage reaches the user at query time.

Let me share my clearest field observation. In 2025, everyone asked "which model?" GPT, Claude, open source, how many billion parameters? In the enterprises succeeding in 2026, what I see is a full mindset shift: they make the primary investment not in the model, but in the knowledge source.

The logic is very sound. Models are rapidly commoditizing; the leading model today becomes average in six months and its price falls. But your enterprise knowledge, your cleaned, structured, related knowledge layer wrapped in governance, is your true competitive advantage. You can swap the model; your knowledge layer stays and gains value over time.

That's why in consulting I now say this: rent the model, own the knowledge layer. The bulk of your budget should go to data quality, source management, the knowledge graph, and the compilation pipeline; not to chasing the latest model.

Let me list the pitfalls I've accumulated over the years, the ones that recur in nearly every failed project. I recommend using them as a checklist.

Pitfall 1 — Treating retrieval as a simple function. I touched on this at the start, but it's worth repeating. Retrieval is a process that requires dynamic decisions; it is not a single-step "fetch" call.

Pitfall 2 — An ungoverned flat vector store. Dumping everything into one vector store without access control, source tagging, and freshness tracking. This is an invitation to disaster for both security and compliance.

Pitfall 3 — Ignoring source quality. "Garbage in, garbage out" works mercilessly here. A system fed from a bad, contradictory, stale source gives confidently wrong answers; which is more dangerous than giving no answer at all.

Pitfall 4 — Going to production without evaluation. If you don't have an evaluation set that measures retrieval quality, you can't know whether the system is improving. I set up a "golden question-answer" set first in every project.

Pitfall 5 — Lack of traceability. If the system, when it gives an answer, can't answer "which source did you get this from?", you can't defend that system in an enterprise setting.

Now I come to the topic that technical excitement overshadows most, but that I dwell on most in consulting: data governance and KVKK. As elegant as the knowledge-layer idea sounds, what you put in that layer puts you directly under legal liability.

The questions you must ask yourself while building a knowledge layer:

• Does personal data enter this layer? When special-category data such as customer records, employee information, or health data enters a vector store or knowledge graph, KVKK's processing, storage, and disclosure obligations are triggered. Being turned into an embedding does not make data "anonymous"; in most cases it is reversible or makes the individual identifiable.
• Is there access control? Who, with which role, can access the knowledge layer? If the content of a document an employee shouldn't access leaks indirectly through RAG, you have an access breach. You must design access control at the chunk and source level.
• Is auditability ensured? Which data entered the layer when, who accessed it, in which answer was it used? When a data-subject request comes in under KVKK, you must be able to show where and how that person's data was processed. An untraceable vector store can't meet this request.
• Can the right to erasure and rectification be exercised? When a person requests their data be deleted, you must be able to remove it from the source, the derived embeddings, and the knowledge graph. A compilation-staged knowledge layer is actually an advantage here: when you fix the source and recompile, the derived layer is cleaned too.

There's a hidden advantage of the compilation-staged knowledge layer here. Because you apply governance at a central point, you can enforce KVKK obligations in a single compilation pipeline instead of chasing them across scattered vectors. Source quality control, personal-data masking, access tagging, all done at compile time. This is a far more defensible architecture, both technically and legally.

With the EU AI Act, high-risk use areas (like hiring, credit, health, and public services) require transparency, traceability, and human oversight from AI systems. For exporters in Turkey and companies touching the European market, this isn't a distant topic; any organization with customers in Europe will eventually face this framework.

This is exactly where a well-designed knowledge layer is not a burden but a compliance tool. Because:

• If you can show which source an answer came from, you meet the explainability obligation.
• If what the knowledge layer contains is auditable, you meet the data governance obligation.
• If a human operator can see and intervene in the sources the system relies on, you meet the human oversight obligation.

So the investment you make in the knowledge layer is also an investment in regulatory compliance. I always summarize it to my clients like this: here, good engineering and good compliance look in the same direction.

Let's leave theory and get to the field. Here are the steps I follow when building a knowledge layer at an organization. Take this as a framework, not a recipe; every organization's context differs.

1. Inventory and classify sources. What knowledge is where? Which source is current, which is reliable, which contains personal data? Nothing starts without this inventory.
2. Gate source quality at the door. Weed out contradictory, stale, low-quality sources before they enter the layer. Quality control happens at the entrance, not after.
3. Model entities and relationships. If you expect relational questions, build a knowledge-graph layer. Think not just "which topic" but also "which relationship" questions.
4. Build the compilation pipeline. Centralize interpretation, normalization, contradiction cleanup, personal-data masking, and tagging in this pipeline.
5. Embed access control into the layer. Define roles at the chunk and source level; retrieval must respect these roles.
6. Make retrieval agentic, but measured. Give the agent autonomy to pick tools, but don't route every question into the full agentic flow; leave a fast path for simple questions.
7. Measure continuously with an evaluation set. Regularly measure retrieval and answer quality with a golden question-answer set; catch regressions early.
8. Design traceability from the start. Every answer must be able to show the source it relies on. Adding this later is very expensive; put it in the architecture from the start.

The most critical of these eight steps, if you ask me, are the second and eighth: gating source quality at the door and putting traceability in from the start. Because what crashes a RAG system in production is almost never the model; it's dirty sources and untraceable answers.

If I had to gather this whole evolution into a single sentence: RAG is ceasing to be a "search feature" and turning into an "enterprise knowledge infrastructure." Flat vector RAG was a beginning; agentic RAG added decision intelligence; graph-augmented architectures brought back relationships; context engineering turned context into a discipline; and the compilation-staged knowledge layer moved reasoning out of query time into a pre-built, managed, auditable asset.

I have one final piece of advice, distilled from the field. On your next RAG project, don't let your first question be "which model?" Let your first question be: "Is our knowledge clean, structured, governed?" The day you can honestly answer "yes" to that question, it won't matter which model you use. Because the real value was never in the model; it was in the quality of your knowledge layer and how well you govern it. The organizations winning in 2026 are the ones understanding this early; and in the field, I see this difference more clearly every day.

There's a topic most people underestimate within the knowledge-layer discussion: chunking. Let me confess, the thing that wasted the most time in my career, that made me ask "why doesn't this system work" the most, was almost always chunking. It sounds like a mechanical detail, but its consequences directly determine answer quality.

The problem is this: how you chunk a document determines what retrieval can see. Chunk too small and context breaks; a sentence becomes meaningless on its own. Chunk too large and irrelevant information tags along, wasting the context window, even misleading the model. The classic mistake I see in the field is blindly cutting documents by a fixed character count; you split a table down the middle, a procedure halfway through a step, and the system can never recover.

My approach is structure-aware chunking. I chunk a contract by its clauses, a technical manual by its sections, a table while preserving row integrity. Preserving the unit of meaning always comes before preserving the character count. The beauty of the compilation-staged knowledge layer is exactly this: you design the chunking strategy centrally, once, correctly, and apply it across the whole system. No re-chunking headache on every query.

Another under-discussed but critical topic is reranking and pruning. Retrieval may bring you ten chunks, but handing all of them to the model is usually wrong. The context window is a limited resource, and wasting it both raises cost and causes the "lost in the middle" phenomenon, where the model overlooks information in the middle.

Here's what I learned in the field: a little but correct context always beats a lot of noisy context. I take the returned chunks through a reranker model, sort by relevance, then pick the top few. The rest do nothing but distract the model. I always tell clients: think of the context window like a shop window; you don't cram everything into it, you put only the best products in the most visible spot.

Setting aside the technical dimension, let me state the human truth I observe most in consulting. The quality of a knowledge layer ultimately depends on the discipline of the people who produce and feed that knowledge. Even the most elegant compilation pipeline won't save you if the source documents are scattered, unupdated, and ownerless.

That's why in successful projects, a cultural transformation happens alongside the technical one. Who is responsible for which knowledge? When a procedure changes, who updates its source? Who weeds out a stale record? These are not technical but organizational questions, and no knowledge layer is long-lived until their answers are clear. The best organizations I've seen in the field are the ones that have clearly defined data ownership. For them, the knowledge layer is not a software project but a living enterprise asset.

One final warning, because excitement sometimes clouds judgment. Agentic RAG and rich knowledge layers are powerful but expensive. A system that runs five tool calls, three retrieval rounds, and a reflection loop for every question is both slow and bill-inflating. At enterprise scale, when tens of thousands of queries arrive per day, this cost reaches serious figures.

The principle I apply is a tiered architecture. A fast, cheap path for simple, frequent questions; the full agentic flow for complex, rare ones. I put a router in front of the system, assess the question's complexity, and send it to the right pipeline. This keeps cost under control and preserves the user experience. Solving everything the most expensive way isn't engineering, it's waste.

I'd also like to add a few Turkey-specific realities, because global content often skips these nuances. First, language: Turkish is still treated as a second-class citizen by many RAG components. The success of embedding models at capturing Turkish morphology varies compared to English; affixed structures and words derived from the same root can strain retrieval. When working with Turkish sources in the field, choosing the embedding model carefully and, where possible, measuring its Turkish performance with your own evaluation set is essential.

Second, in enterprise Turkey much of the knowledge is still scattered: scanned PDFs, Excel tables, email attachments, even paper archives. Before building a knowledge layer, a digitization and cleanup phase is often needed. Accepting this upfront grounds the project in reality. Third, alongside KVKK, sector-specific regulations (the BDDK in banking, relevant legislation in health) determine what may and may not enter the knowledge layer. That's why bringing the legal and compliance team in at the start of the project is far cheaper than firefighting later.

Putting all this together, the conviction I've drawn from the field is this: the knowledge layer is both the biggest opportunity and the most patience-demanding work for organizations in Turkey. An opportunity, because most of your competitors are still fiddling with a flat vector RAG demo; patience-demanding, because clean, structured, governed knowledge isn't built overnight. But once built, that layer will be your most durable AI asset; and I can already see that organizations starting early are widening the gap over the coming years.