5 min read Long Context

1M-token context windows: is it time to rethink your RAG architecture?

1M-token context windows: is it time to rethink your RAG architecture?

Two announcements in one week pushed long context from exception to standard. OpenAI shipped GPT-5.6 on July 9, 2026, with a 1.05 million token context window across the entire family (Luna, Terra, Sol), from the cheapest model to the flagship. A week later, on July 16, Moonshot AI released Kimi K3 through its apps and API, positioned as the largest open-weight model to date at 2.8 trillion parameters, also with a 1 million token context window and substantial attention efficiency gains through its Kimi Delta Attention architecture. The full open weights were not scheduled for publication until July 27, so at the time of writing the model is reachable through Moonshot's hosted API rather than a downloadable release.

1M-token context is no longer reserved for a premium-priced niche model. It has spread across the entire price range, open source included. The question architecture teams now face is no longer "when will long context be big enough," but "at what document volume does RAG remain justified."

The counterintuitive instinct to correct: 1M-token context landing on every model is often read as RAG's coming obsolescence. The cost figures below say the opposite. The pattern that recurs among teams that have already migrated part of their architecture isn't replacement, it's adding a fallback tier, not scrapping the existing pipeline.

What the economics still say

The answer starts with a cost gap that hasn't disappeared with the new context windows. A query processed with a 1M-token context typically costs between $0.80 and $3.50, versus $0.02 to $0.08 for the same query through a well-tuned RAG pipeline, a gap that widens further once prompt caching is layered onto the RAG side at high query volume. For a call center handling 10 million queries a month, the gap between the two approaches runs into tens of millions of dollars a year.

Multi-fact recall, the ability to retrieve several pieces of information scattered across a very long context, still plateaus around 60% for long-context models on realistic tasks, while a properly tuned RAG pipeline remains superior on that specific metric, with latency under two seconds versus several seconds to load and process a full context. An analysis attributed to Gartner and relayed by several industry outlets projects that roughly 75% of enterprise GenAI deployments will run on RAG architecture by 2028, up from about 35% in 2025, with cost economics cited as the primary driver of that growth rather than a technical limit of long context.

Illustration comparing a costly massive context to an efficient RAG retrieval pipeline
1M-token context is going mainstream with GPT-5.6 and Kimi K3, but the cost gap with RAG remains several orders of magnitude wide.

Where long context genuinely replaces RAG

Long context has a real edge, and it's narrower than often presented. On static, moderately sized corpora, below roughly 500,000 tokens, that rarely change and don't require granular per-document access control, loading the full context eliminates the retrieval errors specific to RAG: the wrong chunk selected, a poorly calibrated embedding, reranking that drops the right passage. A single contract to analyze, a frozen codebase to audit, an annual report to summarize are cases where long context simplifies the architecture without prohibitive cost, since the query stays one-off rather than repeated at scale.

Long context also benefits from prompt caching, a feature GPT-5.6 pushes further with configurable cache breakpoints. When the same large context serves as the basis for several successive queries, the cost per subsequent query drops sharply once the context is cached, which changes the calculation for extended conversational use on a single corpus.

Where RAG remains the right architecture

RAG keeps the edge whenever three factors come into play. The first is data freshness: a long context loaded once goes stale as soon as the source changes, while a well-designed RAG index updates continuously without reloading the entire corpus. The second is access control: RAG lets you filter retrieved documents by user permissions at query time, something a pre-loaded context shared across all users cannot do without duplicating contexts per access profile, an approach that blows up costs. The third is volume: beyond a few hundred thousand tokens of useful corpus and at a sustained query rate, the cost-per-query gap makes long context economically untenable, regardless of model quality.

The hybrid architecture taking hold

The pattern emerging in mature 2026 deployments no longer pits RAG against long context, it combines them in tiers. A fast, low-cost RAG first pass handles the majority of queries. A fallback to a full long context activates only for queries where RAG fails to retrieve a satisfactory answer, typically cross-cutting questions that require combining information scattered across several documents that classic retrieval struggles to assemble correctly.

Diagram of the two-tier hybrid RAG and long-context architecture
Two-tier hybrid architecture: RAG as the first pass for most queries, long context as a fallback for complex retrieval cases.

This architecture also answers a growing compliance requirement in regulated industries: documenting the sources used for each answer, measuring hallucination rate, and maintaining retrieval quality metrics. A RAG pipeline natively produces that traceability through chunk citation. A long context loaded as a block makes it harder to reconstruct after the fact, a point compliance teams often discover too late.

To put in motion this week

Segment your current RAG use cases by corpus volume and query frequency. Those covering under 500,000 tokens of static data at low volume are candidates for simplification into a directly loaded context, without a retrieval pipeline to maintain.

Calculate the real cost per query of your high-volume use cases at the new long-context pricing, before deciding whether they justify a migration. The cost gap observed this year is structural, not a transitory limitation of current models.

Identify the queries your current RAG fails to handle correctly, the ones requiring you to combine several scattered sources, and test a long-context fallback only on that subset.

Verify that your RAG pipeline produces source traceability usable in an audit, before considering dropping it on use cases subject to regulatory requirements.

Conclusion

GPT-5.6 and Kimi K3 make 1M-token context accessible across the entire price range, open source included, but they don't remove the cost and control gap that justifies RAG in most high-volume deployments. Long context gains ground on static, moderate corpora where it genuinely simplifies the architecture. RAG keeps the edge wherever data freshness, access control, and query volume dominate the calculation. The architecture that wins in 2026 is neither one in isolation, it's the combination of both, with RAG in the front line and long context as a targeted fallback.

Sources: As of July 2026

Godwin AVODAGBE