8 min read Agent Architecture

Seven percent of your agent's calls carry 68% of its bill

Seven percent of your agent's calls carry 68% of its bill

NVIDIA released Nemotron 3.5 Lightning on 11 August 2026, an open 30 billion parameter model with 3 billion active, presented plainly as built for the execution layer of long-running agents. The technical post states the thesis in one sentence: an always-on agent spends most of its time on tool calls, result validation and subagent delegation, and running a frontier reasoning model on every one of those steps adds cost and latency. The same day, LangChain published a measurement from its own agent evaluation suite: across 145 multi-turn tasks, only 7% of turns went to the frontier model, and those 7% carried 68.4% of the spend.

The reflex on reading those two numbers is to conclude that you should route. LangChain's data deserves a slower reading. The routed arm scored 80.0% accuracy against 86.0% for the frontier model alone, and 77.7% for the small model used alone on everything. Routing gains 2.3 points over the simplest option, while run-to-run variance is 2.7 points. On that workload, LangChain states outright that it cannot claim routing beat the small model. What routing buys is not accuracy, and not average savings either: it is a ceiling on cost when you cannot tell in advance which request is hard.

The 3 July article on inference costs approached the subject through technical levers, and the 11 July article on token budgeting took the session as its unit. The useful unit here is neither the token nor the lever, it is the trajectory step. What follows covers what three independent measurements actually say, the formula that decides before the architecture does, and the audit to run against your own traces.

A chain of mostly small blue steps, of which a single orange one occupies most of the proportion bar below.
Not every step in an agent trajectory carries the same value, or the same cost.

What the execution layer actually covers

Nemotron 3.5 Lightning is a mixture-of-experts model with 30 billion parameters, 3 billion of which activate per token, distilled from the frontier model of the same family, on a hybrid architecture that interleaves Mamba-2 layers, expert layers and attention layers. Weights, training data and recipes are published under the OpenMDW-1.1 licence, with an NVFP4 checkpoint alongside BF16, which makes it deployable from a workstation to a data centre.

The architecture words

Mixture of experts (MoE): an architecture where an internal router sends each token to only a small subset of the model's sub-networks. You get the capacity of a large model at the compute cost of a small one.

Active parameters: the fraction of parameters actually used to produce a token. Here 3 billion out of 30, one tenth.

Harness: the software layer around the model that decides how it receives context, calls its tools and chains its turns. It shapes an agent's behaviour as much as the model does.

Speculative decoding: a small model drafts several tokens ahead and the full model verifies them in one pass. Identical output, faster generation.

Two points in this release deserve a technical leader's attention more than the speed figures do. The first is harness-optimized training: the model was trained for named agent harnesses, which extends directly what the 5 July article on harness engineering described as the main determinant of an agent's behaviour in production. The second is the simultaneous release of NeMo Switchyard, an open routing library that exposes this small model as a routing target alongside your closed models, with strategies that read agent state turn by turn rather than a task category fixed at design time.

The modesty of the positioning is to the vendor's credit. On the Artificial Analysis general capability index, which aggregates nine evaluations, Lightning scores 24, tied with gpt-oss-120b and behind several models in its size class sitting at 30. NVIDIA does not claim otherwise, and its claim is narrow: on PinchBench, 86% accuracy while completing 10,000 tasks 30% faster than a comparably sized competitor at similar accuracy. That is a speed-versus-accuracy trade, not a capability win, and it is worth remembering that the figure comes from the vendor itself.

Three measurements, three very different results

Third-party tests give a more useful picture than the vendor post, because they disagree.

LangChain ran its suite of 145 multi-turn agentic tasks, averaging 6.3 model calls each, through the router in escalation mode. The frontier model alone scored 86.0% at $11.45 per run. Routing scored 80.0% at $3.00. The small model alone scored 77.7% at $0.72. The small model handled 93% of calls for 10.4% of the spend. An important caveat comes with those numbers, and LangChain raises it itself: the suite is saturated, only 8 points separate the small model from the frontier model, which leaves routing less room to prove its value than a harder workload would.

The cost line nobody anticipates is the judge model. It runs on every turn until a task escalates, it gets no benefit from prompt caching, and it accounts for 21.2% of the routed arm's spend, the second largest item after the frontier model. Your router carries a tax, and that tax is paid even when no escalation occurs.

Dispersion is the second lesson. Across five identical runs, the share of traffic sent to the frontier model ranged from 4.1% to 9.1%, and cost from $2.16 to $3.61. Nothing changed except the router's decisions, and the bill moved by 67%. A router lowers your average spend and widens the interval around it, which means budgeting against the top of the range rather than the mean.

Two other results, reported by VentureBeat from figures NVIDIA shared, land in a noticeably tighter band. Ramp says it matched frontier model performance on its own software engineering benchmark while cutting costs 58% and runtime 33%. Cognition integrated stage routing into its desktop product and reports near-frontier performance on its production coding benchmark at 28% lower mean cost. The gap between 28% and 74% is not noise: it says the gain is not a property of the router, it is a property of your traffic mix.

Diagram of the minimum offload formula, showing the wide price gap case and the narrow gap case that makes routing impossible.
The gain from routing depends first on the ratio between judge cost and the price gap between your two models.

The formula to run before you touch the architecture

LangChain publishes, alongside its results, the one genuinely transferable thing in the whole file. Routing only reduces total cost relative to the frontier model alone when the share of turns you send to the small model clears this bar:

minimum offload = judge cost / (expensive model cost - cheap model cost)

On their setup, the judge cost $0.64 per run against a price gap of $10.73, a bar of 5.9%. They offloaded 93% of turns and cleared it sixteen times over. With a pairing that lopsided the formula is close to a formality. It becomes decisive when the gap narrows.

Its main value is negative, which is what makes it useful in an architecture review. If your two models are close in price, the saving per offloaded turn is small and the formula asks you to send more than 100% of your turns to the cheap model, which no setting allows. No judge configuration fixes that. The one exception is a small model hosted on your own infrastructure, whose marginal inference cost falls low enough to reopen the gap. This is the point where open weights stop being a sovereignty argument and become an economic one, and the two reinforce each other: the high-volume steps are also the ones whose data most easily stays in house.

Thoughtworks, an early access partner, put numbers on that path in its own trials. Its teams post-trained two domain adapters, legal and healthcare, in a few hours on a single eight-GPU node with no data leaving their environment, and the legal adapter was preferred over the base model in 75% of blind comparisons. On speed, they measured the default speculative decoder delivering 1.46 to 1.96 times the throughput of unaccelerated decoding depending on load, which roughly halves GPU cost per generated token on self-hosted hardware. The notable finding sits elsewhere: their own purpose-trained draft head only matched the one shipped by default, which is unusual and removes an optimization project from the list.

This formula lets you rule routing out on cost. It does not let you rule it in. For that you have to look at your trajectories, and that is where the work starts.

In the field, teams that actually open their traces find a distribution that does not match the architecture diagram they drew. The steps consuming the most tokens are rarely the ones the team describes as the agent's intelligence. They are context re-reads, result formatting and tool output validation. Most organisations have never counted them, because their observability was built around task success and never around step class.

What to put in motion this week

Pull a week of production agent traces and classify every step into three categories: planning, execution, judgement. Count calls and tokens by category, not by task. Until that distribution is known, any discussion about routing runs on assumptions.

Run the minimum offload formula with your own prices, including the cost of the judge model you have in mind. If the result exceeds what your traffic can realistically offload, the decision is made and you have saved a quarter of integration work.

Write a switching threshold per step class rather than one global rule. A tool output validation and a planning decision do not call for the same accuracy requirement, and a single threshold makes the whole system carry the constraint of the hardest case.

Build a trajectory evaluation set before you switch anything, and make it the gate on the switch. It is the same instrument described in the 5 July harness article, applied here as an acceptance condition rather than a diagnostic tool.

Check the reversibility of what you put in place. A routing layer that cannot fall back to a single-model configuration reproduces exactly the dependency described in the 25 July article on the multi-model roadmap, with one more piece of infrastructure to maintain.

Conclusion

The question in front of a technical leadership this week is not whether an open model with 3 billion active parameters can replace a frontier model, because the answer depends too heavily on the workload to generalise. It is what share of the steps in your production trajectories is executor work, and why those steps run today on your most expensive model without anyone ever having to justify the choice. The figures published this month do not settle your case, they make the question measurable, and that is already more than most teams had a month ago.

Sources: As of August 2026

Godwin AVODAGBE