NVIDIA Says Full-Stack Codesign Gets More Work Out of Every Byte of AI Memory
Announced NVIDIA argues that agentic AI's growing memory demands are best met by designing GPUs, CPUs, networking, storage and software together rather than simply adding HBM capacity.. NVIDIA says its full-stack codesign approach relieves GPU memory pressure from model weights, KV cache and RAG data across the Blackwell Ultra to Vera Rubin generations, reporting up to 30x higher…

NVIDIA is making the case that the next leap in AI inference economics will come from how memory is used, not just how much of it a GPU carries. In a post on its official blog, the company says larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise, a trend it expects to continue through the near term. The post argues that simply packing more high-bandwidth memory onto a processor is not the answer, because capacity only delivers what bandwidth can move.
The company describes an AI factory as running on a memory and storage hierarchy. GPU high-bandwidth memory is the fastest, most expensive and most contested tier, with model weights and KV cache competing for limited capacity; below it sit CPU memory, local SSDs and shared storage, each trading speed for more capacity at a lower cost per bit. NVIDIA says how much useful work a byte does depends on how well that whole hierarchy is used, and that its approach is to design GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer.
NVIDIA says it was first to adopt HBM3e and HBM4 technologies into an accelerated computing stack, but frames full-stack codesign as the larger effort. The post cites SemiAnalysis AgentX figures showing that from Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose roughly 3% while agentic throughput per dollar rose by up to 30x. It also says GB300 NVL72 delivers up to 10x higher throughput per dollar than the H200 platform at 100 tokens per second per user, with HBM capacity per GPU roughly doubling, and that Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72 at 125 tokens per second per user with HBM capacity per GPU up about 3%.
The post details several techniques. NVFP4 reduces the space required for model weights compared with FP8, and NVIDIA says recent models including FLUX.2 klein from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark from Meta, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL have shipped with NVFP4 checkpoints. Within an NVL72 system, model weights can be distributed across up to 72 GPUs so each GPU holds a smaller share of the model, leaving more HBM for KV cache and concurrent users.
For KV cache, NVIDIA says applying NVFP4 reduces its footprint compared with FP8 so more context can stay resident in HBM, while the Vera CPU helps in two ways: faster tool execution shortens the time KV cache sits inactive during tool calls, and its large LPDDR memory, connected to NVIDIA Rubin GPUs over NVLink-C2C, gives paused context a nearby place to wait. When context is not actively in use, NVIDIA Dynamo inference software moves it to NVIDIA CMX context memory storage, a tier the company says is built for capacity at far lower cost per bit than HBM and shared across servers. Dynamo's KV Cache Reuse then manages KV cache as one system-wide resource across GPU memory, CPU memory and storage so context computed once can be reused across the system.
On retrieval-augmented generation, NVIDIA says its Nemotron Retriever models produce smaller embeddings with higher retrieval accuracy while the cuVS library lets the index be built and served across CPU and storage, so the knowledge base never occupies HBM and only the passages the model needs enter context. The post names CMX context memory storage partners including Cloudian, DDN, Dell Technologies, Everpure, HPE, IBM, NetApp, VAST and WEKA, and points to a video explainer and technical deep dive on how full-stack codesign reduces inference memory pressure.
Source: NVIDIA.
InteliToys is editorial. We do not sell NVIDIA full-stack codesign (Vera Rubin NVL72, Vera CPU, Dynamo, CMX). Watch the source tape in the branded lightbox below.


