en.Wedoany.com Reported - At the GTC conference in Taipei held in June 2026, NVIDIA founder and CEO Jensen Huang pointed out that the most challenging issue in AI infrastructure is the memory system, involving the management of KV cache for AI agent working memory, as well as the retrieval of structured and unstructured data and the establishment of data ontologies. Huang stated that AI's memory system will completely revolutionize storage systems.
To address the surging demand for KV cache storage in the AI inference era, NVIDIA has launched the CMX Context Memory Storage Platform, managed by the BlueField-4 DPU. This platform adds a Pod-level context layer between local SSDs and shared storage.
The rise of agentic AI is reshaping CPU architecture. Huang noted that agents live in a nanosecond-scale world, where every wait hinders their progression to the next step, making ultra-low latency a top priority. As both NVIDIA and Arm have introduced CPU rack solutions designed specifically for agents, the industry is shifting from throughput-oriented architectures to latency-oriented architectures, opening up an incremental market for CPU memory.
NVIDIA's public data shows that since the second half of 2024, the average number of output tokens per question has surged more than fivefold annually, reaching approximately 30,000 to 40,000 tokens. This indicates that the industry has entered the "thinking" phase of test-time scaling, one of NVIDIA's three scaling laws. The explosive growth in token output per question directly translates into greater demand for memory and computing resources.
In the AI inference era, the hardware requirements for AI chips and overall systems differ significantly from those for AI training. Inference imposes three key demands on hardware: higher queries per second, longer context windows, and more inference steps and agentic AI loops. These factors collectively drive a structural shift in memory demand, which can be analyzed from three dimensions: model weights, KV cache, and agentic AI.
