NVIDIA: AI Chip Competition Shifts Toward Overall Data Center Efficiency, with Memory and Interconnect Emerging as Two Major Bottlenecks
nashnova research
Nvidia chief scientist Bill Dally identified memory bandwidth and chip interconnect as the two most pressing AI bottlenecks, signaling that competition is moving from single-chip performance to system-level data-center efficiency.
Why doesn't a faster chip fix everything?
AI models are too large for a single GPU. Tasks must be split across multiple GPUs running in parallel, with constant data exchange between devices.
This means → the bottleneck is not how fast one chip computes, but how quickly and efficiently data moves between chips and between chips and memory.
Executives from Nvidia, Broadcom, and Qualcomm agreed at the Applied Materials EPIC Center opening: memory bandwidth, chip interconnect, power, and thermal management are the core constraints on AI infrastructure today.
Where exactly does memory choke?
Data in high-bandwidth memory — HBM, the high-speed memory stacked next to the GPU — must pass through multiple transfer layers before reaching the GPU's compute cores.
Reading a single bit costs roughly 200–300 femtojoules at source, but after traversing the substrate, the physical-layer interface, and on-chip routing, energy consumption climbs to 4–5 picojoules — more than a tenfold increase.
In plain terms = the data itself is light; the "shipping cost" is enormous. Moving bits is both slow and power-hungry, especially during the decode phase of AI inference.
Dally's conclusion: the industry needs to boost memory bandwidth by roughly an order of magnitude while cutting per-bit energy consumption by a similar factor — requiring deep collaboration between memory vendors and chip designers.
Interconnect bottleneck: why does latency matter more than bandwidth?
Traditional communications engineering optimizes for bandwidth — how much data moves at once. AI systems, however, are far more sensitive to latency — how long each transfer takes.
This means → the time a user waits for the first token, and the interval between subsequent tokens, both depend partly on the delay at every chip-to-chip hop.
Put simply = bandwidth is the number of highway lanes; latency is the wait at each toll booth — more lanes don't help if the booths are slow.
This priority shift is reshaping system-architecture design: latency, not bandwidth, is now the primary optimization target for interconnect.
How is Nvidia trying to solve this?
System-on-wafer integration — placing multiple GPU dies on a single wafer — shortens physical distances, cutting both energy use and latency simultaneously.
Co-packaged optics (CPO) — replacing electrical signals with light for chip-to-chip data transfer — targets longer-distance links, reducing per-bit power and raising communication speeds.
Dally acknowledged that CPO still faces significant challenges: comb-laser reliability, modulator design, thermal stability, and materials-and-manufacturing process issues. Early cross-disciplinary collaboration is the key to compressing the development cycle.
The "token factory" — how does Dally see the full processing chain?
Dally described the data center as a "token factory": user request enters → prefill → attention computation → feed-forward computation → decode → output.
The prefill stage is driven mainly by compute power; the decode stage is constrained mainly by memory bandwidth and latency — two entirely different bottlenecks.
This means → making a single chip faster in isolation does not necessarily speed up the whole chain. Nvidia's goal is to maximize tokens generated per joule of energy and per dollar of cost.
What does this mean for the supply chain?
This reflects a shift in the competitive dimension: from "whose chip has the most raw compute" to "who can achieve coordinated optimization across the entire data center."
The strategic value of high-bandwidth memory, advanced packaging, and optical interconnect supply chains will continue to rise.
In plain terms = the next-stage winners in AI infrastructure will not be single-category champions, but integrators who can weld chips, memory, packaging, and interconnect into one coherent system.
市场有风险，内容仅供研究参考，不构成投资建议。