What is the bottleneck when moving data between AI chips?

When Chips Are Faster Than the Connections Between Them

Modern AI chips are remarkably fast at raw computation, especially for the kind of parallel matrix math that dominates AI training and inference. But a large AI system isn’t just one chip; it’s often thousands of chips working together, and that collaboration requires moving data between them constantly. The bottleneck emerges because the speed of that data movement, whether between chips inside the same server or across many servers in a data center, has historically struggled to keep pace with how fast the chips themselves can compute.

This creates a situation where an individual chip might finish its portion of a calculation well before the data it needs from, or needs to send to, another chip has actually arrived. The chip’s computational power effectively goes to waste during that wait, since it can’t move forward until the necessary data exchange completes.

Where This Bottleneck Shows Up

This isn’t a single, uniform problem; it appears at multiple levels of an AI system’s architecture. Within a single server, multiple GPUs need to exchange data with each other extremely quickly, which is why specialized high-bandwidth connections exist specifically to link GPUs together more directly than general-purpose computer buses would allow. Across an entire data center, servers themselves need to communicate with each other over networking infrastructure, and that longer-distance data movement introduces its own latency and bandwidth constraints.

At both levels, the underlying challenge is similar: the physical and engineering work required to move data quickly, reliably, and at scale is different from the work required to make an individual chip compute faster, and progress on one doesn’t automatically translate into progress on the other.

Why This Gap Persists

Improving a chip’s raw computational throughput is, broadly speaking, a matter of packing more processing capability into the chip’s design. Improving data transfer speed between chips involves a different set of engineering challenges: physical signaling limits, cabling and connector technology, and the complexity of coordinating data movement across many components without introducing errors or excessive delay. These are genuinely different engineering domains, and progress in chip compute power has generally outpaced progress in the infrastructure needed to move data between chips at a matching rate.

This is precisely why so much investment goes into specialized interconnect technologies for AI infrastructure. They exist specifically to narrow this gap, even though fully closing it remains an ongoing challenge rather than a solved problem.

Bottom Line

The core bottleneck in moving data between AI chips is that data-transfer speeds haven’t kept pace with the raw computational speed of the chips themselves, leading to situations where expensive compute capacity sits idle waiting for data. This mismatch shows up both within individual servers and across data centers, and addressing it is a major, ongoing focus of AI hardware and infrastructure engineering.