AI Data Center Networking: Scale-Up vs Scale-Out, InfiniBand vs Ethernet, and the Road to 1.6T

In AI data centers, the network fabric — not the GPU alone — now determines training speed and cost, because distributed models spend much of their time exchanging data between accelerators, and any fabric that stalls leaves expensive silicon idle.

The GPU is no longer the only bottleneck. As frontier models are split across tens of thousands of accelerators, the interconnect that stitches them together decides how fast a training run finishes and how much it costs. This guide explains scale-up versus scale-out fabrics, the InfiniBand versus Ethernet debate, and the 800G-to-1.6T transition reshaping enterprise AI builds. It is a companion to our pillar on AI data center development.

What is the difference between scale-up and scale-out fabrics?

Scale-up connects GPUs inside a single rack into one tightly coupled memory domain over ultra-fast links, while scale-out connects many racks across a cluster over a switched fabric. Scale-up maximizes bandwidth between nearby GPUs; scale-out maximizes reach across thousands of them. Modern AI clusters use both, layered together.

Inside the rack, the scale-up fabric behaves almost like one giant GPU. According to NVIDIA, a single Blackwell GPU supports up to 18 NVLink connections at 100 GB/s each — roughly 1.8 TB/s of total bandwidth, about 14 times PCIe Gen5. In the GB200 NVL72, this fifth-generation NVLink ties 72 GPUs into a single coherent domain delivering around 130 TB/s of GPU-to-GPU bandwidth. That density is why liquid cooling for GPU clusters has become mandatory at the rack level.

Scale-out is a different problem. Once you leave the rack, you cross into a switched network — InfiniBand or Ethernet — that must carry traffic across thousands of nodes. According to reporting on GB200 deployments, each GPU connects out through an 800 Gb/s SuperNIC, so a 72-GPU rack pushes roughly 57.6 Tb/s into the scale-out fabric. Scale-up delivers something on the order of 18 times the bandwidth of scale-out per link, which is precisely why keeping the most communication-heavy operations inside the rack matters so much.

Is InfiniBand or Ethernet better for AI clusters?

Both are viable in 2026. InfiniBand still leads on raw latency and lossless delivery for the largest training jobs, while Ethernet — reinforced by the Ultra Ethernet Consortium — has closed most of the gap and wins on cost, vendor choice, and multi-tenant flexibility. The correct answer depends on scale, budget, and operational skill, not ideology.

InfiniBand's advantage is deterministic behavior. It offers sub-2-microsecond latency and effectively zero packet loss, which historically made it the default for tightly synchronized training. Ethernet with RoCEv2 typically lands in the 5–10 microsecond range but can be tuned aggressively for AI workloads. The important shift is that Ethernet is no longer standing still.

According to the Linux Foundation, the Ultra Ethernet Consortium released its 1.0 specification in 2025 — a 500-plus-page rebuild of the network stack for AI and HPC — with founding members including AMD, Broadcom, Cisco, Arista, Meta, and Microsoft. For 2026, the UEC is standardizing programmable congestion management, congestion signaling, and in-network collectives that offload gradient reductions into the switch. These directly attack the tail-latency problems that used to favor InfiniBand.

The market reflects the momentum. According to Dell'Oro Group, InfiniBand held over 80 percent of AI back-end networks when coverage began in late 2023, but Ethernet is now forecast to overtake it in AI back-end networks by 2027. For teams deciding how to allocate scarce GPU capacity, the tradeoffs mirror those we cover in training vs inference data centers.

How do InfiniBand and Ethernet compare on the specs that matter?

InfiniBand and Ethernet now converge on 800G links today with 1.6T arriving, but they differ on latency, ecosystem openness, and operational model. InfiniBand is a single-vendor performance play; Ethernet is a multi-vendor, standards-driven ecosystem. The table below summarizes the practical differences enterprise buyers weigh.

The strategic read: InfiniBand remains the safe choice for a dedicated, single-tenant training superpod where every microsecond of synchronization counts. Ethernet is increasingly the default for enterprises that want vendor leverage, run mixed workloads, or expose GPUs to multiple internal teams. According to IEEE ComSoc analysis published in early 2026, white-box and ODM Ethernet vendors have become the top choice among AI hyperscalers — a signal that openness is winning at the largest scale.

What does the 800G to 1.6T transition change?

The jump from 800G to 1.6T per port doubles fabric bandwidth to match ever-larger GPUs, cuts the number of switch hops needed, and forces a shift toward co-packaged optics to keep power in check. It is the single clearest reason the network has become a first-class design constraint rather than an afterthought.

The bellwether product is here. According to Broadcom, its Tomahawk 6 "Davisson" switch — shipping since late 2025 — delivers 102.4 Tbps of switching with 64 ports of 1.6TbE and integrated co-packaged optics, claiming roughly 70 percent lower optical power. On the InfiniBand side, NVIDIA is expected to bring its own 102.4 Tbps and co-packaged optics generations to market through the second half of 2026, with NVLink-integrated photonics slated for its Feynman generation later this decade.

Why 1.6T matters in practice:

• Fewer hops, less latency. Higher per-port capacity flattens the topology, so traffic crosses fewer switches between distant GPUs.
• Power becomes the ceiling. Traditional pluggable optics do not scale to 1.6T cleanly; co-packaged optics move the light source next to the switch silicon to cut watts per bit.
• Cabling and thermals compound. Denser links raise heat and cabling complexity, tightening the link between network design and facility power and cooling planning.

The takeaway for enterprise buyers is that networking, power, and cooling decisions can no longer be made in isolation. A 1.6T fabric only pays off if the racks feeding it are provisioned and cooled to keep GPUs saturated.

Why is the network now the real differentiator?

Because distributed AI is communication-bound: training a large model means constantly exchanging gradients and activations between GPUs, and inference at scale means routing tokens across accelerators. When the fabric stalls, idle GPUs burn capital. A superior network converts the same silicon into more usable throughput.

This reframes procurement. Two clusters with identical GPUs can differ sharply in effective performance based on fabric design, congestion control, and topology. That is why hyperscalers invest so heavily in back-end networks — according to Dell'Oro Group, AI back-end network switch sales are projected to exceed $100 billion over five years. The network is where a meaningful share of AI capital now flows, and where poor design quietly erodes ROI. For organizations running these environments day to day, disciplined cloud infrastructure management is what keeps utilization high after the build is done.

How Gain America helps you build the right AI fabric

The hardest part of AI networking is not choosing a switch — it is having engineers who can design the topology, tune congestion control, validate GPU utilization, and align the fabric with power and cooling. That talent is scarce, and hiring it permanently is slow.

Gain America is a US-based IT consulting and staffing firm that places specialized AI infrastructure engineers — fabric architects, HPC networking specialists, and data center build teams — into enterprise programs. We help you decide between InfiniBand and Ethernet, plan the 800G-to-1.6T path, and staff the people who make the network deliver. Start with our AI data center development pillar, then contact Gain America to scope the talent your buildout needs.