The GPUs you rent are just part of what determines your cluster's speed and efficiency. This comes into play through interconnect, the network that connects multiple nodes (machines) on the backend.
Historically, networking was viewed primarily as a cost center. Within AI factories, networking has become a business accelerator because higher GPU utilization, lower latency, improved synchronization, and predictable performance directly translate into greater productivity and improved return on AI investments.
Interconnects connecting GPUs typically come in two forms:
- RDMA over Converged Ethernet (RoCE), which includes RoCEv1 and RoCEv2
- NVIDIA Quantum InfiniBand
Here's a quick breakdown of how they work and why they matter:
RoCE operates at a networking layer called "layer 2." Meaning, it uses Ethernet and MAC addresses to communicate across the same network (broadcast domain). Nodes can communicate across multiple switches, but they must remain within the same "base" network.
RoCEv2 builds on this by using UDP/IP packets, allowing nodes to communicate through different subnets, buildings, or data centers. As a result, latency can increase since these nodes aren't guaranteed to be sitting in the same rack or data center.
NVIDIA Quantum InfiniBand uses a dedicated networking fabric rather than Ethernet/IP and can cross multiple switches and locations. However, it requires the foundations of the InfiniBand-enabled switches and fabric to be in place first to get the maximum performance possible with minimal latency. Guaranteed if you use a Lambda 1-Click-Cluster.
But what if you're not running on them? How much does interconnect affect performance?
At a high level, both InfiniBand and RoCE can achieve extremely low latency, often measured in just a few microseconds. InfiniBand generally delivers lower and more predictable latency, typically around 0.5–2 µs, while RoCE on a well-optimized local fabric can achieve roughly 1–3 µs.
However, RoCE is more sensitive to network configuration, congestion, and overall topology. In less optimal conditions (such as other traffic on the node, misconfigured switches, and non-optimized traffic), latency can be several times higher, potentially reaching tens of milliseconds under congestion. For now, we'll use these low-latency ranges as our baseline, bearing in mind that real-world performance will vary substantially, which we will see next.
The theory in practice
A 1-3 µs latency between communications doesn't seem like a large amount, but that's per transfer, and doesn't include anything about packet sizes, tensor allocation, and the like.
To see how these differences play out in practice, we ran an inference experiment:
- 2 nodes of 1-Click-Cluster NVIDIA HGX™ B200 connected through NVIDIA Quantum InfiniBand switches and adapters
- 2 nodes of HGX B200 off On-Demand, connected through RoCE.
To keep things fair, we spun up two HGX B200 On-Demand instances, ending up with two nodes in the same cloud region, but not per se in the same data center.
Here are the key details of how we ran our tests:
- Model:
nvidia/GLM-5.2-NVFP4, an NVIDIA Blackwell-optimized variant of Z.ai's GLM 5.2 model - Serving framework: SGLang
- Parallelism: TP16/DP16/EP16
- Benchmarking tool: NVIDIA's AIPerf
- Workload: 8,192 input tokens and 1,024 output tokens per request
- Concurrency: 10-300 simulated concurrent requests
Both servers used a core set of SGLang flags:
--tp 16 --dp 16 --ep 16 \
--enable-dp-attention \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--moe-dense-tp-size 1 \
--nnodes 2 \
--chunked-prefill-size 8192 \
--cuda-graph-max-bs-decode 16 \
--disable-prefill-cuda-graph \
--mem-fraction-static 0.70 \
--max-running-requests 512 \
--watchdog-timeout 315360000 \
--enable-dp-lm-head
RoCE also used:
NCCL_IB_HCA=mlx5_0:1
NVSHMEM_HCA_LIST=mlx5_0:1
NVSHMEM_IB_ENABLE_IBGDA=0
NCCL_IB_GID_INDEX=3
NVSHMEM_IB_GID_INDEX=3
NCCL_IB_ROCE_VERSION_NUM=2
NCCL_IB_ADDR_FAMILY=AF_INET
NCCL_SOCKET_FAMILY=AF_INET
While IB used:
NCCL_IB_HCA=mlx5_5:1,...,mlx5_12:1
NVSHMEM_HCA_LIST=mlx5_5:1,...mlx5_12:1
NVSHMEM_IB_ENABLE_IBGDA=1
The data
Our results show that as concurrency reaches about 250 users (hardware saturation), RoCE delivers roughly 70% lower throughput than NVIDIA Quantum InfiniBand, with a 2.5x slower time to first token. As the number of users decreases, the effects of the different fabrics become more apparent, with low concurrency showing the slowest inter-token latency and tokens per second overall.
The lesson
The results show that the interconnect is not a background detail. At scale, it directly shapes GPU utilization, responsiveness, and the return on your infrastructure investment.
When serving at scale, communication can become the bottleneck. Every matrix multiplication needs to move across the fabric between multiple nodes, leading to large gaps in speed if your nodes don't have an excellent interconnect.
In a pinch? Spin up two Lambda On-Demand or Reserved Instance nodes, and temper your expectations accordingly.
Looking for a more viable solution? Lambda 1-Click-Clusters connected through a dedicated NVIDIA Quantum InfiniBand fabric is the obvious solution for your serving needs.