High-Performance GPU Memory Transfer on AWS Sagemaker Hyperpod

TL;DR

Perplexity needed to transfer non-contiguous GPU memory between machines efficiently.
Standard NCCL was not ideal due to its static 'world' requirement and synchronous model.
The solution uses RDMA with AWS Elastic Fabric Adapter (EFA) for direct, zero-copy GPU memory transfer.
Key RDMA network design principles include buffer ownership, memory registration, separate control/data planes, pre-posted receives, poll-based completion, and hardware topology awareness.
AWS p5 instances have a specific hardware topology with 4x 100 Gbps EFA cards and 1x NVIDIA H100 GPU per PCIe switch.
The custom solution uses libfabric with two-sided RDMA for control messages and one-sided RDMA WRITE for data transfer.
Optimizations for scaling to 32 network cards include operation queuing, network warmup, multi-threading, CPU core pinning, state sharding, operation batching, lazy operation posting, and NUMA-aware resource allocation.
The final achieved bandwidth was 3,108 Gbps, representing 97.1% of the theoretical maximum.
The open-source repository contains full technical details and code examples.

Continue reading the original article