DGX Spark Clustering - RoCE/RDMA Networking Setup & NCCL Benchmarking
I’ve been running a single DGX Spark (GB10) as an EKS Hybrid Node for a while, and recently a second one landed on my desk. The plan is to stack the two Sparks with one QSFP112 cable to create a little two-node EKS Hybrid cluster for distributed inference. This post documents how to set up RoCE/RDMA networking and perform NCCL testing across the two Spark nodes, including all lessons learned. ...