vLLM

In this post, I’ll walk you through deploying DeepSeek-V4-Flash-DSpark — a 284B-parameter MoE model (13B active) — across two NVIDIA DGX Sparks running as EKS Hybrid Nodes, connected over a ConnectX-7 RoCE stacking link. A single DGX Spark has a 128 GB unified memory pool, so the weights alone (~167 GB) don’t fit on one node. Spreading the model across two Sparks with tensor parallelism (TP=2) leaves just enough room for the weights, the KV cache, and framework overhead. I’ll show you how to reach 50–60 tok/s decode with a 1.79M KV pool — a 512K context window at >3× concurrency as deployed, or up to a single 1M context stream. ...

I’ve recently been working on an AI-at-the-edge project (SafetyLens) — that’s a topic for another post. During the demo development, I frequently needed to tweak the model serving engine, swap models in and out, and refine configurations. Every one of those changes meant restarting vLLM, and each restart triggered a full cold start that took five minutes or more before the server was ready to serve again. And that’s with just a single Qwen3.6-35B-A3B model running off local NVMe storage on 1x NVIDIA DGX Spark node. This will only get worse once I scale the deployment to a multi-node cluster backed by an external PVC, where the weights have to travel over the network on every start. So I went looking for ways to cut down the vLLM cold-start time — and it turns out most of it is recoverable, with no changes to the model or its output quality. ...