https://route179.dev/tags/genai/ Recent content in GenAI on Route179 Hugo en-us 2026 Sheng Chen Tue, 30 Jun 2026 00:00:00 +1000 https://route179.dev/2026/06/30/optimizing-vllm-cold-start-with-model-streaming-and-compile-caching/ Tue, 30 Jun 2026 00:00:00 +1000 https://route179.dev/2026/06/30/optimizing-vllm-cold-start-with-model-streaming-and-compile-caching/ <p>I&rsquo;ve recently been working on an AI-at-the-edge project (<a href="https://github.com/sc13912/safetylens_v2/">SafetyLens</a>) — that&rsquo;s a topic for another post. During the demo development, I frequently needed to tweak the model serving engine, swap models in and out, and refine configurations. Every one of those changes meant restarting vLLM, and each restart triggered a full cold start that took five minutes or more before the server was ready to serve again.</p> <p>And that&rsquo;s with just a single <code>Qwen3.6-35B-A3B</code> model running off local NVMe storage on 1x NVIDIA DGX Spark node. This will only get worse once I scale the deployment to a multi-node cluster backed by an external PVC, where the weights have to travel over the network on every start. So I went looking for ways to cut down the vLLM cold-start time — and it turns out most of it is recoverable, with no changes to the model or its output quality.</p> https://route179.dev/publications/repost-eks-deepseek-vllm/ Mon, 01 Jan 0001 00:00:00 +0000 https://route179.dev/publications/repost-eks-deepseek-vllm/ https://route179.dev/publications/blog-eks-hybrid-genai-dgx/ Mon, 01 Jan 0001 00:00:00 +0000 https://route179.dev/publications/blog-eks-hybrid-genai-dgx/