Cold starts are becoming a weight-movement problem

For large-model serving, startup latency now has a direct cost: it slows autoscaling, rolling updates, and reinforcement-learning loops. NVIDIA AI said on July 24 that ModelExpress cut DeepSeek-V4 Pro startup from “8 minutes to under 2 minutes” by moving weights along a faster path to GPU memory.

“8 minutes to under 2 minutes” — NVIDIA AI

The linked NVIDIA Technical Blog gives the precise result: ModelExpress moved DeepSeek-V4 Pro weights and JIT kernel cache artifacts from a serving replica into a fresh replica in under 10 seconds, reducing total startup time from 8 minutes to 1 minute 44 seconds. The system looks for a compatible copy of the weights already resident in GPU memory, then transfers tensors directly GPU to GPU over P2P RDMA through NIXL.

Downloading once, then scaling from peers

The operational issue is duplication. A new worker may need to fetch weights from remote storage, local storage, or another worker. With checkpoints measured in hundreds of gigabytes or more, pulling the same model repeatedly becomes expensive and slow. NVIDIA notes that 10 replicas fetching an 806 GiB DeepSeek-V4 Pro model could create roughly 8 TiB of duplicate ingress if every replica downloads independently.

NVIDIA AI’s account typically points developers to deeper infrastructure posts, and this tweet is a concise entry point to ModelExpress inside NVIDIA Dynamo. The blog also describes cache reuse, GPUDirect Storage fallback, vLLM and SGLang integrations, and receiver-driven refit flows for RL post-training. The next test is portability: whether the same startup reduction holds across mixed clusters, different serving stacks, and training-to-inference pipelines where weights change constantly.

#modelexpress

NVIDIA ModelExpress Cuts DeepSeek-V4 Pro Startup From 8 Minutes