NVIDIA's new Dynamo Snapshot tool slashes the time it takes to start running AI inference models on Kubernetes clusters. The company says the solution brings cold-start times down to less than five seconds, a significant improvement over the minutes that such operations can typically require. It does this by combining two existing technologies: the open-source Checkpoint/Restore in Userspace (CRIU) and NVIDIA's own GPU Memory Service.
How the snapshot works
Cold-start latency has long been a headache for teams deploying AI models at scale. When a new pod spins up on Kubernetes, the model must load into memory, initialize its data structures, and often warm up its GPU caches. Dynamo Snapshot sidesteps that process entirely. Using CRIU, it takes a checkpoint of a running model's full state — including CPU registers, memory pages, and open file descriptors — and saves it to disk. When a new inference request arrives, the tool restores that checkpoint on a new container, bringing the model back to life almost instantly.
The GPU Memory Service handles the graphics-card side of the equation. It pre-allocates and manages GPU memory regions so that the restored model can jump straight into processing without waiting for memory allocation or CUDA context initialization. Together, the two components let Kubernetes reschedule inference workloads rapidly, even on fresh nodes that have never run the model before.
Why the speed matters
In production, AI inference pipelines often handle unpredictable traffic spikes. A sudden surge of requests can require rapid scaling from zero replicas to dozens. If each new replica takes several minutes to become ready, users experience timeouts or degraded service. By cutting that window to under five seconds, Dynamo Snapshot makes it feasible to scale inference pods up and down aggressively without sacrificing responsiveness.
The approach also benefits edge deployments, where compute resources are scarce and network bandwidth limited. Rather than downloading and initializing a large model on the fly, an edge device can receive a pre-computed checkpoint and start inferring nearly immediately.
Built on open source
CRIU, the underlying checkpoint-restore engine, has been under development for over a decade and is used in production by projects such as Docker and LXC. NVIDIA's Dynamo Snapshot extends the tool with GPU-awareness, allowing it to capture and restore the state of NVIDIA GPUs — something the vanilla CRIU does not support. The GPU Memory Service fills that gap by managing the GPU memory pools that the restored process needs.
The company has not disclosed whether Dynamo Snapshot will be released as an open-source project itself or remain a proprietary addition. For now, it's a demonstration of what's possible when checkpoint-restore technology meets NVIDIA's GPU infrastructure stack.
