NVIDIA's inference optimisation library for LLMs and visual generation models, fully open source since 2025-03-22. Its 28 tech blogs form an auditable engineering ledger: Skip Softmax and sparse attention for long context, the three-part expert parallelism series with one-sided AlltoAll over NVLink, DWDP for NVL72, guided decoding cooperating with speculative decoding, inference-time compute, and evaluating agentic serving with trace replay and job-level metrics. Claims include Llama 4 Maverick above 1,000 TPS per user and 40,000+ tok/s on B200; Bing and NAVER Place run it in production.
SANA is NVIDIA Labs open-source codebase for high-resolution image and video generation (9k+ stars, Apache-2.0, PyTorch). Its efficiency comes from two structural changes: linear attention replaces vanilla attention in the transformer, and a deep compression autoencoder compresses images 32x instead of the usual 8x. Official figures: the 0.6B model renders a 1024 image in 0.9 s, roughly one fortieth of FLUX-dev 23 s, while FID improves from 10.15 to 5.61. The series now spans one-step generation, video, world models and streaming editing, and runs 4K in 8 GB of VRAM under 4-bit quantization. No benchmarks reproduced; recorded as unverified.
NVIDIA's open-source (Apache-2.0) orchestration layer, Rust for the performance path and Python for extensibility. It explicitly does not replace SGLang, TensorRT-LLM or vLLM - it wires them into a coordinated multi-node system: disaggregated prefill/decode, KV-aware routing (2x faster TTFT on Qwen3-Coder 480B), KVBM offloading KV cache to CPU/SSD/remote, ModelExpress streaming weights over NVLink for 7x faster cold starts, an SLA-driven Planner (80% fewer breaches at 5% lower TCO in Alibaba production), and Grove for topology-aware NVL72 scheduling.
As an Amazon Associate, we earn from qualifying purchases.