Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

Large diffusion transformers require 20-30 GB of VRAM in BF16 precision, making them inaccessible to most consumer GPUs. Most existing quantization backends like bitsandbytes, GGUF, torchao, and Quanto are weight-only solutions that reduce memory usage but don't speed up inference. SVDQuant, the method behind the Nunchaku inference engine, takes a different approach with W4A4 (4-bit weights and activations), reducing memory while also accelerating the denoising loop.

Diffusers now supports loading Nunchaku checkpoints directly via from_pretrained() with no local CUDA compilation required, thanks to the kernels package. A new "Nunchaku Lite" integration path allows Diffusers to load Nunchaku-style checkpoints without a custom pipeline or separate inference engine by patching relevant nn.Linear modules with runtime SVDQ/AWQ linear layers before checkpoint loading.

Two kernel families are available: svdq_w4a4 for 4-bit weights and activations with SVDQuant low-rank correction (used for attention and MLP projections in INT4 and NVFP4 variants), and awq_w4a16 for 4-bit weights with 16-bit activations (used for adaptive normalization layers like FLUX adanorm). The companion diffuse-compressor toolkit also lets users quantize new architectures and publish them as regular Diffusers repositories.