What is LoRA?

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that adapts a large pre-trained model to a new task without updating all of its weights. Instead, LoRA freezes the original model and trains a small set of low-rank matrices that are added to specific layers, drastically cutting the memory and compute needed to fine-tune. A LoRA adapter is typically a few megabytes rather than the full multi-gigabyte model, so many adapters can be stored and swapped on top of one base model. QLoRA combines LoRA with quantization to fine-tune large models on a single GPU. LoRA is the dominant approach for customizing open-weight models on modest hardware.