Low-rank adaptation: a fine-tuning method that freezes a model's weights and learns a small low-rank update for chosen weight matrices, W + BA. It trains a fraction of a percent of the parameters, and the update can be merged into W afterwards, so inference costs nothing extra.
Card 1 of 292. Answer: Low-rank adaptation: a fine-tuning method that freezes a model's weights and learns a small low-rank update for chosen weight matrices, W + BA. It trains a fraction of a percent of the parameters, and the update can be merged into W afterwards, so inference costs nothing extra.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
For a d × k weight matrix W, LoRA learns B (d × r) and A (r × k) with a small rank r such as 8 or 16, scaled by α / r. B starts at zero, so training begins exactly at the base model. Hu and colleagues at Microsoft introduced it in 2021, and it is now the most common parameter-efficient fine-tuning (PEFT) method; on GPT-3 175B they reported 10,000 times fewer trainable parameters and a third of the GPU memory of full fine-tuning.
Example: one 4,096 × 4,096 attention projection holds 16.8 million weights. A rank-8 LoRA update for it trains 8 × (4,096 + 4,096) = 65,536 parameters, 0.39% of the matrix. Because only those are trained, optimizer state and gradients shrink by the same factor.
Follow-ups: QLoRA (Dettmers and colleagues, 2023) keeps the frozen base in 4-bit precision and trains LoRA adapters on top, enough to fine-tune a 65B model on one 48 GB GPU; one base model can serve many swappable adapters, one per task; the original paper adapted only the attention query and value matrices, while today all linear layers are common; and for large domain shifts full fine-tuning can still learn more, though LoRA also tends to forget less of the base model's abilities.
