LoRA (Low-Rank Adaptation)
What is it?
LoRA is a cheap way to fine-tune a large model: instead of updating all its weights, you train a small set of add-on weights and leave the original model frozen.
Explain like I'm 5
Why was it created?
Fully fine-tuning a large model is slow, memory-hungry, and produces a full-size copy each time. LoRA was created to adapt big models cheaply and store each adaptation as a tiny file.
Where is it used?
- Fine-tuning open LLMs on a domain or style
- Custom image-generation styles
- Per-customer model variants
- Rapid experimentation
Why should developers care?
LoRA makes customizing large models practical on modest hardware, so it's the default way many teams and hobbyists fine-tune open models.
How does it work?
LoRA freezes the original weights and injects small low-rank matrices into chosen layers. Only those small matrices are trained, so far fewer parameters are updated. At inference the add-on is combined with the frozen weights.
Real-world example
You adapt an open LLM to write in your company's voice by training a few-megabyte LoRA instead of a multi-gigabyte full fine-tune.
Common use cases
- Domain or style adaptation
- Multiple lightweight model variants
- Fine-tuning on a single GPU
- Swappable 'personalities' for one base model
Advantages
- Tiny, fast, and cheap to train
- Adapters are small files (easy to share)
- Base model stays untouched and reusable
- Runs on modest hardware
Disadvantages
- Slightly less capacity than full fine-tuning
- Choosing where/how much to adapt takes tuning
- Many adapters can be hard to manage
- Not a fix for missing base-model capability
When should you use it?
When you want to customize a large model cheaply, keep the base reusable, or maintain many small variants.
When should you avoid it?
When you need the deepest possible adaptation and have the compute for a full fine-tune, or when prompting alone suffices.
Alternatives
Related terms
Interview questions
Beginner
- What does LoRA leave unchanged during fine-tuning?
- Why are LoRA files so small?
Intermediate
- Why is a low-rank update enough to adapt a large model?
- How do LoRA and full fine-tuning differ in cost?
Senior
- How does QLoRA combine quantization with LoRA?
- What determines a good rank for a LoRA adapter?
Common misconceptions
- "LoRA changes the base model" — the base is frozen; the adapter is separate.
- "LoRA is lower quality than fine-tuning" — for many tasks it matches full fine-tuning at a fraction of the cost.
Fun facts
- A LoRA adapter can be just a few megabytes versus gigabytes for a full model.
- QLoRA made fine-tuning very large models possible on a single consumer GPU.
Timeline
- 2021 — LoRA introduced by Microsoft researchers
- 2023 — QLoRA extends it to quantized models
Learning resources
Quick summary
LoRA fine-tunes a large model by training small add-on matrices while freezing the original weights, making customization cheap and adapters tiny.
Cheat sheet
- Freeze base, train small add-ons
- Adapters are tiny files
- Cheap fine-tuning on modest hardware
- QLoRA adds quantization