Gradient Descent
What is it?
Gradient descent is the optimization method that trains most models: it repeatedly nudges the model's parameters in the direction that most reduces its error.
Explain like I'm 5
Why was it created?
Training a model means finding parameter values that minimize error, but you can't try them all. Gradient descent gives an efficient, iterative way to search for good values.
Where is it used?
- Training neural networks
- Fitting most ML models
- Fine-tuning
- Any differentiable optimization
Why should developers care?
It's the engine behind neural network training — understanding it demystifies what 'training' actually does under the hood.
How does it work?
You measure error with a loss function, compute its gradient (the slope telling which way error rises), and step the parameters in the opposite direction. The step size is the learning rate. Repeating this over many batches of data gradually lowers the loss.
Real-world example
Training an image classifier, each batch nudges millions of weights slightly downhill on the error surface until predictions improve.
Common use cases
- Neural network training
- Model fine-tuning
- Logistic/linear regression fitting
- Large-scale optimization
Advantages
- Scales to huge models and datasets
- Simple, general idea
- Works with any differentiable loss
- Well-supported by frameworks
Disadvantages
- Sensitive to learning rate
- Can get stuck or oscillate
- Needs many iterations
- Non-convex surfaces have tricky spots
When should you use it?
Whenever you train a differentiable model — it's the default optimization approach.
When should you avoid it?
For non-differentiable objectives or tiny closed-form problems, other methods may fit better.
Alternatives
Related terms
Interview questions
Beginner
- What does gradient descent minimize?
- What is the learning rate?
Intermediate
- What is the difference between batch, mini-batch, and stochastic gradient descent?
- What happens if the learning rate is too high or too low?
Senior
- How do optimizers like Adam improve plain gradient descent?
- Why can non-convex loss surfaces still train well in practice?
Common misconceptions
- "It finds the global best" — it usually finds a good local minimum, which is often enough.
- "Bigger steps train faster" — too large a learning rate can diverge.
Fun facts
- Stochastic gradient descent uses small random batches, adding noise that can actually help.
- The learning rate is one of the most important hyperparameters to tune.
Timeline
- 1847 — Gradient descent described by Cauchy
- 2010s — Adaptive variants (Adam) become deep-learning defaults
Learning resources
Quick summary
Gradient descent trains models by iteratively nudging parameters in the direction that most reduces error, guided by the loss gradient.
Cheat sheet
- Step parameters downhill on the loss
- Learning rate sets step size
- SGD uses random batches
- Engine of neural network training