Gradient Descent definition
Gradient descent is an iterative method that seeks a minimum by taking steps opposite the gradient of a function.
Gradient Descent in a picture
How it works
With a positive step size α, replace the current input x with x − α∇f(x). The negative gradient gives a direction of local decrease when the gradient is nonzero. A step that is too large can overshoot, so the choice of step size affects convergence.
Worked examples
Take one step on f(x) = x² from x = 2 with step size 1/4.
The gradient is 4. The next point is 2 − 1 = 1.
What happens in the same function at x = 0?
The gradient is zero, so the update leaves x fixed.
Can a gradient-descent update increase the objective?
Yes. A step that is too large can overshoot and land at a higher value.
Common mistake
Recompute the gradient at the current point on each step. For nonconvex functions, a stationary point need not be a global minimum.
Quick check
What does α control?
Show answer
The scale of each update.
Does a zero gradient cause a move under this rule?
Show answer
No. The update is zero.
Sources
Flashcard
Gradient Descent
Reveal the definition
Gradient descent is an iterative method that seeks a minimum by taking steps opposite the gradient of a function.
Effortless Math. “Gradient Descent.” Math Dictionary.
