Calculus

Gradient Descent

Students follow a bead rolling down a bowl in a series of smaller steps.

Gradient Descent definition

Gradient descent is an iterative method that seeks a minimum by taking steps opposite the gradient of a function.

In Spanish
descenso del gradiente
Topic
Optimization and constraints

Gradient Descent in a picture

Gradient Descent diagramFor f(x) = x², starting at x = 4 with step size 1/4 gives a next point of 2.Current input: x = 4The derivative of x² is 2xGradient: 2 × 4 = 8Multiply by step size 1/4Next input: 4 − 2 = 2The value falls from 16 to 4
For f(x) = x², starting at x = 4 with step size 1/4 gives a next point of 2.

How it works

With a positive step size α, replace the current input x with x − α∇f(x). The negative gradient gives a direction of local decrease when the gradient is nonzero. A step that is too large can overshoot, so the choice of step size affects convergence.

Worked examples

Take one step on f(x) = x² from x = 2 with step size 1/4.

The gradient is 4. The next point is 2 − 1 = 1.

What happens in the same function at x = 0?

The gradient is zero, so the update leaves x fixed.

Can a gradient-descent update increase the objective?

Yes. A step that is too large can overshoot and land at a higher value.

Common mistake

Recompute the gradient at the current point on each step. For nonconvex functions, a stationary point need not be a global minimum.

Quick check

Which direction is used to seek a minimum?

What does α control?

Show answer

The scale of each update.

Does a zero gradient cause a move under this rule?

Show answer

No. The update is zero.

Sources

Flashcard

Gradient Descent

Reveal the definition

Gradient descent is an iterative method that seeks a minimum by taking steps opposite the gradient of a function.

Effortless Math. “Gradient Descent.” Math Dictionary.