English interface, partly translated: navigation, lab, projects, series, about and contact are; the home page sections, the site search and the content itself (articles, tutorials, research) are still in French.

Gradient descent

Pick a function, a starting point and a learning rate, then watch the point walk downhill: convergence, oscillations, divergence, local minima.

Field: Artificial intelligence
Level: Beginner
Topics: Optimisation · Learning rate · Dérivée

Figure 1 — The point follows the slope downhill. Tangent in orange, next move −η·f′(x) in blue. Iteration 0. Drag the starting point along the curve, or use the x₀ slider.

Figure 2 — Loss f(xₖ) at each iteration.

Computation of the iteration k = 0

x₀ = 2.5 f(x₀) = 6.25 f′(x₀) = 5

x₁ = x₀ − η·f′(x₀) = 2.5 − 0.1 × (5) = 2

f(x) = x^2 \qquad f'(x) = 2x

All iterations

Values computed at each iteration
k	x_k	f(x_k)	f′(x_k)
0	2.5	6.25	5
1	2	4	4
2	1.6	2.56	3.2
3	1.28	1.6384	2.56
4	1.024	1.048576	2.048
5	0.8192	0.671089	1.6384
6	0.65536	0.429497	1.31072
7	0.524288	0.274878	1.048576
8	0.41943	0.175922	0.838861
9	0.335544	0.11259	0.671089
10	0.268435	0.072058	0.536871
11	0.214748	0.046117	0.429497
12	0.171799	0.029515	0.343597
13	0.137439	0.018889	0.274878
14	0.109951	0.012089	0.219902
15	0.087961	0.007737	0.175922
16	0.070369	0.004952	0.140737
17	0.056295	0.003169	0.11259
18	0.045036	0.002028	0.090072
19	0.036029	0.001298	0.072058
20	0.028823	0.000831	0.057646
21	0.023058	0.000532	0.046117
22	0.018447	0.00034	0.036893
23	0.014757	0.000218	0.029515
24	0.011806	0.000139	0.023612
25	0.009445	8.92·10^−5	0.018889
26	0.007556	5.71·10^−5	0.015112
27	0.006045	3.65·10^−5	0.012089
28	0.004836	2.34·10^−5	0.009671
29	0.003869	1.5·10^−5	0.007737
30	0.003095	9.58·10^−6	0.00619
31	0.002476	6.13·10^−6	0.004952
32	0.001981	3.92·10^−6	0.003961
33	0.001585	2.51·10^−6	0.003169
34	0.001268	1.61·10^−6	0.002535
35	0.001014	1.03·10^−6	0.002028
36	0.000811	6.58·10^−7	0.001623
37	0.000649	4.21·10^−7	0.001298
38	0.000519	2.7·10^−7	0.001038
39	0.000415	1.73·10^−7	0.000831
40	0.000332	1.1·10^−7	0.000665

Understanding gradient descent

Gradient descent looks for a minimum of a function $f$ by repeating one simple move: measure the slope, then take a step in the downhill direction.

x_{k+1} = x_k - \eta \, f'(x_k)

$f'(x_k)$ is the derivative at the current point: its sign tells which way the function rises, its magnitude how steep the slope is;
$\eta$ (eta) is the learning rate: it sets the length of the step;
the minus sign moves against the slope, hence downhill.

In a neural network, $x$ becomes the vector of all weights and $f$ the loss function, but the rule is the same, applied to every coordinate.

Why the learning rate decides everything

Take $f(x) = x^2$ , whose derivative is $2x$ . The update becomes:

x_{k+1} = x_k - \eta \cdot 2x_k = (1 - 2\eta)\,x_k

At each iteration, $x$ is multiplied by the same factor $r = 1 - 2\eta$ . The whole behaviour can be read from that factor:

Rate $\eta$	Factor $r$	Behaviour
$0 < \eta < 0.5$	$0 < r < 1$	steady convergence, from one side
$\eta = 0.5$	$r = 0$	minimum reached in a single step
$0.5 < \eta < 1$	$-1 < r < 0$	damped oscillations around the minimum
$\eta = 1$	$r = -1$	perpetual oscillation, no progress
$\eta > 1$	$r < -1$	divergence: each step moves away

More generally, for a function of curvature $L$ (here $L = 2$ ), descent is stable as long as $\eta < 2/L$ . That is why the same rate can suit one function and make another diverge.

What the demo lets you observe

Stable convergence: a moderate rate brings the point closer to the minimum at each step.
Slow convergence: a tiny rate moves by tiny steps; the iteration budget is not enough.
Oscillations: the point jumps from one side of the minimum to the other while getting closer.
Divergence: beyond the stability bound, the loss grows at every iteration.
Influence of the starting point: on a non-convex function (double well), two different starts lead to two different minima. Gradient descent only finds a local minimum.
Plateau: where the slope is almost zero, so are the steps, whatever $\eta$ .

Limits and next steps

"Plain" descent is sensitive to curvature: in a narrow valley, it zigzags. Variants with momentum and Adam fix this by remembering past slopes; they are compared in the optimizers demo.

Other demos

Optimizers: gradient, momentum, Adam
Artificial intelligence, Intermediate
Three optimizers start from the same point on a two-variable surface. Compare their trajectories over the level curves: zigzag, momentum, adaptive step.
K-Means step by step
Artificial intelligence, Beginner
Assign, update, assign… Follow the centroids until convergence, and discover where K-Means goes wrong: bad start, stretched clusters, rings.
Linear regression
Artificial intelligence, Beginner
Move the points and watch the least squares line follow. Then learn the same line by gradient descent and compare.

Understanding gradient descent

Why the learning rate decides everything

What the demo lets you observe

Limits and next steps

Other demos

Optimizers: gradient, momentum, Adam

K-Means step by step

Linear regression