Gradient descent
Pick a function, a starting point and a learning rate, then watch the point walk downhill: convergence, oscillations, divergence, local minima.
English interface, partly translated: navigation, lab, projects, series, about and contact are; the home page sections, the site search and the content itself (articles, tutorials, research) are still in French.
Four algorithms explore the same grid. Draw walls, add costly terrain and compare the visited cells and the cost of the path.
Ready. Run the search or step through it.
| Algorithm | cells explored | path length | path cost |
|---|---|---|---|
| Breadth-first (BFS) | 498 | 39 | 39 |
| Depth-first (DFS) | 96 | 95 | 95 |
| Dijkstra | 498 | 39 | 39 |
| A* | 244 | 39 | 39 |
The four algorithms follow the same scheme: keep a frontier of cells to explore, take one out, add its neighbours. They only differ in the order in which the frontier is emptied.
| Algorithm | Frontier | Takes out first | Optimal path? |
|---|---|---|---|
| BFS (breadth-first) | queue | the cell discovered first | yes, in number of steps |
| DFS (depth-first) | stack | the cell discovered last | no |
| Dijkstra | priority queue | the lowest cost from the start | yes, in cost |
| A* | priority queue | the lowest | yes, if never overestimates |
A heuristic is admissible if it never overestimates the real remaining cost. Manhattan distance is: no move costs less than 1, and at least moves are needed. With , A* becomes Dijkstra again; with an overestimating heuristic, it runs faster but may miss the best path.
The table under the figure compares the four algorithms on the current map: same cost for Dijkstra and A*, very different numbers of explored cells.
Pick a function, a starting point and a learning rate, then watch the point walk downhill: convergence, oscillations, divergence, local minima.
Three optimizers start from the same point on a two-variable surface. Compare their trajectories over the level curves: zigzag, momentum, adaptive step.
Assign, update, assign… Follow the centroids until convergence, and discover where K-Means goes wrong: bad start, stretched clusters, rings.