Momentum

Momentum adds a kind of motion memory to the optimization process. In standard gradient descent, each step depends only on the current gradient, while momentum also takes past update directions into account. This often leads to more stable movement across the loss surface and faster convergence, especially in narrow valleys. Widely used in deep learning training, momentum can improve both efficiency and stability. Although it may seem like a small addition, its effect is often substantial.