Newton's Method

Newton’s method aims to accelerate optimization by using not only gradient information, but also second-derivative information. As a result, it can converge much faster than standard gradient descent on certain problems. However, the cost of Hessian computation and matrix operations can become prohibitive in large-scale models. Even so, it is extremely powerful in theoretical optimization and in smaller, well-behaved problems. Newton’s method asks not only “Which direction should I move?” but also “How is the surface curving here?”