What is saddle point in machine learning?
When we optimize neural networks or any high dimensional function, for most of the trajectory we optimize, the critical points(the points where the derivative is zero or close to zero) are saddle points. Saddle points, unlike local minima, are easily escapable.”
What are the challenges of using Newton’s method with neural networks?
However, the study of using Newton’s method in neural networks faces difficulties due to the complexity of the activation functions used, especially for the complex-valued net- works. The split functions are out of consideration, since they are not differentiable.
How to check if a point is a saddle point?
A simple criterion for checking if a given stationary point of a real-valued function F (x, y) of two real variables is a saddle point is to compute the function’s Hessian matrix at that point: if the Hessian is indefinite, then that point is a saddle point. For example, the Hessian matrix of the function
What is a saddle point between two hills?
Saddle point between two hills (the intersection of the figure-eight z {\\displaystyle z} -contour) In mathematics, a saddle point or minimax point is a point on the surface of the graph of a function where the slopes (derivatives) in orthogonal directions are all zero (a critical point), but which is not a local extremum of the function.
How did the saddle point get its name?
The name derives from the fact that the prototypical example in two dimensions is a surface that curves up in one direction, and curves down in a different direction, resembling a riding saddle or a mountain pass between two peaks forming a landform saddle.
Which is saddle surface has negative Gaussian curvature?
Saddle surfaces have negative Gaussian curvature which distinguish them from convex/elliptical surfaces which have positive Gaussian curvature. A classical third-order saddle surface is the monkey saddle. In a two-player zero sum game defined on a continuous space, the equilibrium point is a saddle point.