Contents
Can a loss function be non convex for gradient descent algorithm?
The function you have graphed is indeed not convex. However, it is quasiconvex. Gradient descent is a generic method for continuous optimization, so it can be, and is very commonly, applied to nonconvex functions.
What is a non convex optimization problem?
A non-convex optimization problem is any problem where the objective or any of the constraints are non-convex, as pictured below. Such a problem may have multiple feasible regions and multiple locally optimal points within each region.
Does gradient descent always converge for convex?
Gradient Descent need not always converge at global minimum. It all depends on following conditions; If the line segment between any two points on the graph of the function lies above or on the graph then it is convex function.
Can a gradient descent method be applied to a convex function?
Gradient descent is a generic method for continuous optimization, so it can be, and is very commonly, applied to nonconvex functions.
When to use gradient descent in continuous optimization?
Gradient descent is a generic method for continuous optimization, so it can be, and is very commonly, applied to nonconvex functions. With a smooth function and a reasonably selected step size, it will generate a sequence of points with strictly decreasing values .
What makes non-convex optimization hard to do?
What makes non-convex optimization hard is the presence of saddle points and local minima, where the gradient is (0,…,0) and that have an arbitrarily bad objective value. Finding the global minmizer in such a setting is generally NP-hard and one instead settles with the goal of finding a local minimizer.
Which is the update rule for non convex stochastic gradient descent?
Non-convex Stochastic Gradient Descent Stochastic Gradient Descent •The update rule is the same for non-convex functions •Same intuition of moving in a direction that lowers objective •Doesn’t necessarily go towards optimum •Even in expectation w t+1= w t↵ trf˜ t(w t)