Contents
When to use sub-gradient descent in convex optimization?
InSection2, we formally introduce (sub-)gradient descent, and prove explicit convergence rates when gradient descent is applied to convex functions. Section3introduces a stronger assumption know as strong convexity, which allows (sub-)gradient descent to enjoy even faster rates.
What makes non-convex optimization hard Cornell University?
Non-Convex Optimization CS6787 Lecture 7—Fall 2017 Review —We’ve covered many methods •Stochastic gradient descent •Mini-batching •Momentum •Variance reduction •Nice convergence proofs that give us a rate •But only for convex problems! Non-Convex Problems •Anything that’s not convex What makes non-convex optimization hard?
How long is a lecture on convex optimization?
The focus is on methods that arise in machine learning and modern data analysis, highlighting concerns about complexity, robustness, and implementation in these domains. Each section in these notes roughly corresponds to an 80 minutes lecture, adding up to one semester course.
Can a neural net approximate a convex function?
•To do this, they need to be able to approximate non-convex functions •Convex functions can’t approximate non-convex ones well. •Neural nets also have many symmetric configurations •For example, exchanging intermediate neurons •This symmetry means they can’t be convex.
When to use Taylor’s theorem in convex optimization?
If f :R! R is twice-continuously differentiable, then Taylor’s theorem tells us that that f(x +d) ˇf(x)+df0(x)+ 1 2 d2f00(x). This approximation directly reveals that if we move from x to x +dwhered=hf0(x) for sufficiently smallh> 0, we generally expect to decrease the function value by about
Which is a stronger assumption, strong convexity or strong gradient descent?
Section3introduces a stronger assumption know as strong convexity, which allows (sub-)gradient descent to enjoy even faster rates. [Max’s Note:finish]