Contents
Why does pre-training help?
Pre-training finds a good spot of weights in the error surface. Intuitively, were finding a good set of weights to compressing the input data in the pre-training phase. Ideally, this compressed representation is good for solving generic tasks.
Is Pretraining necessary?
1 Answer. No, pretraining was mostly used in the days when sigmoid was the most used activation function. Since it suffered from vanishing gradient problem, unsupervised pretraining helped a lot. With ReLUs you don’t have such problem, so pretraining is just an artifact of the past.
What can be done about the vanishing gradient problem?
Since these initial layers are often crucial to recognizing the core elements of the input data, it can lead to overall inaccuracy of the whole network. The simplest solution is to use other activation functions, such as ReLU, which doesn’t cause a small derivative.
When does a gradient become too small for training?
Note how when the inputs of the sigmoid function becomes larger or smaller (when |x| becomes bigger), the derivative becomes close to zero. For shallow network with only a few layers that use these activations, this isn’t a big problem. However, when more layers are used, it can cause the gradient to be too small for training to work effectively.
What is the problem with vanishing gradients in FFN?
This is referred to as the “ exploding gradient ” problem. The term vanishing gradient refers to the fact that in a feedforward network (FFN) the backpropagated error signal typically decreases (or increases) exponentially as a function of the distance from the final layer.
Vanishing gradients is a particular problem with recurrent neural networks as the update of the network involves unrolling the network for each input time step, in effect creating a very deep network that requires weight updates. A modest recurrent neural network may have 200-to-400 input time steps, resulting conceptually in a very deep network.