Contents
How does Xavier initialization work?
Xavier initialization works with tanh activations. Myriad other initialization methods exist. If you are using ReLU, for example, a common initialization is He initialization (He et al., Delving Deep into Rectifiers), in which the weights are initialized by multiplying by 2 the variance of the Xavier initialization.
What is Xavier initializer?
Xavier initialization, originally proposed by Xavier Glorot and Yoshua Bengio in “Understanding the difficulty of training deep feedforward neural networks”, is the weights initialization technique that tries to make the variance of the outputs of a layer to be equal to the variance of its inputs.
Why does Xavier initialization work?
Why’s Xavier initialization important? In short, it helps signals reach deep into the network. If the weights in a network start too small, then the signal shrinks as it passes through each layer until it’s too tiny to be useful.
How to do Xavier initialization of a neural network?
Xavier Initialization 1 Vanishing Gradients. Initially, one of the challenges preventing the efficient training of very deep neural networks was the phenomenon of extreme gradients. 2 Glorot and Bengio. 3 Notation and Assumptions. 4 Forward pass. 5 Backwards pass. 6 Conclusion.
How does Xavier initialization work in Caffe library?
We arrived at the Xavier initialization formula. We need to pick the weights from a Gaussian distribution with zero mean and a variance of 1/N, where N specifies the number of input neurons. This is how it’s implemented in the Caffe library. In the original paper, the authors take the average of the number input neurons and the output neurons.
What’s the difference between Xavier and Kaiming initialization?
It’s clear we need a better way of initializing the network, that doesn’t lead to exploding or vanishing activations and gradients. That’s (one of) the contribution of Xavier initialization, that was later refined into Kaiming initialization. The Xavier and Kaiming papers follow a very similar reasoning, that differs a tiny bit at the end.
How many hidden layers in the Xavier paper?
The Xavier paper worked with neural networks with (only!) 5 hidden layers, but the effect is already very visible if no care is given to initialization. Here’s the histogram of the mean values of the activations: And here’s the histogram of the mean values of the backpropagated gradients: