Contents
What is weight sharing in RNN and why is it useful?
Weight Sharing across the time stamps thus helps in understanding the sequence as well as in applied point of view reduces the training time.
To reiterate parameter sharing occurs when a feature map is generated from the result of the convolution between a filter and input data from a unit within a plane in the conv layer. All units within this layer plane share the same weights; hence it is called weight/parameter sharing.
How does weight sharing work in a CNN?
A CNN has multiple layers. Weight sharing happens across the receptive field of the neurons (filters) in a particular layer. Weights are the numbers within each filter. These filters act on a certain receptive field/ small section of the image. When the filter moves through the image, the filter does not change.
How to calculate trainable weights without weight sharing?
We are going to derive the number of trainable weights without weight sharing and also with weight sharing, within the first convolutional layer of two popular CNN architectures: LeNet and AlexNet. (Input size width — filter size + (2*Padding) / stride )+ 1 = output width of convolutional layer 2.
How are weight values learnable in a CNN?
Each of these filters has a set width and height, which corresponds to the local receptive field of a single unit within the layer. The filters acting upon the input data creates the output of a convolutional layer, the feature map. The weight values within filters are learnable during the training phase of a CNN.
What’s the difference between a CNN and a RNN?
RNNs also employ the idea of weight sharing. They use a network that faces each input in the sequence separately to see whether a similar pattern is observed or not. However, there is a difference between CNNs and RNNs. The input of RNNs is a sequence, and the order matters.