Contents
What is residual block?
A residual block is a stack of layers set in such a way that the output of a layer is taken and added to another layer deeper in the block. The non-linearity is then applied after adding it together with the output of the corresponding layer in the main path.
What is bottleneck residual block?
A Bottleneck Residual Block is a variant of the residual block that utilises 1×1 convolutions to create a bottleneck. The use of a bottleneck reduces the number of parameters and matrix multiplications. The idea is to make residual blocks as thin as possible to increase depth and have less parameters.
How are bottleneck architectures used in deep networks?
The bottleneck architecture is used in very deep networks due to computational considerations. To answer your questions: 56×56 feature maps are not represented in the above image. This block is taken from a ResNet with input size 224×224. 56×56 is the downsampled version of the input at some intermediate layer.
How are inverted residuals and linear bottlenecks related?
With inverted residual blocks we do the opposite and squeeze the layers where the skip connections are linked. This hurts the performance of the network. The authors introduced the idea of a linear bottleneck where the last convolution of a residual block has a linear output before it’s added to the initial activations.
How are residual blocks used to train ResNets?
While training ResNets, we either train the layers in residual blocks or skip the training for those layers using skip connections. So for different training data points, different parts of networks will be trained at different rates based on how the error flows backwards in the network.
How does a linear bottleneck affect the performance of a network?
This hurts the performance of the network. The authors introduced the idea of a linear bottleneck where the last convolution of a residual block has a linear output before it’s added to the initial activations. Putting this into code is super simple as we simply discard the last activation function of the convolutional block: