Contents
- The number of hidden neurons should be between the size of the input layer and the size of the output layer.
- The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.
- The number of hidden neurons should be less than twice the size of the input layer.
Generally, 2 layers have shown to be enough to detect more complex features. More layers can be better but also harder to train. As a general rule of thumb — 1 hidden layer work with simple problems, like this, and two are enough to find reasonably complex features.
How is the hidden state of a RNN calculated?
Each unrolled RNN unit has a hidden state. The current time steps hidden state is calculated using information of the previous time step’s hidden state and the current input. This process helps to retain information on what the model saw in the previous time step when processing the current time steps information.
– If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state. – The best we can do is to infer a probability distribution over the space of hidden state vectors. • This inference is only tractable for two types of hidden state model.
The current time steps hidden state is calculated using information of the previous time step’s hidden state and the current input. This process helps to retain information on what the model saw in the previous time step when processing the current time steps information.
What is the main speciality of a RNN?
As we mentioned earlier the main speciality in RNNs is the ability to model short term dependencies. This is due to the hidden state in the RNN. It retains information from one time step to another flowing through the unrolled RNN units.