Contents
The output of an LSTM cell or layer of cells is called the hidden state. This is confusing, because each LSTM cell retains an internal state that is not output, called the cell state, or c.
Cell state is a memory of LSTM cell, hidden state is an output of this cell. Hidden state and cell input are used to control what to do with memory: to forget or to write new information.
How do you calculate parameters in LSTM?
We can find the number of parameters by counting the number of connections between layers and by adding bias.
- connections (weigths) between layers:
- between input and hidden layer is.
- i * h = 3 * 5 = 15.
- between hidden and output layer is.
- h * o = 5 * 2 = 10.
- biases in every layer.
- biases in hidden layer.
- h = 5.
How to calculate the number of parameters in a LSTM?
Since there are 4 gates in the LSTM unit which have exactly the same dense layer architecture, there will be We can formulate the parameter numbers in a LSTM layer given that $x$ is the input dimension, $h$ is the number of LSTM units / cells / latent space / output dimension:
I was following some examples to get familiar with TensorFlow’s LSTM API, but noticed that all LSTM initialization functions require only the num_units parameter, which denotes the number of hidden units in a cell.
How many layers are there in the LSTM?
*As we will see soon, LSTM has 4 dense layers in its internal structure. So this discussion will help us a lot soon. *Assume that h = size of hidden layer (number of neurons in the hidden layer)
According to what I have learned from the famous colah’s blog, the cell state has nothing to do with the hidden layer, thus they could be represented in different dimensions (I think), and then we should pass at least 2 parameters denoting both #hidden and #cell_state.