Contents
The output of an LSTM cell or layer of cells is called the hidden state. This is confusing, because each LSTM cell retains an internal state that is not output, called the cell state, or c.
What is a hidden state?
Hidden Markov model is basically a Markov chain whose internal state cannot be observed directly but only through some probabilistic function. That is, the internal state of the model only determines the probability distribution of the observed variables.
Cell state is a memory of LSTM cell, hidden state is an output of this cell. Hidden state and cell input are used to control what to do with memory: to forget or to write new information. We decide what to do with memory knowing about previous output (hidden state) and current input.
What are hidden states LSTM?
A key idea in LSTM is the (star)Gate. h is the hidden state, representing short term memory. C is the cell state, representing long term memory and x is the input. The gates perform only few matrices transformations, sigmoid and tanh activation in order to magically solve all the RNN problems.
“An RNN has a looping mechanism that acts as a highway to allow information to flow from one step to the next. Passing Hidden State to next time step. This information is the hidden state, which is a representation of previous inputs.
What’s the difference between cell state and hidden state?
To summarise: RNNs are great, but issues occur with log terms dependencies because of the chain rule in their hidden state. To alleviate the issues above, LSTM architectures introduce the cell state, additional to the existing hidden state of RNNs. Cell states give the model longer memory of past events.
The hidden state and the cell state could in turn be used to initialize the states of another LSTM layer with the same number of cells. We can access both the sequence of hidden state and the cell states at the same time. This can be done by configuring the LSTM layer to both return sequences and return states.
When to use hidden state and cell state in LSTM?
The LSTM cell state for the last time step. The hidden state and the cell state could in turn be used to initialize the states of another LSTM layer with the same number of cells. We can access both the sequence of hidden state and the cell states at the same time.
Each unit or cell within the layer has an internal cell state, often abbreviated as “ c “, and outputs a hidden state, often abbreviated as “ h “. The Keras API allows you to access these data, which can be useful or even required when developing sophisticated recurrent neural network architectures, such as the encoder-decoder model.