Contents
How to calculate variable sequence lengths in TensorFlow?
But I’ve tried using a Keras api, X = keras.preprocessing.sequence.pad_sequences (sequences, maxlen=100). And it results in the latter one. Thank you. If you are using a seq2seq model then the encoder inputs must be padded with zeros on the left and the decoder inputs with zeros on the right.
How do you get zero vectors in TensorFlow?
Now that we have a vector holding the sequence lengths, we can pass that to dynamic_rnn (), the function that unfolds our network, using the optional sequence_length parameter. When running the model later, TensorFlow will return zero vectors for states and outputs after these sequence lengths.
Do you have to align sequences to the left in TensorFlow?
In short: Yes, the sequences should be aligned to the left and have additional zeros at the end. Echoing what Rohit mentioned, I want to add that in seq2seq paper, they also found out that reversing the order of input improved their accuracy by a lot. That is to say, you input will be [0,0,0,2,1].
How does weight affect the output of TensorFlow?
Therefore, weights will not affect those outputs and don’t get trained on them. Note that our output will still be of size batch_size x max_length x out_size, but with the last being zero vectors for sequences shorter than the maximum length.
How to index a GRU network in TensorFlow?
Let’s hope the TensorFlow guys can provide proper indexing soon so this gets much easier. On a side node: A one-layer GRU network outputs its full state. In that case, we can use the state returned by tf.nn.dynamic_rnn () directly. Similarly, we can use state.o for a one-layer LSTM network.
How to train RNN with examples of different lengths in keras?
In the keras documentation, it says the input to an RNN layer must have shape (batch_size, timesteps, input_dim). This suggests that all the training examples have a fixed sequence length, namely timesteps. But this is not especially typical, is it?