Contents
How to train a RNN to write text?
There are several ways we can formulate the task of training an RNN to write text, in this case patent abstracts. However, we will choose to train it as a many-to-one sequence mapper. That is, we input a sequence of words and train the model to predict the very next word.
How is a RNN different from a machine learning model?
A machine learning model that considers the words in isolation — such as a bag of words model — would probably conclude this sentence is negative. An RNN by contrast should be able to see the words “but” and “terribly exciting” and realize that the sentence turns from negative to positive because it has looked at the entire sequence.
How is output sent back to itself in RNN?
With an RNN, this output is sent back to itself number of time. We call timestep the amount of time the output becomes the input of the next matrice multiplication. For instance, in the picture below, you can see the network is composed of one neuron.
How to create a RNN in TensorFlow example?
The object to build an RNN is tf.contrib.rnn.BasicRNNCell with the argument num_units to define the number of input basic_cell = tf.contrib.rnn.BasicRNNCell (num_units=n_neurons) Now that the network is defined, you can compute the outputs and states outputs, states = tf.nn.dynamic_rnn (basic_cell, X, dtype=tf.float32)
How is text as a sequence passed to a RNN?
Text as a sequence is passed to a RNN. The embeddings matrix is passed to embedding_layer. The outpu of embedding layer is passed to LSTM layer. This model has 256 LSTM cells. Here we ignore the hidden states of all of the cells and only take the output from last LSTM cell.
What happens in the first step of a RNN?
During the first step, inputs are multiplied by initially random weights, and bias, transformed with an activation function and the output values are used to make a prediction. This step gives an idea of how far the network is from the reality.