Contents
Why do we need a target network in DQN?
An important component of DQN is the use of a target network, which was introduced to stabilize learning. In Q-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state.
How is the deep Q-learning algorithm different from Q-learning?
A core difference between Deep Q-Learning and Vanilla Q-Learning is the implementation of the Q-table. Critically, Deep Q-Learning replaces the regular Q-table with a neural network. Rather than mapping a state-action pair to a q-value, a neural network maps input states to (action, Q-value) pairs.
What is Target Q Network?
The Target network predicts Q values for all actions that can be taken from the next state, and selects the maximum of those Q values. Use the next state as input to predict the Q values for all actions. The target network selects the max of all those Q-values.
What is double deep Q-Network?
A Double Deep Q-Network, or Double DQN utilises Double Q-learning to reduce overestimation by decomposing the max operation in the target into action selection and action evaluation. We evaluate the greedy policy according to the online network, but we use the target network to estimate its value.
Why is a target network required in deep learning?
So, in summary a target network required because the network keeps changing at each timestep and the “target values” are being updated at each timestep? The difference between Q-learning and DQN is that you have replaced an exact value function with a function approximator.
How are target networks used in Q training?
To make training more stable, there is a trick, called target network, by which we keep a copy of our neural network and use it for the Q (s’, a’) value in the Bellman equation. That is, the predicted Q values of this second Q-network called the target network, are used to backpropagate through and train the main Q-network.
How is Q learning used in deep learning?
Q-learning is value-based reinforcement learning algorithm that learns “optimal” probability distribution between state-action that will maximize it’s long term discounted reward over a sequence of timesteps. The Q-learning is updated using the bellman equation, and a single step of the q-learning update is given by
How are q values used to train the Q Network?
The idea is that using the target network’s Q values to train the main Q-network will improve the stability of the training. Later, when we present the code of the training loop, we will enter in more detail how to code the initialization and use of this target network.