Contents
What is dueling deep Q Network?
A Dueling Network is a type of Q-Network that has two streams to separately estimate (scalar) state-value and the advantages for each action. The two streams are combined via a special aggregating layer to produce an estimate of the state-action value function Q as shown in the figure to the right.
What is Q target?
Moving Q-Targets As we saw, the first component of the TD Error (TD stands for Temporal Difference) is the Q-Target and it is calculated as the immediate reward plus the discounted max Q-value for the next state. When we train our agent, we update the weights accordingly to the TD Error.
How does the dueling Double DQN theory work?
Dueling DQN (aka DDQN) Theory. Remember that Q-values correspond to how good it is to be at that state and taking an action at that state Q(s,a). So we can decompose Q(s,a) as the sum of: V(s): the value of being at that state.
How to calculate Q ( s, a ) in ddqn?
A (s,a): the advantage of taking that action at that state (how much better is to take this action versus all other possible actions at that state). With DDQN, we want to separate the estimator of these two elements, using two new streams: And then we combine these two streams through a special aggregation layer to get an estimate of Q (s,a).
How are value and advantage functions used in dueling Q networks?
Advantage function captures how better an action is compared to the others at a given state, while as we know the value function captures how good it is to be at this state. The whole idea behind Dueling Q Networks relies on the representation of the Q function as a sum of the Value and the advantage function.
Can a ddqn learn to decouple the estimation?
By decoupling the estimation, intuitively our DDQN can learn which states are (or are not) valuable without having to learn the effect of each action at each state (since it’s also calculating V (s)). With our normal DQN, we need to calculate the value of each action at that state. But what’s the point if the value of the state is bad?