Contents
What does it mean for a learning algorithm to be off-policy?
Off-Policy learning algorithms evaluate and improve a policy that is different from Policy that is used for action selection. In short, [Target Policy != Behavior Policy]. Some examples of Off-Policy learning algorithms are Q learning, expected sarsa(can act in both ways), etc.
Is SARSA or Q-learning better?
If your goal is to train an optimal agent in simulation, or in a low-cost and fast-iterating environment, then Q-learning is a good choice, due to the first point (learning optimal policy directly). If your agent learns online, and you care about rewards gained whilst learning, then SARSA may be a better choice.
What is off-policy evaluation?
Off-policy evaluation (OPE) is the problem of estimating the value of a target policy from samples obtained via different policies. For the theoretical guarantees of an estimator of the policy value, the OPE methods require various conditions on the target policy and policy used for generating the samples.
How can off policy algorithms learn from data?
Off-policy algorithms can (in principle) learn from data without interacting with the environment. This is a nice property, this means that we can collect our data by any means that we see fit and infer the optimal policy completely offline, in other words, we use a different behavioral policy that the one we are optimizing.
What is the difference between off-policy and on-policy learning?
On-policy and off-policy learning is only related to the first task: evaluating Q(s, a). The difference is this: In on-policy learning, the Q(s, a) function is learned from actions that we took using our current policy π(a | s). In off-policy learning, the Q(s, a) function is learned from taking different actions (for example, random actions).
How is Q learning different from off policy?
In Q-Learning, the agent learns optimal policy with the help of a greedy policy and behaves using policies of other agents. Q-learning is called off-policy because the updated policy is different from the behavior policy, so Q-Learning is off-policy.
How are off-policy reinforcement learning algorithms trained?
Instead of using a policy directly to calculate the TD-error update, a separate perturbation policy is trained that applies small perturbations to the actions which are sampled from the generative model. The generative model, in this case, is trained on the data encountered.