How does discount factor affect reinforcement learning?

How does discount factor affect reinforcement learning?

The discount factor essentially determines how much the reinforcement learning agents cares about rewards in the distant future relative to those in the immediate future. If γ=0, the agent will be completely myopic and only learn about actions that produce an immediate reward.

What is discounted reward?

Discount factor is a value between 0 and 1. A reward R that occurs N steps in the future from the current state, is multiplied by γ^N to describe its importance to the current state. For example consider γ = 0.9 and a reward R = 10 that is 3 steps ahead of our current state.

Why is the discount factor important in reinforcement learning?

Longer time horizons have have much more variance as they include more irrelevant information, while short time horizons are biased towards only short-term gains. The discount factor essentially determines how much the reinforcement learning agents cares about rewards in the distant future relative to those in the immediate future.

What is the basic aim of reinforcement learning?

The basic aim of our RL agent is to maximize the reward. The RL agent basically works on a hypothesis of reward maximization. That’s why reinforcement learning should have best possible action in order to maximize the reward. The cumulative rewards at each time step with the respective action is written as:

What is the trade off in reinforcement learning?

There is an important concept of the exploration and exploitation trade off in reinforcement learning. Exploration is all about finding more information about an environment, whereas exploitation is exploiting already known information to maximize the rewards.

What happens if there is not a discounted problem?

If it was not a discounted problem β = 1 the sum would not converge. All policies that have obtain on average a positive reward at each time instant would sum up to infinity. The would be a infinite horizon sum reward criteria, and is not a good optimization criteria. Here is a toy example to show you what I mean: