Contents
What are the drawbacks of linear function approximation?
The main drawback of linear function approximation compared to non-linear function approximation, such as the neural network, is the need for good hand-picked features, which may require domain knowledge. In the non-linear function approximator we will redefine once again the state and action value function V and Q such as:
How are function approximators used in gradient descent?
The methods that compute these approximations are called Function Approximators. There are many function approximators: … Since we will use gradient descent in order to find the best result, the function approximators must be differentiable, which leads us to Linear combinations of features and Neural Networks.
Which is more powerful nonlinear or linear function approximators?
There are also results indicating that nonlinear function approximators may be more powerful in general than linear function approximators for learning high-dimensional functions.
How are function approximators used in the state space?
The function approximator exploits the structure in the state space to efficiently learn the value of observed states and generalise to the value of similar, unseen states.
How to use Q-values with function approximation?
To use Q-values with function approximation, we need to find features that are functions of states and actions. This means in the linear function regime, we have What’s tricky about this, however, is that it’s usually a lot easier to reason about features that are only functions of the states.
How to calculate function approximation in reinforcement learning?
Suppose an agent is in a 4×4 grid, so the location of the of the agent on the grit is a feature. This gives 16 different locations meaning 16 different states. Bu t that’s not all, suppose the orientation (north, south, east, west) is also a feature. This gives 4 possibilities for each location, which makes the number of states to 16*4 = 64.