What is the limitation of ReLU?

What is the limitation of ReLU?

Key among the limitations of ReLU is the case where large weight updates can mean that the summed input to the activation function is always negative, regardless of the input to the network. This means that a node with this problem will forever output an activation value of 0.0. This is referred to as a “dying ReLU“.

What is the advantage of using ReLU?

The ReLU function is another non-linear activation function that has gained popularity in the deep learning domain. ReLU stands for Rectified Linear Unit. The main advantage of using the ReLU function over other activation functions is that it does not activate all the neurons at the same time.

Can a ReLU function handle a negative input?

With the backpropagation algorithm it should be possible that the outputs of the previous hidden layers are changed in such a way that, eventually, the input to the ReLU function will become positive again. Then the ReLU would not be dead anymore.

Can a ReLU function be used in a hidden layer?

If you really use an activation function with the input layer, I would suggest either using another activation function like ELU or transform your data to the range [0,1], for example. If the ReLU function is in some hidden layer, the ReLU function should become dead only temporarily.

When to use relu in a neural network?

You can use relu function as activation in the final layer. You can see in the autoencoder example at the official TensorFlow site here. Use the sigmoid/softmax activation function in the final output layer when you are trying to solve the Classification problems where your labels are class values.

Is there any way to feed the data into a RELU network?

Is there any way to feed the data into a ReLU network without converting it all to positive and having a separate input which says if the data is negative or positive?