Contents
Is sin a good activation function?
sin(x) seems to zero centered which is a desirable property for activation functions. Even the gradient won’t vanish at any point.
What is the disadvantage of Heaviside step function when used as an activation function?
A requirement for backpropagation algorithm is a differentiable activation function. However, the Heaviside step function is non-differentiable at x = 0 and it has 0 derivative elsewhere. This means that gradient descent won’t be able to make a progress in updating the weights.
What is PReLU activation function?
A Parametric Rectified Linear Unit, or PReLU, is an activation function that generalizes the traditional rectified unit with a slope for negative values. Formally: f ( y i ) = y i if y i ≥ 0 f ( y i ) = a i y i if y i ≤ 0.
Is there pre-processing step that I need to perform before using cosine?
Is there some pre-processing step that I need to perform before using cosine if not why is it that this activation function works so badly? Cosine is not a commonly used activation function. Looking at the Wikipedia page describing common activation functions, it is not listed.
Why is cosine not a monotonic error surface?
Combined with its periodic nature, this makes it look like it could be particularly tricky to get correct starting conditions and other hyper-parameters in order to have a network learn whilst using it. In addition, cosine is not monotonic, which means that error surface is likely to be more complex than for e.g. sigmoid.
Why is neural network example not working with sigmoid activation?
The neural network learned XOR, using tanh as activation function by default. However, after I changed the activation function to “sigmoid” The output for the 4 inputs are all near 0.5. The result shows that neural network (with the sigmoid function) didn’t learn XOR. Can somebody explain why this example with sigmoid doesn’t work with XOR?
When does the activation function not approximate identity near the origin?
When the activation function does not approximate identity near the origin, special care must be used when initializing the weights. c o s ( 0) = 1, a basic cosine function does not have this property.