Contents
How can train error be reduced?
A modern approach to reducing generalization error is to use a larger model that may be required to use regularization during training that keeps the weights of the model small. These techniques not only reduce overfitting, but they can also lead to faster optimization of the model and better overall performance.
How do I reduce overfitting in Lstm?
1 Answer. You could try: Reduce the number of hidden units, I know you said it already seems low, but given that the input layer only has 80 features, it actually can be that 128 is too much. A rule of thumb is to have the number of hidden units be in-between the number of input units (80) and output classes (5);
Which is larger validation loss or training loss?
As I am fitting the model, training loss is constantly larger than validation loss, even for a balanced train/validation set (5000 samples each): In my understanding the two curves should be exactly the other way around such that training loss would be an upper bound for validation loss. Predictions are more or less ok here.
Is it better to have small error gap in cross validation?
But that does not mean that a smaller gap means a better model; it’s just that if we have a small or no gap between training and test set performance, we know we are definitely not overfitting so adding regularization/introducing more bias to the model will not help. Thanks for contributing an answer to Cross Validated!
Is the gap between training and validation accuracy a problem in itself?
So to answer your question, if you have used the same data both for training and testing then it’s a bad practice. You must atleast split your data into training and testing datasets (80:20 or 70:30). But for better results split them into 3 datasets i.e. training, cross validation and testing (70:10:20).
What does a gap between train and test error mean?
However, then there’s advice that you do see, from very good sources that suggest that a gap between train and test error is indicative of overfitting.