How do you bring a validation loss down?
Solutions to this are to decrease your network size, or to increase dropout. For example you could try dropout of 0.5 and so on. If your training/validation loss are about equal then your model is underfitting. Increase the size of your model (either number of layers or the raw number of neurons per layer)
Why is validation loss more than training loss?
Your model is completely overfitting. The training loss is constantly decreasing but the validation loss isn’t. This means that the your current model is complex enough to ‘memorize’ the patterns in the training data. In such situations, you need to regularize your model.
Why is my validation loss lower than my training loss?
Regularization methods often sacrifice training accuracy to improve validation/testing accuracy — in some cases that can lead to your validation loss being lower than your training loss. Secondly, keep in mind that regularization methods such as dropout are not applied at validation/testing time.
Can you change the size of the validation set?
These sizes cannot be changed. Early stopping is used based on the validation set to prevent overfitting (which is of course highly likely on this small training dataset). There’s a separate test set for evaluation afterwards. The datasets are fairly unbalanced with ~15% of observations being one class.
How many observations are in the validation set?
The training set is 200 observations and the validation set is 50 observations. These sizes cannot be changed. Early stopping is used based on the validation set to prevent overfitting (which is of course highly likely on this small training dataset). There’s a separate test set for evaluation afterwards.
Do you ignore validation loss when optimizing cross entropy?
To put it simply, you’re optimizing cross-entropy in hopes it gives you parameters that help your model be accurate. But that isn’t a necessary nor a sufficient condition for accuracy. Common practice is to ignore your validation loss and focus on your target metric: accuracy.