Contents
When to impute missing values in training set?
Keeping the past/future analogy in mind, this means anything you do to pre-process or process your data, such as imputing missing values, you should do on the training set alone. You can then remember what you did to your training set if your test set also needs pre-processing or imputing, so that you do it the same way on both sets.
How is data preprocessing used in machine learning algorithms?
For machine learning algorithms to work, it is necessary to convert the raw data into a clean data set and dataset must be converted to numeric data. You have to encode all the categorical lables to column vectors with binary values. Missing values or NaNs in the dataset is an annoying problem.
When to use anomaly detection in unsupervised learning?
In Unsupervised Learning, when I have no labels. The anomaly detection model (Isolation forests, Autoencoders, Distance-based methods etc.), it should fit on a training data and then test ( Train- Test split) just like a common supervised technique of creating the datafolds?
When to use missing values in machine learning?
Missing values can appear as a question mark (?) or a zero (0) or minus one (-1) or a blank. As a result, it is always important that a data scientist always perform exploratory data analysis (EDA) first before writing any machine learning algorithm.
Is it OK to impute missing values with the mean?
Yes. It is fine to perform mean imputation, however, make sure to calculate the mean (or any other metrics) only on the train data to avoid data leakage to your test set. Is it ok to impute mean based missing values with the mean whenever implementing the model?
What happens when you impute both training and testing?
If you fit imputation on both training and testing, then any new testing dataset requires you to re-impute all data again, and this allows leaking information/feature into the model because the information from testing dataset is included in training the model, and consequently, your model won’t be able to predict a new data.
What happens if you remove a value from a test set?
Likewise, if you remove values above some threshold in the test case, make sure that the threshold is derived from the training and not test set.