Contents
How are train data similar to test data?
For each row of training data we calculate a coefficient w = P (test)/P (train). This w tells us how close is the observation from the training data to our test data. Here is the punchline: We can use this w as sample weights in any of our classifier to increase the weight of these observation which seems similar to our test data.
Is there a covariate shift between test and train?
If there exist a covariate shift, then upon mixing train and test we’ll still be able to classify the origin of each data point (whether it is from test or train) with good accuracy. Let’s understand why. Consider the above example where age was the drifting feature between test and train.
How to calculate the coefficient of training data?
Now here is the magic trick: For each row of training data we calculate a coefficient w = P (test)/P (train). This w tells us how close is the observation from the training data to our test data. Here is the punchline:
What happens when your training and testing data come from?
The answer is 1% variance error and 7% data mismatch error. Why so? This time, it is because the classifier performs well on a dataset it hasn’t seen before if it comes from the same distribution, such as the bridge set. It performs poorly if it comes from a different distribution, like the dev set.
What happens when I normalize the train and test data?
After which I will train and test the model ( A, B as features, C as Label) and get some accuracy score. Now my doubt is, what happens when I have to predict the label for new set of data. Say, Because when I normalize the column the values of A and B will be changed according to the new data, not the data which the model will be trained on.
How to find weights in train and test data?
These weights can be calculated using the below code. weights = (1./predictions_train) — 1. You can pass the weights calculated in the model fit method like this: Almost 70% of training samples have sample weight of close to 1 and hence comes from a feature space which is not very specific to train or test high density region.