What is a disadvantage to using a categorical encoder with a tree-based model?
One-hot encoding categorical variables with high cardinality can cause inefficiency in tree-based ensembles. Continuous variables will be given more importance than the dummy variables by the algorithm which will obscure the order of feature importance resulting in poorer performance.
Do I need to hot encode for decision tree?
Decisions trees work based on increasing the homogeneity of the next level. Thus you won’t need to convert them to integers. You will however need to perform this conversion if you’re using a library like sklearn. One-Hot encoding should not be performed if the number of categories are high.
Are there any downsides to one hot encoding?
Another downside of one-hot encoding is that it causes multicollinearity between the individual variables, which potentially reduces the model’s accuracy. Also, if the categorical variable is an output variable, you may want to convert the values back into a categorical form in order to present them in your application.
How are categorical variables disadvantaged in one hot encoding?
Categorical variables are naturally disadvantaged in this case and have only a few options for splitting which results in very sparse decision trees. The situation gets worse in variables that have a small number of levels and one-hot encoding falls in this category with just two levels.
Why does one hot encoding make your tree-based ensembles worse?
If the tree decides to make a split on one of the dummy variables, the gain in purity per split is very marginal. As a result, the tree is very unlikely to select one of the dummy variables closer to the root.
Which is better random forest or hot encoding?
In general, one hot encoding provides better resolution of the data for the model and most models end up performing better. It turns out this is not true for all models and to my surprise, random forest performed consistently worse for datasets with high cardinality categorical variables.