Contents
Do decision trees require dummy variables?
Since these are categorical variables if we want to apply decision tree classifier and fit to data, first we need to create dummy variables. For a small data-set like this one, we can always use Pandas data-frame features to check this and calculate Gini Impurity for each features.
What is a disadvantage to using a categorical encoder with a tree based model?
One-hot encoding categorical variables with high cardinality can cause inefficiency in tree-based ensembles. Continuous variables will be given more importance than the dummy variables by the algorithm which will obscure the order of feature importance resulting in poorer performance.
How are boosted trees used in model ensembling?
For the boosted trees model, each base classifier is a simple decision tree. This broad technique of using multiple models to obtain better predictive performance is called model ensembling. Random forests improve upon bagged trees by decorrelating the trees.
Why does one hot encoding make your tree-based ensembles worse?
If the tree decides to make a split on one of the dummy variables, the gain in purity per split is very marginal. As a result, the tree is very unlikely to select one of the dummy variables closer to the root.
How are categorical variables disadvantaged in decision trees?
The resulting tree from a dataset containing a majority of continuous variables that would look like something in the figure to the left. Categorical variables are naturally disadvantaged in this case and have only a few options for splitting which results in very sparse decision trees.
Which is worse random forest or tree based ensembles?
It turns out this is not true for all models and to my surprise, random forest performed consistently worse for datasets with high cardinality categorical variables. Being a Kobe Bryant fan, I decided to use his shot selection data to build a model to predict whether a shot given the circumstances would land in or not.
https://www.youtube.com/watch?v=fzzxd2NvwHA