Contents
How to handle columns with categorical data and many unique values?
I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world. I also have another column with 145 nunique values that I could also use in my model that represents product category. Can I use one hot encoding to these columns or there’s a problem with that solution?
How to handle large number of categorical values?
One of the ideas is to divide the 3000 variables into fewer groups based on either some dependent variable in data set or based on information gain on outcome variable. Lets say if you have outcome variable 0/1 and ratio of it 12%.
Why are m distinct labels important for categorical data?
Essential for any categorical feature of m distinct labels, you get m separate features. This can easily increase the size of the feature set causing problems like storage issues, model training problems with regard to time, space and memory.
How to select features in a categorical variable?
You can try to select features using correlation plots and select some of the top features (If you have not done that yet) or else I believe Random forest model to get feature importance, a lot of people do that to get feature importance. You can try binning techniques if your categorical variables have a lot of distinct values.
When do you use categorical data in Excel?
When using categorical data, you usually convert those to either number labels (one additional column with one integer number for each different entry) or use a one-hot encoding (x new columns for x categories, each with a 1 if the category is present for that row). Both have their advantages and disadvantages.
How to handle categorical column missing data in Excel?
Step 1. Create a new column and replace 1 if the category is NAN else 0. This column is an importance column to the imputed category. Step 2. Replace NAN value with most occurred category in the actual column. # 2. Take most occured category in that vairable (.mode ())
Can you check which columns in Dataframe are categorical?
BE CAREFUL – As @Sagarkar’s comment points out that’s not always true. The difficulty is that Data Types and Categorical/Ordinal/Nominal types are orthogonal concepts, thus mapping between them isn’t straightforward. @Jeff’s answer below specifies the precise manner to achieve the manual mapping.