Tips and Tricks
What is high cardinality?
Almost all datasets now have categorical variables. Each categorical variable consists of unique values. A categorical feature is said to possess high cardinality when there are too many of these unique values. One-Hot Encoding becomes a big problem in such a case since we have a separate column for each unique value (indicating its presence or absence) in the categorical variable. This leads to two problems, one is obviously space consumption, but this is not as big a problem as the second problem, the curse of dimensionality. I will talk about the curse of dimensionality in more detail but first, let’s look at the data before and after one-hot encoding.
A peek at our categorical feature before and after one-hot encoding
We will be looking at the Qualifications feature. Since this data was collected from a form filled by many people the column contains many different Qualifications. Here is what the column looks like and all the unique values it has.
We can see that there are 15 unique values in the feature and that it consumes 316KB of space. Let’s one_hot encode this feature using pandas.
We now see how big our original feature has become and naturally the space required to store it has increased to 592KB. This is just ONE feature, if during training we possess hundreds of categorical variables we will end up with many hundreds of features which is not conducive for model training in some cases. Simple models cannot handle so many variables. But now let’s look at another major problem, the curse of dimensionality.
The Curse of Dimensionality
Here is a simple summarization:
As the number of features grows, the amount of data we need to accurately be able to distinguish between these features (in order to give us a prediction) and generalize our model (learned function) grows EXPONENTIALLY.
Feel free to skip to the next section if you do not want to read the technicalities below.
I would like to use Yoshua Bengio’s (Yes the legendary Yoshua Bengio !) quora answer to explain this in more detail. I strongly advise reading the whole answer here. According to the answer, increasing the number of different values in a feature simply increases the total number of possible combinations that can be made using the input row (containing n such features). Say we have two features with two distinct values each, this gives us a total of 4 possible ways to combine the two features. Now if one of these had three distinct values we would have 3X2 =6 possible ways to combine them.
In classical non-parametric learning algorithms (e.g. nearest-neighbor, Gaussian kernel SVM, Gaussian kernel Gaussian Process, etc.) the model needs to see at least one example for each of these combinations (or at least as many as necessary to cover all the variations of configurations of interest), in order to produce a correct answer, one that is different from the target value required for other nearby configurations.
There is a workaround to this, that is the model even in the absence of a lot of training data can discern between configurations (not in the training set) for future predictions provided there is some sort of structure (pattern) in these combinations. In most cases, high cardinality makes it difficult for the model to identify such patterns and hence the model doesn’t generalise well to examples outside the training set.
Reducing Cardinality by using a simple Aggregating function
Below is a simple function I use to reduce the cardinality of a feature. The idea is very simple. Leave instances belonging to a value with high frequency as they are and replace the other instances with a new category which we will call other.
- Choose a threshold
- Sort unique values in the column by their frequency in descending order
- Keep adding the frequency of these sorted (descending) unique values until a threshold is reached.
- These are the unique categories we will keep and instances of all other categories shall be replaced by "other".
Let’s run through a quick example before going through the code. Say our column colour has 100 values and our threshold is 90% (that is 90). We have 5 different categories of colours: Red (50), Blue(40), Yellow (5), Green (3) and Orange (2). The numbers within the bracket indicate how many instances of that category are present in the column.
We see that Red (50)+Blue (40) reaches our threshold of 90. In that case, we retain only 2 categories (Red, Blue) and mark all other instances of other colours as "Other".
Thus we have reduced cardinality from 5 to 3 (Red, Blue, Other)
Here is the utility function I wrote to facilitate this. It’s well commented and follows exactly what I described above so you won’t have a problem following along. We can set a custom threshold and the return_categories option optionally lets us see the list of all unique values after reducing cardinality.
As you can see using this function we reduced the cardinality of the qualifications column from 15 to 6!
Conclusion
We saw how to reduce cardinality by using a simple function and more importantly why that is necessary (curse of dimensionality). Keep in mind however that we were fortunate that the distribution of values in our column allowed us to use this method. If all the 15 categories had been equally distributed we would not have been able to use this and in that case would probably need to apply PCA in combination with the other features of the dataset but more on that some other time.
If you liked this article here are some more!
Scatter Plots on Maps using Plotly
Check out my GitHub for some other projects. You can contact me here. Thank you for your time!
