Using PyCaret to generate a comprehensive EDA report in 1 line of code.
What is EDA and why is it essential?
Before we jump to model building, understanding the data at hand is essential. Analysing the data alone can give us valuable insights to solve our problems. Moreover understanding data is very useful to determine which features would help our model, which features can be done away with , how can we deal with missing values & correlations and so on. Building a model after considering all these factors will ensure that it is robust & can be generalised well.
What is PyCaret?
Since the data science pipeline from EDA to model prototyping is somewhat structured (for simple ML tasks at least ) we can use tools like PyCaret to automate EDA, model building and tuning hyper-parameters. It has a ton of other features like bagging , boosting, stacking models, generating polynomial features from data , transforming categorical data with high cardinalities and many more, all of which I will talk about in a later article. Today our focus will be on EDA. It is worthwhile to note that PyCaret uses pandas-profiling under the hood to perform EDA.
Let’s generate an EDA report for a regression problem in seconds.
I have used the popular California Housing Prices dataset (1990) here. Let’s try and analyse what features affect housing prices within a block. I would strongly recommend users to try this code on Google Colab or Jupyter unless your device has enough RAM.
- Installing PyCaret and making it Colab ready.
!pip install pycaret
from pycaret.utils import enable_colab
enable_colab()
import pandas as pd
from pycaret.regression import *dataset=pd.read_csv('/content/housing.csv') #Read the Csv File- Generate the EDA report !
exp1 = setup(data = dataset, target = 'median_house_value', session_id=123 , profile=True)Make sure that the profile parameter is set to True
Let’s see what we got
Firstly it’s remarkable that the report was generated in only 18.99 seconds. Plus we have this really cool navbar to jump to sections within the report.
Quickly head over to sample which basically shows us the head of the dataset.
Next up
The overview section tells us about the dataset structure, the number of missing cells , duplicate cells, size taken up by the dataset and the number of variables along with their types.
Let’s observe a numeric feature in detail (Longitude) under the Variables section.
Once we click on toggle details under the variable name we have tons of data, the quantile statistics, standard deviations , distinct values within the column. This is just the beginning! Once we switch over to the histogram tab we can see the distribution of the variable.
We also have the Common Values and Extreme Values tabs here that show us the frequency of the values in the column & also the Minimum and Maximum values in the column.
Now let’s look at a categorical variable (Ocean_Proximity)
On switching to the categories tab we get some amazing plots , especially the all important pie chart. We can also see the distinct values in our columns & their counts through charts.
Impressive Right? There’s More !
Interactive Bivariate Analysis
Under the interactions tab we can see how two variables are correlated. Feel free to play around with the features. As an example we can see how median_income is correlated with median_house_value.
Heatmaps depicting correlations.
We can see how all numeric features are correlated & also toggle the correlation types. The report also describes how that specific type of correlation is calculated.
Missing Values !
This section tells us about the missing values in our columns. My favourite feature is the Matrix depiction of missing values since it shows us the rows that contain missing values ! In the diagram below, the white horizontal lines indicate the missing values.
Conclusion
This EDA report is super helpful and covers almost all the basic requirements we need for simple EDA tasks. It does miss out on some important features like One Way Anova Test & Chi -Square Tests but you can’t have it all in 19 seconds and one line of code!
Although this is very handy, I feel that EDA is not a sclerotic process and often for tough problems creative techniques that are not part of this library at the moment can be very insightful.
Check out my GitHub for some other projects and the entire code. You can contact me on my _website._ Thank you for your time!
