4 Tools to Speed Up Exploratory Data Analysis (EDA) in Python

 

4 Tools to Speed Up Exploratory Data Analysis (EDA) in Python

An easy to use one-line of code to automate EDA

Data preparation and Exploratory data analysis take a lot of time and effort from data professionals. Wouldn’t it be nice to have a package(s) that enable you to explore your data quickly? In one line of code?

This article will show the best four packages in Python that can automate your data exploration and analysis. I will go through each one of them, what it does and how you can use it.

  1. DataPrep
  2. Pandas Profiling
  3. SweetViz
  4. AutoViz

DataPrep

The DataPrep ecosystem currently consists of three components: connector, EDA, and Clean API.

The Connector enables a simple data collection from Web APIs by providing a standard set of operations. The EDA component handles the exploration data analysis part, and finally, Clean API provides functions for quickly and efficiently cleaning and validating data.

Forexample, using the Philly Parking Violations dataset, we can call plot() to get an overview of EDA on the data frame or plot correlations with a single line of code, using plot_correlations().

It is also possible to generate a detailed report with one line of code using DataPrep. Here is a create_report() method called on a DataFrame.

import pandas as pd
from dataprep.eda import create_reportdf = pd.read_csv("parking_violations.csv")create_report(df)

And you get back an extensive and interactive report for not only variables but also correlations as well as interactions and missing value.

DataPrep EDS report.

DataPrep eases the amount and effort you need as a data scientist to explore the dataset. With Just one line of code, you can get an overview of your dataset, missing values, correlations, and statistical description of the dataset, as we have seen above.

To install DataPrep, run:

pip install dataprep

Visit also the DataPrep Documentation here for more information:

DataPrep 0.3.0-alpha.0 documentation

DataPrep, documentation site.

docs.dataprep.ai

Pandas Profiling

Pandas profiling also enables you to perform similar EDA as all packages in this article. It has an extensive use case and more tutorials than all of the packages.

With just one line of code, you can generate an EDA report using Pandas Profiling with descriptive statistics, Correlations, Missing value, text analysis, and more.

Let us call ProfileReport() on the Philly DataFrame to generate an EDA report.

from pandas_profiling import ProfileReport
profile = ProfileReport(df, title=”Report”)
profile

Pandas Profiling generates a similar report with a sleek User Interface (UI).

You can install using the pip package manager by running

pip install pandas-profiling[notebook]

Visit the Github repository for more tutorials and documentation:

pandas-profiling/pandas-profiling

Documentation | Slack | Stack Overflow Generates profile reports from a pandas DataFrame. The pandas' df.describe()…

github.com

SweetViz

SweetViz also provides an interactive EDA with just two lines of code. In addition, you can compare two datasets easily, like training and test dataset for your machine learning projects.

To get a report from SweetViz, you can run the following command on any data frame, and it will generate an HTML report.

AutoViz

AutoViz provides similar functionality. You can generate much more detailed plots for your dataset with AutoViz using only one line of code. Here is a report generated with AutoViz using the Philly Parking Dataset.

from autoviz.AutoViz_Class import AutoViz_ClassAV = AutoViz_Class()df_av = AV.AutoViz(‘parking.csv’)

Note that you do not even need Pandas to read the data. AutoViz will load it when you provide the path to the dataset. Here is the report we generate with Autoviz.

In AutoViz, you have much more plots (i.e., violin, boxplots, and more) as well as statistical and probability values. However, the UI is not neat as others Report, and you do not have interactive plots.

To install AutoViz, run the following command:

pip install autoviz

Final Thoughts

The four packages offer almost a similar functionality. You can automate your EDA with simple, intuitive, and one line of code.

Of all the four packages in this article, DataPrep provides much more functionality than simple EDA. It can help you ingest more data sources and offer a speedup for large datasets.


Comments

Post a Comment

Popular posts from this blog

What is Gini Impurity?

Image
  Gini Impurity A measurement used to build Decision Trees to determine how the features of a dataset should split nodes to form the tree. What is Gini Impurity? Gini Impurity is a measurement used to build Decision Trees to determine how the features of a dataset should split nodes to form the tree. More precisely, the Gini Impurity of a dataset is a number between 0-0.5, which indicates the likelihood of new, random data being misclassified if it were given a random class label according to the class distribution in the dataset. For example, say you want to build a classifier that determines if someone will default on their credit card. You have some labeled data with features, such as bins for age, income, credit rating, and whether or not each person is a student. To find the best feature for the first split of the tree – the root node – you could calculate how poorly each feature divided the data into the correct class, default ("yes") or didn't default ("no&quo
Read more

Artificial Neural Network Interview Questions

Image
  Artificial Neural Network Interview Questions 1. Explain Neural Network?  Neural Network is a sequence of an algorithm that gives its best to recognize the underlying relationship in a data-set through a process. The artificial neural network has several differences from biological brains. These networks play a crucial role in deep learning.    2. Explain Biological Neural Network and Artificial Neural network? Biological Neural Network  is prepared with the help of real neurons. Our nervous system is made via neurons and brain. A biological neural network is our nervous system in which neurons are linked with each other. Human has a mind to understand and can operate any task in a particular situation, but how can a robot do that? For that purpose, an artificial brain is designed, which is known as  Neural Network . The human brain has neurons to transfer the information or data. Similarly, the neural network has nodes to perform the task. Nodes are the mathematical functions. 3. Wh
Read more

20+ Questions to Test your Skills on Logistic Regression

Image
1. What do you mean by the Logistic Regression? It’s a classification algorithm that is used where the target variable is of categorical nature. The main objective behind Logistic Regression is to determine the relationship between features and the probability of a particular outcome. For Example,  when we need to predict whether a student passes or fails in an exam given the number of hours spent studying as a feature, the target variable comprises two values i.e. pass and fail. Therefore, we can solve classification problem statements which is a supervised machine learning technique using Logistic Regression. 2. What are the different types of Logistic Regression? Three different types of Logistic Regression are as follows: 1. Binary Logistic Regression:  In this, the target variable has only two 2 possible outcomes. For Example,  0 and 1, or pass and fail or true and false. 2. Multinomial Logistic Regression:  In this, the target variable can have three or more possible values witho
Read more