Harpreet Sahota 🥑

For hands on content there is nothing better than Francois Chollet’s book: Deep Learning with Python. It’s all in Keras, but you can always use ChatGPT to learn it using PyTorch in parallel.

Whether automation is your cup of tea or not, remember the main steps when cleaning data: • Identify the problematic data • Clean the data • Remove, encode, fill in any missing data • Remove outliers or analyze them separately • Purge contaminated data and correct leaking pipelines • Standardize inconsistent data • Check if your data makes sense (is valid) • Deduplicate multiple records of the same dataForesee and prevent type issues (string issues, DateTime issues) • Remove engineering errors (aka structural errors) • Rinse and repeat

You can consider three approaches: Remove Outliers: Outliers can skew averages and distort statistics. Remove them by eliminating data points in the upper and lower X-percentile of your dataset. Segment Outliers: Group outliers separately from the rest of the data. This is valuable for focused analysis. Ex: identifying high-paying customers who spend significantly more than the average can inform targeted marketing and sales strategies. Use Statistical Methods: Keep outliers in your analysis but employ statistical methods. Weighted means, which give more weight to the "normal" part of the distribution, and trimmed means are common approaches for analyzing datasets with outliers while mitigating their negative impact.

To handle missing data, consider these three main approaches: Drop Rows and Columns: If missing data isn't valuable, remove corresponding rows or entire columns from your analysis. For example, if no one has bought a product, there's no need to analyze the "Quantity of NewAwesomeProduct Bought" column. Recode Missing Data: Transform missing values into a different format. This is useful when numerical computations can't handle missing data. For instance, convert an "empty" payment date into a "payed_yet" column with 0 for "no" and 1 for "yes." Fill in Missing Values: Use techniques like moving averages and backfilling to estimate missing values. This is crucial in time-series analyses, as missing data can distort your conclusions.

Start data cleaning by determining what is wrong with your data. Look for the following: Are there rows with empty values? Entire columns with no data? Which data is missing and why? How is data distributed? Remember, visualizations are your friends. Plot outliers. Check distributions to see which groups or ranges are more heavily represented in your dataset. Keep an eye out for the weird: are there impossible values? Like “date of birth: male”, “address: -1234”. Is your data consistent? Why are the same product names written in uppercase and other times in camelCase? Wear your detective hat and jot down everything interesting, surprising or even weird.

The first step is to put something small into practice and validate it. If it's okay, then it's time to move forward. However, before taking this first step, you must understand the problem's fundamentals. Due to its time-consuming nature, it's a good chance for many to cut corners. Unfortunately, it leads nowhere. Understanding the problem's fundamentals means asking "why" as often as required. Sometimes, it sounds very irritating, but you do it to ensure every step is flawless, logical, and comprehensive. And the next step must stem from the previous one.