Data and Exploratory Data Analysis

Exploratory data analysis is the step everyone is tempted to skip and nobody who has been burned skips twice. Its purpose is not to produce charts. It is to answer, before any model exists:

• What is actually in this data, as opposed to what the schema claims?
• Which variables plausibly carry signal, and which are noise, identifiers or leakage?
• What will break the moment a model touches it — missing values, impossible values, mixed types?
• Is the question even answerable with these columns?

Chapter 5 fixes what this chapter finds. Chapter 6 builds on what it suggests.

4.1 Kinds of Data and Kinds of Variables

Every decision downstream — which plot, which encoding (Chapter 6), which distance metric (14.2), which metric (8.4 vs 8.5) — follows from a variable's type. Getting this table right first is not pedantry; it is the cheapest way to avoid a whole class of later mistakes.

The last row is a rule, not a suggestion. An identifier is unique per row, so it can only ever be memorized, never generalized from.

Reading that list top-down is itself a typing exercise: 2 unique values is binary, a handful is categorical, and a count equal to (or near) the row count is an identifier or a continuous measurement. customer_id has 1000 unique values in 1012 rows — unique except for the duplicate rows found in 3.9.1.

That heuristic gets you started and is wrong in four instructive places:

Cardinality is a hint; the schema and the domain decide. The last two rows are the ones that matter most in practice: a low-cardinality integer can be a category (plan encoded 0/1/2) or a count (support_tickets), and nothing in the data distinguishes them — only knowing what the column means does.

Common mistake — feeding an encoded category to a model as a number. If city is stored as 0–7, a model reading it numerically concludes that Jaipur (7) is "greater than" Kolkata (0) and that the midpoint between them means something. It does not. Only genuinely ordered variables — plan, a rating, a size — may be encoded as ordered integers, and 6.2 covers the alternatives for the rest.

4.2 Getting Data: bundled datasets, synthetic generators and files

Three sources, three purposes. Nothing in this material downloads anything (§0.7).

The make_* generators matter more than they first appear: because you specify the truth, you can check whether a method recovers it. That is how 7.2's feature selection and 9.2's bias–variance simulation are validated at all.

4.3 First Contact: shape, dtypes and the summary triage

3.3.1 introduced head/info/describe. The EDA version adds two things: describe the categoricals too, and write down what you find.

The categorical summary earns its place immediately: top and freq show the most common value, and unique shows cardinality. gender reporting 6 unique values for a two-valued concept is the whole finding, visible without a single plot.

Four numbers to read off the numeric summary every time:

Here age has 59 missing, a minimum of 0 and a maximum of 199 — both impossible for a subscriber, and both would silently corrupt any mean, any scaler and any distance computation.

4.3.1 Sentinel values

A specific trap the summary catches: some systems encode "missing" as a number — -1, 0, 999, -9999. These are not detected by .isna(), and they are catastrophic because they are arithmetically valid. A minimum of exactly -1 or 0 on a quantity that cannot be either is the tell.

4.4 Univariate Analysis

One variable at a time. The question is always the same: what shape is this, and is the shape sensible?

Four findings from one figure:

1. age is a normal-looking bump plus a couple of impossible points stretching the axis to 199 — the giveaway is that the bump occupies only the left fifth of the plot.
2. monthly_charges is clearly multimodal. That is not noise: it is three plans at three price points, mixed into one column. 4.5 confirms it.
3. support_tickets and last_login_days are right-skewed counts, the classic shape for a log or square-root transform (5.6).
4. tenure_months is close to uniform, which in real data would be suspicious — here it is an artefact of how data/customers.csv was generated, and worth naming as such rather than over-interpreting.

Quantify the shape rather than eyeballing it:

age's skew of 2.69 and kurtosis of 28.9 are caused entirely by four impossible values. Kurtosis is the more sensitive alarm of the two — it weights extreme deviations to the fourth power, so a handful of absurd rows moves it enormously while the median does not budge. Both are more reliable outlier alarms than a histogram, because neither depends on the bin width. Recomputing after excluding the impossible values shows how much of the "shape" was four rows:

For categorical variables the equivalent is value_counts, and the thing to look for is rare categories — levels with so few rows that any statistic computed on them is noise (and that may not appear in the training split at all, 6.2).

4.5 Bivariate Analysis

Two variables at a time. The right tool depends on the pair of types:

Start with the multimodality found in 4.4 — the hypothesis was that plan explains it.

The three modes are the three plans. This matters beyond tidiness: a model given monthly_charges alone has to discover that structure, whereas one given plan as well is handed it. It also means the two columns are strongly related, which is a multicollinearity risk for linear models (11.7).

Numeric × numeric, with the target as colour — a scatter that answers a question rather than decorating:

Categorical × categorical, as rates rather than counts. Counts are almost always the wrong view: a large category will dominate a stacked bar regardless of its rate.

Every confidence interval overlaps every other one. There is no evidence here that any city differs from any other, and reporting a ranking would be inventing a finding.

4.6 Multivariate Analysis and Correlation Structure

More than two variables at once. Two standard views:

Read a correlation heatmap for two different things:

• Feature–target correlation — candidate signal.
• Feature–feature correlation — redundancy, which hurts linear-model interpretation (11.7) and wastes capacity everywhere else (7.2).

Here satisfaction_score correlates −0.70 with support_tickets: unhappy customers file tickets, which is sensible and also means the two columns say much the same thing. Everything else is below 0.19 in absolute value, so there is no serious redundancy elsewhere.

4.6.1 Simpson's paradox

The reason multivariate analysis is not optional: an association can reverse when a third variable is taken into account. This is not a curiosity — it is a live risk whenever groups of different sizes and different base rates are pooled.

Treatment B wins in the easy department (0.95 vs 0.90) and wins in the hard department (0.30 vs 0.20), and loses overall (0.43 vs 0.76). Nothing is miscalculated: B was mostly applied in the hard department, where everything scores badly, and A was mostly applied in the easy one.

The lesson for EDA is exact: a marginal statistic can point the opposite way from every conditional one. Whenever a group-level rate is going to inform a decision, check it within the obvious confounders before believing it. This is also the honest limit of model interpretation — 24.4 returns to it.

4.7 Target Analysis

The target deserves its own pass, because three of its properties dictate the entire rest of the project.

1. Its balance decides the metric (8.5) and whether Chapter 22 applies.

2. Its relationship to each feature gives a first, cheap ranking of what might matter. For a binary target, the ROC AUC of each feature used alone is a good single number: 0.5 means no information, and distance from 0.5 in either direction means signal.

tenure_months at 0.24 is as informative as a feature at 0.76 would be — the direction is simply inverted (longer tenure, less churn). Ranking by distance from 0.5 rather than by the raw AUC is what makes the comparison honest.

3. Whether anything predicts it too well. A single feature with an AUC near 1.0 is almost never good news; it is 1.8.1's leakage arriving as a number.

An AUC of exactly 1.000 from a single column means the column is the answer, recorded after the fact. The rule of thumb worth carrying: treat any single feature above about 0.95 AUC as guilty until proven innocent, and ask the 1.8.1 question — would this value exist, in this form, at the moment the prediction is needed? For cancellation_reason the answer is no, and it must be dropped before modelling. 22.7 is the full taxonomy.

4.8 What EDA Is Really For

EDA is not a phase that produces a deliverable of charts. It is the process of forming and killing hypotheses about the data, and it is finished when you can answer these:

When to stop. When further looking stops changing what you would do next. EDA has strongly diminishing returns, and the failure mode of enjoying it is a folder of 60 charts and no decisions.

4.8.1 The uncomfortable part: EDA is itself a mild form of leakage

Every choice you make after looking at the whole dataset — which features to keep, which outliers to remove, which transform to apply — was informed by the test set. The test score afterwards is therefore slightly optimistic. This is not hypothetical; it is why 5.1 insists the split comes first.

In strict practice: split first, explore the training set only, and touch the test set once. The demonstrations above deliberately used the whole file, because their point was to show what the techniques reveal; the worked case in 4.9 does it the right way round so the correct habit is on the page too.

The middle row is what most real projects do, and it is defensible as long as it is disclosed. The bottom row is how a model that scored 0.94 in a notebook scores 0.71 in production.

4.9 In Practice

4.9.1 A worked case: a 30-minute data-quality review

The request: "Here's the customer export. Can we predict churn from it?" The deliverable is not a model. It is a short, written answer to three questions — is the data usable, what has to be fixed, and what should the first model use? — produced in under an hour, before anyone commits to a project.

Step 0 — split first (4.8.1). Everything below looks at training data only.

Step 1 — a data-quality table. One row per column, mechanically generated, is worth more than any number of individual checks because nothing gets skipped.

Three problems are now visible as table rows rather than as things someone had to think to check:

• total_charges is str — it should be numeric (blank entries, 3.3.1).
• satisfaction_score is 9.8% missing, age 5.6%, gender 4.6% — all need a 5.2 decision.
• cancellation_reason is 85.9% missing, and 4.7 already established it is leakage.

Step 2 — impossible values. Distinguish "extreme but real" from "cannot be true", because they get different treatment (5.3).

Step 3 — a signal shortlist. Rank features by univariate association with the target, so the first model starts from evidence rather than intuition. Mutual information is the right tool here because, unlike correlation, it catches non-monotonic relationships (2.8.2).

Two things to read here. The ranking agrees with 4.7's AUC ordering on the top three, which is reassurance rather than coincidence. But last_login_days scores an estimated MI of exactly 0.0000 while its AUC was 0.539 — mutual information is estimated from nearest-neighbour distances, and a genuinely weak relationship rounds to zero. Two weak-signal measures disagreeing is a signal in itself: the feature is marginal, and its fate should be decided by a wrapper method on the actual model (7.3) rather than by either univariate score.

Step 4 — is it worth doing at all? 1.6's question, answered before the project is staffed. A quick, deliberately unpolished model against the dummy baseline.

Step 5 — the written deliverable. Six lines, and it is the actual output of the exercise:

Is the data usable? Yes, with fixes. 1000 unique customers after removing 12 duplicate rows; churn rate 14.1%, giving roughly a 1:6 imbalance.

What must be fixed first? (1) total_charges is stored as text because blank entries exist — coerce to numeric. (2) age contains impossible values (0, 1, 150, 199) — blank them, do not drop the rows. (3) gender has six spellings of two values. (4) Missingness of 4.5% (gender), 5.8% (age) and 9.2% (satisfaction_score) needs an imputation decision.

What must be excluded? customer_id (identifier) and cancellation_reason (target leakage — AUC 1.000 on its own, populated only after a customer has left).

What should the first model use? tenure_months, support_tickets and satisfaction_score, which carry almost all the univariate signal, plus monthly_charges. Note that support_tickets and satisfaction_score correlate at −0.70 and largely duplicate each other, and that monthly_charges is essentially a proxy for plan. last_login_days and age look marginal.

Is it worth doing? Yes. A five-minute forest on six raw features, with rows containing missing values simply dropped, reaches 0.79 test AUC against a 0.47 baseline — before any cleaning, encoding, feature engineering or tuning.

That last line is what secures the project, and it took under an hour. Note also what it is not: 0.79 AUC is a floor, not a result. It was produced by dropping rows rather than imputing (5.2), ignoring every categorical column (6.2), and accepting default hyperparameters (23.5).

4.9.2 Which plot answers which question

Two rules that eliminate most misleading charts: plot rates, not counts, when comparing groups, and always show the group size next to the rate (4.5).

4.9.3 What goes wrong in production

The mitigation for the first five is the same and is worth stating plainly: the quality report in step 1 is not a one-off. Run it on every batch of incoming data, store the output, and compare it over time. A column whose missingness jumps from 5% to 60%, or whose mean moves three standard deviations, is a production incident that no model metric will surface until the damage is done. 24.9 turns this into monitoring.