25.1 Chapter Coverage Tracker
What is written versus what is still an empty heading.
| Chapter | Status | Notes |
|---|---|---|
| 1. Machine Learning Fundamentals | [DONE] |
1.1 (Mitchell's T/P/E definition, a rule recovered from data alone, generalization as the defining problem), 1.2 (rule-based vs. learned comparison table, a keyword spam rule failing in both directions, the common mistake of modelling a known rule), 1.3 (the five learning types keyed on what the training data contains; 1.3.1 regression vs. classification shown as a difference in y alone, 1.3.2 clustering with no y, 1.3.3 a self-training demo where 80 unlabelled rows beat 20 labelled ones alone, 1.3.4 reinforcement learning defined and scoped out), 1.4 (the full synonym table, X/y shape convention, and the parameter-vs-hyperparameter distinction demonstrated via scikit-learn's trailing-underscore convention), 1.5 (the eight-stage workflow diagram, a failure-mode table keyed to chapters, the entire workflow in fifteen lines, and the real time split), 1.6 (the seven disqualifying conditions, and a 30-feature/28-row model scoring a perfect 1.000 on train and -38.3 on test against a dummy's -0.05), 1.7 (classical-vs-deep comparison table, why tabular still favours boosting, an explicit exclusion list, and what transfers anyway), 1.8 (In Practice — framing "fix churn" into instance/T/E/P on data/customers.csv, the 86%-accuracy/0%-recall do-nothing baseline, cancellation_reason caught as target leakage, a framing-choice decision table including the unfixable corrupt-labels row, and the six post-deployment failure modes) — all filled |
| 2. Mathematical Foundations | [DONE] |
2.1 (vectors, the dot product as a linear model's prediction, its geometric/angle reading, the (3,)-vs-(3,1) shape trap), 2.2 (matrices, X @ w as every prediction at once, the *-vs-@ mistake), 2.3 (L1/L2/Linf norms, distance as the norm of a difference, cosine similarity on documents of different lengths), 2.4 (linear systems, rank vs. singularity, the condition number, inv vs. solve vs. pinv, and 2.4.2 ridge as a numerical fix), 2.5 (eigenvectors, 2.5.1 covariance eigendecomposition as PCA with 97.9% variance in one direction, 2.5.2 the SVD and an eig-vs-svd table, 2.5.3 eigenvector sign non-determinism), 2.6 (derivatives, the numerical-check habit, gradients, 2.6.2 the MSE gradient verified numerically and the dropped-2/n mistake, 2.6.3 the chain rule), 2.7 (expectation/variance, the six distributions that come up, Bayes' theorem with a 99%-sensitive test that is right under 2% of the time), 2.8 (descriptive stats and robustness, covariance vs. correlation, Pearson's blind spot on a U-shape shown with a figure, the CLT demonstrated on an exponential population, standard error, p-values and the 20-tests-of-nothing multiple-comparisons demo), 2.9 (In Practice — a duplicated column giving cond(X)=5.3e9 where rank says 4; three solvers returning three different answers; a proof of non-identifiability by shifting coefficients 2000 apart for identical predictions; the drop-vs-ridge fixes; a when-you-need-the-maths table; and six silent numerical failures including probability underflow to exactly 0.0) — all filled |
| 3. The Python Machine Learning Stack | [DONE] |
3.1 (ndarray attributes and constructors, the integer-dtype truncation trap, indexing, the slice-is-a-view demo, broadcasting rules with a table and the failure case, vectorization timed at 47x), 3.2 (the axis-is-what-collapses rule, keepdims, default_rng over legacy global state), 3.3 (Series/DataFrame, pandas 3's str dtype, and the four-call triage that exposes total_charges stored as text with 9 blanks), 3.4 (loc vs iloc and the inclusive-endpoint trap, boolean filtering and .query, .assign chains, .apply vs vectorized timed at 25x, .map as ordinal encoding, .isna and why == np.nan fails), 3.5 (groupby/agg, merge types, a non-unique key silently adding 461 rows, pivot_table exposing six spellings of two genders), 3.6 (the Figure/Axes model and the four exploratory plots, then seaborn's hue=), 3.7 (the estimator API contract, transformer/predictor/meta-estimator, fit-on-train-only shown via a test mean that is correctly NOT zero, 3.7.1 the trailing-underscore convention, 3.7.2 four models swapped in one loop, 3.7.3 a first Pipeline), 3.8 (random_state int vs RandomState instance, the ValueError from passing a modern Generator, and five sources of non-determinism a seed does not fix), 3.9 (In Practice — a churn segment report built correctly: dedup, gender canonicalization, impossible ages blanked rather than dropped, revenue-at-risk as the decision column, a chi-square test returning p=0.82 that makes the honest finding a negative one; a which-tool table; and index misalignment silently producing NaN) — all filled |
| 4. Data and Exploratory Data Analysis | [DONE] |
4.1 (the variable-type table driving every downstream choice, a cardinality heuristic and the four places it is wrong, the encoded-category-as-number mistake), 4.2 (bundled loaders, the make_* generators and why knowing the truth matters), 4.3 (numeric and categorical describe, a four-check summary reading, 4.3.1 numeric sentinel values), 4.4 (a distribution-shape table, six histograms revealing multimodal monthly_charges, skew/kurtosis as bin-width-independent outlier alarms - age kurtosis 28.9 caused by four rows), 4.5 (the pair-of-types tool table, plan explaining the multimodality, and city churn rates whose confidence intervals all overlap), 4.6 (correlation heatmap read for signal and redundancy, a programmatic top-pairs extraction, a pairplot, and 4.6.1 Simpson's paradox where B wins in both departments and loses overall), 4.7 (target balance, per-feature AUC ranked by distance from 0.5, and cancellation_reason scoring exactly 1.000 as leakage), 4.8 (what EDA answers and when to stop, 4.8.1 EDA as mild leakage with a strict/pragmatic/wrong rigour table), 4.9 (In Practice — a 30-minute data-quality review done split-first: a mechanical quality table, domain-rule violations, a mutual-information shortlist where MI and AUC disagree, a 0.786-vs-0.468 go/no-go check, and a six-line written deliverable) — all filled |
| 5. Data Preprocessing | [DONE] |
5.1 (the split first; stratification shown to hold the test churn rate at 0.140 where plain splits range 0.085-0.205; scaling-before-splitting as the commonest leak; a fit-on-train table), 5.2 (5.2.1 MCAR/MAR/MNAR with a 1.9-standard-error test on whether missingness relates to churn, 5.2.2 deletion costing 20% of rows, 5.2.3 the imputer catalogue and the repair-before-impute ordering, 5.2.4 missingness indicators and the lopsided economics of adding them), 5.3 (impossible vs. extreme-but-real vs. influential; detection methods; the exact bound |
| 6. Feature Engineering | [DONE] |
6.1 (feature engineering as making the implicit reachable, XOR defeating logistic regression at 0.463 AUC and solved perfectly by one product column, with a figure and a model-family sensitivity table), 6.2 (the encoder catalogue; 6.2.1 one-hot, handle_unknown as the setting that decides production survival, and the dummy-variable trap shown via matrix rank; 6.2.2 ordinal encoding and the alphabetical-default trap; 6.2.3 target encoding leaking 0.894 AUC out of a random column, fixed to 0.522 by cross-fitted TargetEncoder), 6.3 (binning strategies, manual tenure bands showing 49% to 3% churn, and binning costing AUC when the relationship was already monotonic), 6.4 (interactions, the O(p^2)/O(p^3) explosion table, polynomial features taking R^2 from 0.603 to 0.968), 6.5 (datetime decomposition, the now() mistake, 6.5.1 cyclical sin/cos encoding with a figure), 6.6 (CountVectorizer vs TF-IDF read honestly including the document where a stop word still wins, n-grams), 6.7 (the four domain-feature patterns, measured and found not to help, plus spend_vs_plan as a hidden group-statistic leak), 6.8 (the is-it-learned test as a table), 6.9 (In Practice — a five-rung feature ladder in which every rung makes the model worse, 0.814 to 0.781, reported as the real finding and used to motivate Chapter 7) — all filled |
| 7. Feature Selection and Dimensionality Reduction | [DONE] |
7.1 (the curse of dimensionality shown three ways: nearest/farthest contrast collapsing from 615 to 0.12, exponential sample requirements, and pure-noise columns dragging AUC from 0.906 to 0.745), 7.2 (filter catalogue; 7.2.1 variance thresholds broken by units; 7.2.2 mutual information recovering all six real features where ANOVA F admits two noise columns and drops two informative ones; chi2 requiring non-negative input; 7.2.3 the univariate blind spot, where two features that determine the target exactly both score below pure noise), 7.3 (wrappers - RFE, RFECV with a figure, sequential selection solving the interaction case - and selection outside the CV loop manufacturing 0.793 AUC from a coin-flip target), 7.4 (embedded methods, L1 producing exact zeros, SelectFromModel), 7.5 (PCA from scratch via eigendecomposition matched against sklearn, scree and cumulative-variance plots, n_components as a float, an adversarial case where PC1 captures 99.95% of variance and discards the only signal, 7.5.3 scaling as mandatory with proline taking 99.8% of an unscaled PC1), 7.6 (LDA vs PCA compared in a table and a figure, the n_classes-1 component limit, and LDA recovering the signal PCA threw away), 7.7 (t-SNE, its missing transform() demonstrated, and three perplexities giving three pictures), 7.8 (selection vs extraction decision table, and forests degrading far more gently than linear models under noise), 7.9 (In Practice — repairing Chapter 6's ladder: 0.7804 with all 26 encoded columns, 0.8104 with mutual-information selection, best k=4 by grid search, and 0.8347 on the held-out test set using 4 of 26 columns) — all filled |
| 8. Model Evaluation and Validation | [DONE] |
8.1 (training score as memorization; a tree at 1.0000 train, and why a LOW training score is the only informative one), 8.2 (train/validation/test, and 40 worthless candidate models on a coin-flip target yielding a best score of 0.565 from selection alone), 8.3 (the splitter catalogue; 8.3.1 stratification; 8.3.2 cross_validate with train scores; 8.3.3 group leakage producing 0.86 AUC on data with no signal, against GroupKFold's honest 0.47; 8.3.4 temporal leakage with a fold-structure figure), 8.4 (regression metrics; 8.4.1 MAE vs RMSE disagreeing tenfold about which error pattern is worse; 8.4.2 negative R^2 and adjusted R^2 falling as 60 noise columns are added; 8.4.3 MAPE's discontinuity at zero), 8.5 (the confusion matrix and every scalar derived from it; 8.5.1 accuracy 0.967 vs MCC 0.000 on a 1% problem; 8.5.2 ROC vs PR curves side by side under imbalance; 8.5.3 log loss punishing confident errors; 8.5.4 micro/macro/weighted averaging), 8.6 (the threshold as a business lever - one model read seven ways, a cost-optimal threshold of 0.14 saving 43%, Youden/F1/fixed-precision rules, and TunedThresholdClassifierCV to avoid tuning on the test set), 8.7 (calibration curves for four model families, the sigmoid-vs-isotonic table, FrozenEstimator replacing cv="prefit", sigmoid leaving AUC exactly unchanged while isotonic's ties move it), 8.8 (dummy baselines; a paired t-test over 30 repeated folds; and the winner's CV score being optimistic against its test score), 8.9 (In Practice — an evaluation report whose headline finding is that the campaign LOSES ₹11,000 at the default threshold, with break-even precision 0.417, a profitable operating point at 0.90, an acceptance-rate sensitivity table, and a caveat that class_weight="balanced" inflated the probabilities every expected value was computed from) — all filled |
| 9. Bias, Variance and Regularization | [DONE] |
9.1 (the under/overfitting table, polynomial degrees 1/3/20 with a figure, and the mistake of treating the train-validation gap rather than the validation score as the criterion), 9.2 (the bias-variance decomposition simulated over 200 training sets, with the scatter of 40 fits at degree 1 and degree 12 drawn side by side, and a model-family table), 9.3 (capacity knobs per family, C as an inverse penalty vs alpha as a direct one, and capacity being relative to n), 9.4 (learning curves and the four shapes, showing when more data cannot help), 9.5 (validation curves - where the literal argmin picks degree 14 on cubic data and the one-standard-error rule recovers degree 3), 9.6 (regularization as loss + penalty, a degree-15 fit with max |
| 10. Optimization and Gradient Descent | [DONE] |
10.1 (model + loss + optimizer, and why the loss surface can be drawn for one parameter and never for a thousand), 10.2 (10.2.1 squared/absolute/Huber compared in a figure and by one outlier dragging OLS from 2.0 to 2.52; 10.2.2 the three classification surrogates plotted against the undifferentiable 0-1 loss), 10.3 (convexity, and three starting points converging to one answer or two), 10.4 (batch gradient descent from scratch, matched against the normal equation and scikit-learn), 10.5 (batch vs SGD vs mini-batch, with SGD's non-monotonic loss drawn up close), 10.6 (the learning rate: too small crawls, 1.05 diverges to 3e+40; 10.6.1 unscaled features needing a rate a million times smaller, tied to the condition number), 10.7 (momentum on an ill-conditioned problem that plain GD never catches, plus the optimizer catalogue and lbfgs as sklearn's actual default), 10.8 (closed form vs iterative timed, with the honest note that memory rather than speed is the real argument; 10.8.1 convergence diagnostics, and a fit still warning after 5000 iterations at AUC 0.905 that 12 scaled iterations beat), 10.9 (In Practice — a five-step diagnostic path for a model that will not train, resolving 'slow and inaccurate' into one cause: at sklearn's default max_iter=100 the unscaled fit caps out at 0.8201, scaled it converges in 13 iterations to 0.8248) — all filled |
| 11. Linear Regression | [DONE] |
11.1 (the model; 11.1.1 'linear in the parameters' with a six-row table of what does and does not qualify), 11.2 (the least-squares objective, why squared error is the Gaussian MLE, 11.2.1 absorbing the intercept, 11.2.2 the normal equation derived and its gradient verified numerically), 11.3 (the normal equation from scratch matching sklearn to 10 decimals; 11.3.1 multiple features; 11.3.2 the two ways it breaks - an exact duplicate raises LinAlgError, while p>n silently returns one of infinitely many exact interpolations of noise), 11.4 (gradient descent converging to the same answer to 1e-15, plus SGDRegressor), 11.5 (the sklearn API and fit_intercept=False turning a true slope of 2.0 into 6.8), 11.6 (the five assumptions with a table separating what breaks prediction from what breaks inference; residual panels for the satisfied case, non-linearity and heteroscedasticity; Breusch-Pagan; HC3 robust errors; and the precise point that non-normality costs precision, not correctness), 11.7 (VIF from scratch matched against statsmodels, collinearity leaving R^2 unchanged while inflating a coefficient's bootstrap spread 9x), 11.8 (polynomial regression, degree by CV, and a degree-8 fit predicting -7072 where the truth is 46), 11.9 (the five clauses of an honest coefficient reading, 11.9.1 raw-vs-standardized at 376x vs 3.1x, 11.9.2 statsmodels intervals and a sign flip under collinearity), 11.10 (In Practice - an inference report on the diabetes data: bmi/bp/sex safe to report, s1 and s5 significant but collinear, s1's bootstrap sd of 430 against bmi's 81, a borderline Breusch-Pagan p=0.052 disclosed rather than rounded, and 64 of 111 test rows already outside the training range) - all filled |
| 12. Regularized Linear Models | [DONE] |
12.1 (60 rows and 100 features where OLS fits perfectly and scores 0.12 on held-out data, lasso reaches 0.76, and ridge does no better than OLS - because the penalty must match the sparsity of the truth), 12.2 (ridge derived to (X'X + lambda I)^-1 X'y, implemented from scratch with centring so the intercept is unpenalized, matched to sklearn; 12.2.2 shrinkage that never reaches zero even at alpha=10000), 12.3 (lasso, no closed form, coordinate descent; 12.3.1 the soft-threshold operator drawn against ridge's proportional shrink; 12.3.2 coordinate descent from scratch agreeing with sklearn to 1e-6 including the exact zero pattern), 12.4 (the geometric argument, circle vs diamond, with ridge never producing a single zero at any alpha), 12.5 (elastic net; lasso picking an arbitrary member of a correlated triple - 21/24/55 across bootstrap resamples - where elastic net keeps all three 100/100 times), 12.6 (regularization paths, real features surviving longest, lasso_path), 12.7 (RidgeCV/LassoCV/ElasticNetCV on a log grid, nothing to choose between them when n>>p, and the one-standard-error rule), 12.8 (scaling as mandatory: the large-scale feature contributing 0.0000 of the penalty, no single alpha regularizing both features, and scaling outside the CV loop leaking), 12.9 (In Practice - ridge cutting s1's bootstrap sd 23x on the diabetes data, stability selection ranking bp/bmi/s5/s3/sex above 99%, and the honest finding that regularization fixed a stability problem rather than an accuracy one) - all filled |
| 13. Logistic Regression | [DONE] |
13.1 (linear regression on a 0/1 target failing twice - unbounded output, and a distant point rotating the boundary from 3.25 to 3.94 hours where logistic regression does not move), 13.2 (the sigmoid and its derivative verified numerically; 13.2.1 odds, log-odds, and the sentence that makes the model interpretable), 13.3 (Bernoulli likelihood -> cross-entropy derived and matched to sklearn's log_loss; 13.3.1 the gradient X'(sigma(Xw) - y)/n verified numerically, and why there is no closed form), 13.4 (gradient descent from scratch agreeing with sklearn to 3 decimals and 2.85e-04 in predicted probability), 13.5 (the API, the penalty= deprecation in sklearn 1.8 with its C=np.inf / l1_ratio replacement table, and the solver-penalty compatibility matrix), 13.6 (the boundary is a hyperplane; degree-3 features lifting CV accuracy on moons from 0.843 to 0.915), 13.7 (OvR vs softmax, the z - z.max() stability trick shown overflowing without it, and softmax reducing to the sigmoid at K=2), 13.8 (13.8.1 C as an inverse penalty and L1 selecting; 13.8.2 odds ratios, and the same odds ratio of 2.0 moving probability by +0.010 at p=0.01 and +0.167 at p=0.50), 13.9 (In Practice - an interpretable churn model: calibrated to within 0.0002 of the base rate, coefficients as odds ratios with bootstrap intervals marking which are conclusive, perfect separation driving an unpenalized coefficient to 15.2 and probabilities from 0.0007 to 0.9996 across one unit, and a closing conversion from odds ratios to probabilities at the real base rate) - all filled |
| 14. k-Nearest Neighbors and Distance Metrics | [DONE] |
14.1 (lazy vs eager learning, with k-NN fitting in 2 ms and predicting 2000 rows in 443 ms where logistic regression is the reverse), 14.2 (the metric catalogue, Minkowski unifying L1/L2/Linf verified against scipy, 14.2.1 Euclidean and Manhattan disagreeing about which point is nearest, and cosine calling a 50x-longer document identical), 14.3 (kNN from scratch matching sklearn exactly; 14.3.1 three implementations timed, where broadcasting gives only 1.1x and the matmul expansion gives 4.9x - rewriting a loop as a broadcast is not automatically an optimization), 14.4 (kNN regression, step-function predictions, and flat extrapolation contrasted with 11.8's explosion), 14.5 (the API, distance weighting, and probabilities that take only 6 distinct values at k=5), 14.6 (k inverted relative to other hyperparameters, effective parameters ~ n/k, k=1 scoring a guaranteed 1.0 on training, and even-k ties), 14.7 (14.7.1 unscaled distance where age contributes 0.000016% of the total; 14.7.2 k-NN degrading from 0.943 to 0.532 under noise features where a forest holds 0.852), 14.8 (brute/KD-tree/ball-tree timed, the ball tree LOSING to brute force at 8 dimensions, and KD-tree speedup falling from 2.41x to 0.03x as dimensions grow), 14.9 (In Practice - a 'customers like this one' retrieval tool, where one-hot city and plan contribute 34% of the distance among actual nearest neighbours against 20% among random pairs, and a single mismatched city contributes 2.0 against a typical near-neighbour distance of 2.36) - all filled |
| 15. Naive Bayes | [DONE] |
15.1 (Bayes as a classifier, the evidence term dropped, and a prior turning an 8:1 likelihood ratio into a 77% posterior), 15.2 (the independence assumption and the parameter count it saves; 15.2.1 a duplicated feature leaving accuracy unchanged while inflating confidence), 15.3 (Gaussian NB from scratch matching sklearn's means, variances and priors exactly, with the fitted model drawn as a grid of per-class bell curves), 15.4 (the family table, and MultinomialNB refusing standardized input), 15.5 (the zero-frequency problem making a spam likelihood exactly 0, Laplace smoothing, and var_smoothing rescuing predict_proba from nan), 15.6 (log space, a product underflowing to 0.0 after 229 factors, and the log-sum-exp trick), 15.7 (text classification on a deliberately hard corpus, the model recovering the generating vocabulary at ~61% accuracy, and MultinomialNB scoring F1 exactly 0.000 on imbalanced data by predicting one positive in the whole test set where ComplementNB predicts 66), 15.8 (speed; 15.8.2 measuring the 'needs less data' folklore and finding regularized logistic regression wins below 100 rows even when independence holds exactly; 15.8.3 94.3% of predictions beyond 1%/99%), 15.9 (In Practice - support-ticket triage where ComplementNB lifts macro F1 from 0.454 to 0.605, the raw confidence turns out UNDER-confident rather than over-confident, and calibration makes claimed and delivered accuracy agree bin by bin) - all filled |
| 16. Support Vector Machines | [DONE] |
16.1 (three boundaries all perfect on training data, and refitting on 3 of 10 points giving identical coefficients), 16.2 (the hard-margin QP derived, margin = 2/ |
| 17. Decision Trees | [DONE] |
17.1 (recursive partitioning and its four consequences; 17.1.1 the axis-aligned limitation needing 18 leaves for a diagonal a line handles), 17.2 (gini/entropy/misclassification, the curvature argument, and a split with a PERFECTLY pure child that misclassification error scores at exactly 0 gain; 17.2.1 exhaustive split search matching sklearn), 17.3 (CART from scratch in ~30 lines, identical at depths 1-2 and 98.2% agreement beyond - the residual divergence being tie-breaking, not error), 17.4 (the API, unrestricted defaults memorizing, and random_state mattering only when ties exist - shown stable on wine and random across 12 fits once features are duplicated), 17.5 (export_text, plot_tree, reading samples, and 4 of 8 leaves backed by a single training row), 17.6 (pre-pruning knobs compared; 17.6.2 cost-complexity pruning, with the test-set-optimal alpha giving 0.9357 against the honest CV-chosen alpha's 0.9240), 17.7 (regression trees, step functions, and every input beyond the training range receiving the same constant), 17.8 (feature importance, correlated features splitting credit 1.0/0.0; 17.8.1 the cardinality bias giving pure-noise continuous features 0.77 importance, corrected by permutation importance), 17.9 (instability: depth 7-12 and 22-34 leaves across resamples, 33% of test points contested, and majority vote beating a single tree by 0.055 - which is Chapter 18), 17.10 (In Practice - churn rules for a retention playbook, with leaf-level coverage and churn rates validated on held-out data, and the root split changing feature in 4 of 15 retrains with thresholds from 2.5 to 30.5 months) - all filled |
| 18. Ensembles I: Bagging and Random Forests | [DONE] |
18.1 (the variance-of-an-average formula rhos^2 + (1-rho)/ks^2 verified by simulation, with the rho*s^2 floor that no number of trees can cross; 18.1.2 the Condorcet jury theorem and its failure mode below p=0.5; 50 identical trees giving 1 distinct prediction vector against 50), 18.2 (hard vs soft voting, and a pairwise agreement matrix showing all four members agree on only 60% of rows), 18.3 (the bootstrap, (1-1/n)^n -> 1/e verified empirically), 18.4 (bagging from scratch matching sklearn's accuracy exactly while agreeing on 93% of rows - the right kind of agreement for a stochastic algorithm - and n_estimators as a budget not a complexity parameter), 18.5 (out-of-bag scoring from scratch, OOB 0.807 against test 0.846 and a 5-fold CV that costs five fits, plus sklearn's UserWarning firing at 5 and 10 trees), 18.6 (random forests decorrelating: pairwise correlation 0.352 -> 0.298, root feature used 97% of the time by bagging against 28% by a forest; 18.6.1 max_features driving correlation from 0.18 to 0.36 while single-tree accuracy rises and ensemble accuracy peaks in between), 18.7 (Extra Trees, half the fit time, neither dominating across three datasets), 18.8 (impurity importance still tracking cardinality with 300 trees; 18.8.1 permutation importance collapsing 8x under a duplicated feature, asymmetrically and arbitrarily), 18.9 (In Practice - tuning by OOB: the trap that oob_score=roc_auc_score returns 0.579 because the callable receives hard labels, the fix via oob_decision_function_ giving 0.728 against CV's 0.745, a 3.2x faster search selecting the identical configuration CV chose, and +0.04 test AUC from min_samples_leaf alone) - all filled |
| 19. Ensembles II: Boosting and Stacking | [DONE] |
19.1 (bagging vs boosting by base learner: bagging a stump makes it slightly WORSE while boosting lifts it 0.08; boosting a deep tree makes it worse while bagging lifts it 0.07), 19.2 (AdaBoost from scratch matching sklearn on 100% of predictions with identical alphas; the alpha formula including the inverted vote below 0.5; 19.2.1 weight trajectories drawn on a log scale, corrupted rows holding 1.7x their fair share, and the honest finding that at 10% noise the accuracy cost is small - the gap opens at 20%, shown in 19.8), 19.3 (gradient boosting from scratch matching GradientBoostingRegressor EXACTLY - correlation 1.000000 - with the ensemble drawn at 1, 5, 20 and 100 trees and tree 6 shown fitting the residuals of the first five; 19.3.2 the learning-rate/n_estimators trade), 19.4 (GradientBoosting vs HistGradientBoosting, 11x faster and handling raw NaNs where the other raises), 19.5 (XGBoost, LightGBM and CatBoost within 0.002 AUC of each other and of sklearn - the honest headline - with native categorical handling beating one-hot on a 60-level feature), 19.6 (early stopping choosing 384 of 2000 rounds, and randomized search beating defaults by 0.001), 19.7 (stacking, the meta-learner's coefficients as trust weights, and a naive in-sample stack handing weight 9.6 to a tree whose out-of-fold AUC is 0.82 where the correct version gives it 2.6), 19.8 (the comparison table, with the forest winning under 20% label noise), 19.9 (In Practice - the nine-model churn scoreboard where BOOSTING COMES SIXTH at 0.709 against logistic regression's 0.795, paired tests confirming that gap at p=0.0001 while the top two rows are a tie at p=0.12, and n_estimators driving test AUC from 0.800 down to 0.706 while a forest rises) - all filled |
| 20. Clustering | [DONE] |
20.1 (clustering never refuses - k-means partitioning uniform noise into 3 and 6 confident, meaningless groups), 20.2 (Lloyd's algorithm from scratch matching sklearn's inertia to 1148.157315 exactly with an identical partition, while the raw label arrays agree on only 50% - the case for ARI made by the contract itself; 20.2.1 k-means++ raising the minimum initial centroid separation from 1.27 to 5.77), 20.3 (25 random starts finding 12 distinct local minima, the worst 3.41x the best; 20.3.2 inertia monotone to 0.00 at k=n, so only the elbow's shape carries information), 20.4 (four shapes k-means gets wrong for one reason, and scaling taking ARI from -0.00 to 1.00 where one noise feature is 500x the scale), 20.5 (four linkages, four trees; eight bridge points - 5% of the data - dropping single linkage to ARI 0.56 with sizes [108 49 1] while the other three stay at 1.00, then single linkage alone recovering the moons at 1.00 where k-means gets 0.25), 20.6 (DBSCAN finding the cluster count itself, and single linkage chaining across the same 40 noise points DBSCAN refuses; 20.6.1 eps controlling the cluster count anyway - 17, 2, 2, 1, 1), 20.7 (GMM 1.00 against k-means 0.67 on sheared blobs, soft assignment leaving 1 point in 6 undecided, BIC and AIC both picking the true k=4; 20.7.1 a collapsed component - min covariance eigenvalue sitting exactly on reg_covar - buying 373 nats of likelihood, with the one-line diagnostic), 20.8 (the silhouette preferring k-means 0.49 over DBSCAN 0.33 on moons where the truth says 1.00 vs 0.25; NMI inflating to 0.135 on random labels at k=100 where AMI stays at 0.016; 20.8.3 the Hopkins statistic, and the seed collision that silently drove it to 0.04), 20.9 (In Practice - segmentation with Hopkins only +0.037 over a column-shuffled null and silhouettes near 0.2 against 0.79 for real blobs, so IMPOSED segments not discovered ones; three criteria disagreeing on k with the GMM's winner a collapsed fit; stability under 80% resampling as the strongest available claim; external validation via a churn column excluded from the features, segment 2 churning at 2.5x; two refits with ARI 0.9943 and 24% raw label agreement, and naming segments by profile rank instead; MiniBatchKMeans at its DEFAULT batch_size 11x SLOWER than KMeans on 200k rows) - all filled |
| 21. Anomaly Detection and Association Rules | [DONE] |
21.1 (point/contextual/collective, and outlier vs novelty detection; a contextual anomaly at global z=1.44 and within-month z=4.35 - a modelling decision, not an algorithm choice), 21.2 (a point at z=2.10 and z=-2.27 on its two features with Mahalanobis d^2=125 against a threshold of 9.21; then masking - the contaminated covariance determinant 4.495 against the robust 0.222, catching 1 anomaly of 60 where MCD catches all 60), 21.3 (Isolation Forest from scratch, AUC 0.9765 against sklearn's 0.9794 with rank correlation 0.954 and 19 of the same top-20 rows; contamination leaving the AUC at 0.9794 for every value because it moves only the cut point), 21.4 (four detectors within 0.01 AUC; then LOF ranking a planted point 1st where Isolation Forest says 10th and global Mahalanobis says 201st of 401), 21.5 (ROC-AUC 0.8613 next to PR-AUC 0.0587 at a 0.6% positive rate, and precision@50 of 0.140 - the number that describes an analyst's day), 21.6 (the basket matrix at 16.4% density, and 2^20 itemsets from 20 items), 21.7 (the confidence trap: onions=>milk at confidence 0.333 and lift 0.946 against cereal=>yogurt at 0.356 and 1.962 - near-identical confidence, opposite verdicts; a rule with confidence 1.00, lift 1.01 and INFINITE conviction; and lift's arithmetic ceiling of 1/support), 21.8 (Apriori from scratch matching mlxtend's 43 itemsets and supports exactly, with the downward-closure pruning that skips most of 1,048,575; 21.8.1 FP-Growth returning identical itemsets; 21.8.2 the lift-sorted table at min_support=0.01 topped entirely by rules resting on 8 baskets of 800), 21.9 (In Practice - 78 candidate rules filtered to 16 by evidence, strength, redundancy and direction; uplift translating lift 3.0 into 768 incremental jam purchases per 10,000 shoppers, and rejecting bread=>milk at -248; then a frozen threshold taking the flag rate from 1.85% to 17.70% under population shift, with the median score as the diagnosis rather than the alarm) - all filled |
| 22. Imbalanced Data and Data Leakage | [DONE] |
22.1 (four sources of imbalance with different fixes, and the ratio vs the absolute positive count), 22.2 (the constant model at 0.9673 accuracy and 0.5000 balanced accuracy, with ROC-AUC barely moving across a 500-fold change in positive rate while PR-AUC tracks it), 22.3 (SMOTE from scratch matching imbalanced-learn's balance, with 100% of synthetic points provably inside the real minority bounding box - the structural fact AND the limitation; then 24.5% of SMOTE's synthetic minority points landing in the majority core when k_neighbors exceeds the cluster size), 22.4 (the weighted cross-entropy and 'balanced' weights derived, then the comparison table where DOING NOTHING WINS on PR-AUC at 0.178 against 0.102-0.115 for every resampler, all of which nudge ROC-AUC up - resampling improving the imbalance-insensitive metric and damaging the other), 22.5 (threshold tuning: the F1 optimum at 0.131 and the cost optimum at 0.034 under a 100:1 cost ratio, plus 29.8% optimism measured from tuning the threshold on test), 22.6 (resample-then-split PR-AUC 0.7027 against 0.1454 inside the fold - a +0.557 inflation, with the wrong number the better-looking one; imblearn's Pipeline as the structural fix), 22.7 (the five leakage forms, each reproduced: cancellation_reason giving ROC-AUC 1.0000 against 0.7517; StandardScaler contamination measured at EXACTLY 0.0000 - the honest result - with the same mistake in feature selection scoring 0.8106 on 3000 pure-noise features where the honest pipeline gets 0.5012; TimeSeriesSplit 0.6788 against a shuffled 0.7555; GroupKFold 0.5248 against a random 0.9380 on coin-flip group labels; duplicates 0.9886 against 0.9288), 22.8 (the nine-item audit, with the label-shuffle test at 0.4912 for an honest pipeline and 0.9928 for a leaky one on RANDOM labels), 22.9 (In Practice - a churn model built to survive review: the feature audit by when-is-it-known as domain work rather than statistics, the whole audit run on the model as built, and a cost-minimising threshold at 0.360 contacting 47% to catch 85.8% of churners at 25.7% precision; then SMOTE's mean predicted probability at 0.3227 against a true rate of 0.0742 - Brier 0.130 against 0.042 - with identical AUC, repaired by calibration) - all filled |
| 23. Pipelines and Hyperparameter Tuning | [DONE] |
23.1 (the manual-vs-pipeline leak measured at -0.0004 - the honest result, and the argument restated as 'one rule, no judgement call per transformer'), 23.2 (ColumnTransformer over six spellings of gender, handle_unknown='ignore' surviving an unseen city at predict time, and remainder='drop' vs 'passthrough' shown to change the output width), 23.3 (custom transformers and the init contract - a derived attribute breaking clone with AttributeError because get_params cannot read the parameter back; FeatureUnion taking 7 columns to 35), 23.4 (55 addressable hyperparameters, the step__param__subparam addressing read from the outside in), 23.5 (grid's multiplicative cost tabulated to 81,920 fits; the imputation strategy tuned alongside forest depth; randomized search over continuous ranges at the same budget), 23.6 (successive halving scoring WORST at 0.7200, with the schedule showing why - round 0 reports 0.8537 on 120 rows and 0.7200 on 360, so the elimination was on noise - and taking the same wall-clock as the grid rather than less), 23.7 (Optuna with the MODEL CHOICE in the search space, winning at 0.8106 with a logistic regression while the other three searched forests only - so the table measures the search SPACE, not the sampler, and the chapter says so), 23.8 (nested CV optimism of +0.0044 against a fold standard deviation of 0.0650 - below the noise floor, stated rather than overclaimed - then isolated on pure noise averaged over 6 independent datasets: mean 0.5011 with no selection, rising monotonically to 0.5526 at 24 configurations; 23.8.1 a paired t-test declining to separate logistic from forest at p=0.18), 23.9 (In Practice - one object from raw row to prediction, ratio features computed inside the fold, three model families searched together, a paired test at p=0.043, and a lambda inside a transformer failing to pickle where the full pipeline succeeds) - all filled |
| 24. Interpretability and Deployment | [DONE] |
24.1 (the interpretability/accuracy trade-off MEASURED rather than conceded: the depth-3 tree wins at 0.8216 against the 200-tree forest's 0.8019 and gradient boosting's 0.7281), 24.2 (the tree printed in full as its own explanation; odds ratios; and the unscaled-vs-scaled coefficient rankings disagreeing, only one of which answers 'which feature matters most'), 24.3 (permutation importance on train vs test, with satisfaction_score positive on train and NEGATIVE on test - its use does not generalise; and a duplicated column splitting tenure's importance from 0.0967 into 0.0281 + 0.0427), 24.4 (partial dependence and ICE, the causal caveat, and the extrapolation problem quantified via r=0.76 between tenure and total_charges), 24.5 (SHAP additivity verified to 1.5e-15; the base value of 0.50 explained as an artefact of class_weight='balanced' rather than the 14% churn rate; a single prediction decomposed to sum exactly to 0.898), 24.6 (LIME's instability MEASURED and found smaller than its reputation - the same four features every run, weights varying by at most 0.012 - so the honest claim is 'reproducible in substance, not in detail'; SHAP identical to the last digit), 24.7 (the incompatible fairness criteria; the unawareness fallacy TESTED with a probe at ROC-AUC 0.4809, meaning the attribute is genuinely not recoverable here; group metrics with bootstrap intervals that overlap, so no disparity is claimed despite a selection-rate ratio of 0.786), 24.8 (joblib with metadata, a load-time validator, the unpickling-executes-code warning and a sha256 for build provenance), 24.9 (input validation before predict; data vs concept drift, with a KS statistic of 0.50 on age moving predictions by 0.015 while an unalarmed shift in support_tickets moves them by 0.087 - drift monitors must be weighted by importance; concept drift with IDENTICAL inputs and predictions, invisible to every input monitor; 24.9.3 a feedback loop whose labelled sample leaves the population value at round 0 and never returns), 24.10 (In Practice - the model review as a ten-question checklist, each drawing on an earlier chapter, ending in a conditional sign-off) - all filled |
25.2 Cross-Reference Index
Pointers for concepts that are discussed in one chapter but looked for in another. The Documented in column is where the concept is actually developed; Also relevant to lists the places that depend on it or revisit it from a different angle.
| Concept | Documented in | Also relevant to |
|---|---|---|
| Bias-variance decomposition | 9.2 | 9.4 learning curves, 18.1 why averaging reduces variance, 24.1 |
| Bootstrap sampling | 18.3 | 18.5 OOB scoring, 24.7 fairness intervals, 20.9.1 cluster stability |
| Calibration of probabilities | 8.8 | 13.6 logistic outputs, 22.9.3 resampling destroys it, 24.9.2 |
| Categorical encoding | 6.2 | 19.5 native handling, 23.2 inside a ColumnTransformer, 15.4 for text |
| Class imbalance | 22.1-22.5 | 8.6 ROC vs PR, 17.6 class_weight, 21.5 detector evaluation |
| Class weights | 22.4 | 13.5, 16.3 SVM C, 17.6, 24.5 shifts the SHAP base value |
| ColumnTransformer | 23.2 | 6.1 mixed dtypes, 22.7.2 the structural leakage fix, 24.8 shipping it |
| Confusion matrix | 8.5 | 22.2 under imbalance, 22.9.1 at a cost-chosen threshold, 24.7 by group |
| Convexity | 10.3 | 13.3 cross-entropy, 16.2 the SVM dual, 20.3.1 k-means is NOT convex |
| Correlation vs causation | 4.5 | 11.8 coefficients, 21.9.1 association rules, 24.4 PDPs, 24.5 SHAP |
| Cost-sensitive decisions | 22.5 | 8.7 thresholds, 21.5 precision@k, 22.9.1, 24.10.1 |
| Cross-validation strategies | 8.3 | 22.6 resampling inside the fold, 22.7.3-22.7.4, 23.8 nested |
| Curse of dimensionality | 7.1 | 14.7 kNN, 20.4 clustering, 21.4 LOF and One-Class SVM |
| Custom transformers | 23.3 | 6.6 domain features, 5.3 domain rules, 24.8 the pickling hazard |
| Decision threshold | 8.7 | 13.6, 21.3 contamination, 22.5 under imbalance, 24.9.2 monitoring it |
| Dendrogram | 20.5 | 20.5.1 where to cut it, 7.5 as an alternative view of structure |
| Distance metrics | 14.2 | 20.4 k-means, 20.6 DBSCAN's eps, 21.2 Mahalanobis |
| Drift (data vs concept) | 24.9.1 | 8.3 TimeSeriesSplit, 20.9.3, 21.9.3, 22.9.3, 24.9.3 retraining |
| Duplicate rows | 5.4 | 22.7.5 as leakage, 20.7.1 GMM singularities, 17.9 tree instability |
| Elbow method | 20.3.2 | 7.5 PCA scree plot, 20.8 why it is a judgement call |
| Ensembling | 18.1 | 18.2 voting, 19.1 boosting vs bagging, 19.7 stacking |
| Entropy and information gain | 17.2 | 15.1 Bayes, 20.8.2 mutual information measures |
| Evaluation without ground truth | 20.8 | 21.5 detectors, 20.8.3 cluster tendency |
| Expectation-maximization | 20.7 | 5.2 IterativeImputer, 20.7.1 singular components |
| Feature importance | 17.8 | 18.8 impurity vs permutation, 22.7.1 as a leakage probe, 24.3, 24.5 |
| Feature scaling | 5.5 | 10.7 convergence, 12.1, 14.7.1, 16.7, 20.4, 24.2 reading coefficients |
| Feature selection | 7.2-7.4 | 12.3 L1 sparsity, 22.7.2 leaks badly if done outside the fold |
| Gradient descent | 10.4 | 11.4 linear regression, 13.4 logistic, 19.3 gradient boosting |
| Grid vs randomized search | 23.5 | 23.6 halving, 23.7 Optuna, 23.8 the optimism each one creates |
| Hopkins statistic | 20.8.3 | 20.9.1 against a shuffled null, 21.1 anomalies vs clusters |
| Hyperparameter vs parameter | 1.5 | 23.4 addressing them in a pipeline, 9.3 what tuning is for |
| Imputation | 5.2 | 5.2.3 the missingness indicator, 19.4 native NaN handling, 23.2, 23.5 |
| Interpretability | 24.1 | 11.8 coefficients, 17.5 reading a tree, 24.2, 24.5 SHAP, 24.6 LIME |
| Kernel trick | 16.4 | 7.5 kernel PCA, 21.4 One-Class SVM, 16.5 choosing a kernel |
| Leakage | 22.7 | 6.2 target encoding, 8.4, 20.9.3, 21.9.3, 23.1, 23.8, 24.9.3 feedback loops |
| Learning curves | 9.4 | 9.5 validation curves, 23.6 why halving eliminates on small samples |
| Lift (association rules) | 21.7 | 21.8.2 sorting by it, 21.9.1 translating it into units |
| Linear separability | 6.4 | 13.2 logistic boundaries, 16.1 margins, 20.4 k-means boundaries |
| Local minima | 10.3 | 20.3.1 k-means n_init, 20.7 EM initialization |
| Loss functions | 10.2 | 11.1 MSE, 13.3 cross-entropy, 16.3 hinge, 19.3 boosting's residuals |
| Mahalanobis distance | 21.2 | 2.3 covariance, 14.2 distance metrics, 21.2 masking |
| Metrics for classification | 8.5 | 8.6 ROC vs PR-AUC, 21.5 precision@k, 22.2 why accuracy lies |
| Metrics for regression | 8.4 | 11.6 R-squared and its adjusted form, 9.2 as a loss |
| Missing data mechanisms | 5.1 | 5.2 imputation strategies, 4.4 spotting them in EDA |
| Multicollinearity | 11.7 | 2.2 rank, 12.2 ridge as the fix, 24.3 splits permutation importance |
| Nested cross-validation | 23.8 | 8.2 selection optimism, 23.8.1 comparing models honestly |
| No free lunch | 9.7 | 19.8 bagging vs boosting, 21.9.2, 24.1 simple models sometimes win |
| Normal equation | 11.3 | 2.2 matrix inversion, 10.6 closed form vs iterative, 12.2 ridge's form |
| One-vs-rest and softmax | 13.8 | 8.5.1 multiclass averaging, 16.6 SVM multiclass |
| Out-of-bag scoring | 18.5 | 8.3 as a CV substitute, 18.9 the hard-label trap |
| Overfitting | 9.1 | 9.3 capacity, 12.1 regularization, 17.7 pruning, 19.6 early stopping |
| PCA | 7.5 | 2.4 eigendecomposition and SVD, 7.6 LDA, 7.7 t-SNE, 20.4 reduce before clustering |
| Paired significance tests | 19.9 | 23.8.1 comparing models, 8.2, 24.7 overlapping intervals |
| Partial dependence and ICE | 24.4 | 11.8 linear coefficients, 24.3 importance, 24.5 SHAP |
| Permutation importance | 18.8.1 | 24.3 on held-out data, 22.8 auditing, 24.9.1 weighting drift |
| Pipelines | 23.1-23.2 | 5.6, 6.7, 22.6 imblearn's variant, 22.7.2, 24.8 what gets shipped |
| Pruning | 17.7 | 9.3 capacity, 19.6 early stopping as its boosting analogue |
| ROC vs precision-recall | 8.6 | 21.5 detectors, 22.2 across a 500-fold imbalance range |
| Random forests | 18.6 | 18.7 extra trees, 19.8 against boosting, 24.3 explaining one |
| Regularization | 9.6 | 12.1-12.4 the linear-model versions, 13.5, 16.3, 20.7.1 reg_covar |
| Reproducibility | 3.5 | 0.2 seeding, 20.8.3 a seed collision that broke a statistic, 24.8 |
| Resampling for imbalance | 22.3 | 22.4 against class weights, 22.6 inside the fold, 22.9.3 |
| SHAP | 24.5 | 24.6 against LIME, 24.3 as a global measure, 24.10.1 in a review |
| SMOTE | 22.3 | 22.6 leakage when applied before the split, 22.9.3 destroys calibration |
| Scikit-learn estimator API | 3.4 | 23.3 the BaseEstimator contract, 23.4 get_params and clone |
| Selection optimism | 8.2 | 23.8 nested CV, 23.5 searching on the test set, 22.8 auditing |
| Silhouette score | 20.8.1 | 20.3.2 choosing k, 20.8 why it favours k-means, 20.9.1 per segment |
| Stacking | 19.7 | 18.2 voting, 19.9 the scoreboard, 8.3 out-of-fold predictions |
| Support vectors | 16.1 | 16.3 soft margin, 16.4 kernels, 21.4 One-Class SVM |
| Target encoding | 6.2.3 | 22.7.2 the worst leakage of any transformer, 23.1 |
| Temporal data | 6.5 | 8.3 TimeSeriesSplit, 22.7.3 temporal leakage, 24.9.1 drift |
| Text features | 6.6 | 15.4 multinomial Naive Bayes, 7.1 dimensionality |
| Tree construction (CART) | 17.3 | 17.2 impurity, 18.6 randomized variants, 19.3 as a weak learner |
| Underfitting | 9.1 | 9.4 diagnosing it from learning curves, 9.3 capacity |
| Validation curves | 9.5 | 14.4 choosing k, 17.6 depth, 23.5 as a one-dimensional search |
| Voronoi partition | 20.4 | 14.3 kNN decision regions, 20.2 what k-means can represent |
25.3 Notation
The symbols used consistently across the algorithm chapters.
| Symbol | Meaning |
|---|---|
| \(n\) | number of samples (rows) |
| \(p\) | number of features (columns) |
| \(X\) | feature matrix, shape \((n, p)\) |
| \(\mathbf{x}^{(i)}\) | the \(i\)-th sample — one row of \(X\) |
| \(x_j\) | the \(j\)-th feature — one column of \(X\) |
| \(y\) | target vector, shape \((n,)\); \(y^{(i)}\) is one sample's target |
| \(\hat{y}\) | predicted target |
| \(\mathbf{w}\), \(b\) | learned weights (coefficients) and intercept (bias) |
| \(\theta\) | all learned parameters of a model, taken together |
| \(\mathcal{L}\) | loss on a single sample |
| \(J(\theta)\) | cost — the loss averaged or summed over the whole training set |
| \(\alpha\) | learning rate (Chapter 10) or regularization strength (Chapter 12) — the two standard uses collide, so each chapter says which it means |
| \(\lambda\) | regularization strength in the mathematics; scikit-learn spells it alpha (12.1) |
| \(K\) | number of clusters (Chapter 20) or number of CV folds (8.3) |
| \(k\) | number of neighbors (Chapter 14) |
Vectors are column vectors in the mathematics and 1-D NumPy arrays in the code; the material flags the places where that distinction actually changes a result.
25.4 Environment Notes
The versions the committed outputs were produced with. Run the cell below to compare against your own environment — a difference here is the first thing to check if a committed output does not reproduce.
# --- Environment check ---
import sys, platform
import numpy, pandas, sklearn, scipy, matplotlib
print(sys.version)
print(platform.platform())
print()
for mod in (numpy, pandas, sklearn, scipy, matplotlib):
print(f"{mod.__name__:<12} {mod.__version__}")
3.13.2 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 18:49:14) [MSC v.1929 64 bit (AMD64)] Windows-11-10.0.26200-SP0 numpy 2.5.2 pandas 3.0.5 sklearn 1.9.0 scipy 1.18.1 matplotlib 3.11.1