Mathematical Foundations

Mathematics earns its place in this material only where it changes what you do. Every section below answers a question that comes up later: why a linear model is a dot product (2.1), why the normal equation sometimes explodes (2.4), what PCA is actually computing (2.5), where gradient descent's update rule comes from (2.6), and why Naive Bayes works in log space (2.7).

Nothing here is proved. Everything here is computed — the point is to make each idea concrete enough to recognize when it reappears wearing an algorithm's clothing. The notation table in §25.3 of the index is the reference for symbols.

2.1 Vectors and Vector Operations

A vector is an ordered list of numbers. In machine learning it is almost always one of two things:

Both live in the same \(p\)-dimensional space, which is what makes the geometry meaningful: a model's weights and a data point can be compared directly.

Addition and scalar multiplication work element-wise, and NumPy writes them exactly as the maths does.

2.1.1 The dot product

The dot product of two vectors multiplies them element-wise and sums the result:

\[\mathbf{w} \cdot \mathbf{x} = \sum_{j=1}^{p} w_j x_j\]

This single operation is the arithmetic core of most of Chapters 11–13. A linear model's prediction for one instance is a dot product plus an intercept:

\[\hat{y} = \mathbf{w} \cdot \mathbf{x} + b\]

Written as a loop it is obvious; written as @ it is the same thing, faster and closer to the notation.

2.1.2 What the dot product means geometrically

The dot product also equals

\[\mathbf{a} \cdot \mathbf{b} = \|\mathbf{a}\| \, \|\mathbf{b}\| \cos\theta\]

where \(\theta\) is the angle between the two vectors. That gives it a directional reading, which is what makes it useful far beyond arithmetic:

Orthogonality is the property PCA is built on (2.5): principal components are chosen to be mutually orthogonal, so each one carries information the others do not.

2.2 Matrices and Matrix Operations

A matrix is a rectangular grid of numbers. The feature matrix \(X\) is the one that matters: \(n\) rows (instances) by \(p\) columns (features), exactly the shape scikit-learn expects.

The second row is the one worth internalizing: X @ w computes every prediction at once. The loop over instances that a beginner writes is already inside the matrix product.

The shapes are the contract: X is (3, 3), w is (3,), and the product is (3,) — one prediction per row. The inner dimensions must match, and NumPy will refuse if they do not.

2.3 Norms, Distance and Similarity

A norm measures the size of a vector. Different norms measure "size" differently, and the choice is not cosmetic — it is what separates ridge from lasso (Chapter 12) and one k-NN metric from another (14.2).

np.linalg.norm computes all of them through its ord argument.

2.3.1 Distance is the norm of a difference

Every distance metric in Chapter 14 is a norm applied to the difference between two points:

\[d(\mathbf{a}, \mathbf{b}) = \|\mathbf{a} - \mathbf{b}\|\]

Change the norm and you change the metric — and, in k-NN, the neighbours it finds.

2.3.2 Cosine similarity

Distance cares about magnitude; cosine similarity cares only about direction. Rearranging the geometric dot-product identity from 2.1.2:

\[\text{cos-sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|} \in [-1, 1]\]

This matters wherever magnitude is an artefact rather than a signal. Two documents represented as word counts (6.6) can be about exactly the same thing while one is ten times longer; Euclidean distance calls them far apart, cosine similarity calls them identical.

2.4 Linear Systems, Rank and Invertibility

Fitting a linear model by the normal equation (11.3) means solving a linear system. Understanding when that system has no unique solution explains a failure you will otherwise meet as an inexplicable error message.

A system \(A\mathbf{z} = \mathbf{b}\) asks: which \(\mathbf{z}\), combined through \(A\), produces \(\mathbf{b}\)?

2.4.1 Rank and singular matrices

The rank of a matrix is the number of linearly independent columns it has — columns that carry genuinely new information rather than being reproducible from the others. A square matrix whose rank is less than its size is singular: it has no inverse, and the system has either no solution or infinitely many.

This is not an abstract edge case. A singular (or nearly singular) \(X^\top X\) is exactly what multicollinearity produces — two features that are duplicates, or one that is a sum of others (11.7).

A determinant of exactly zero is the clean case. The dangerous case is a determinant near zero — an ill-conditioned matrix, which is technically invertible but numerically untrustworthy: tiny changes in the data produce huge changes in the solution. The condition number quantifies it, and it is the single most useful diagnostic when a linear model returns absurd coefficients.

This is precisely why scikit-learn's LinearRegression uses lstsq internally rather than inverting \(X^\top X\) — it returns an answer on collinear data instead of crashing. That is a mixed blessing: the answer's individual coefficients are then unstable and should not be interpreted (11.7, 11.9).

2.4.2 Why ridge regression is a numerical fix as well as a statistical one

Ridge (12.2) adds \(\lambda I\) to \(X^\top X\) before solving. Statistically that shrinks the coefficients; numerically it adds a constant to the diagonal, which drags the condition number down and makes a singular matrix invertible again.

2.5 Eigendecomposition and the Singular Value Decomposition

An eigenvector of a square matrix \(A\) is a vector whose direction is unchanged when \(A\) is applied to it — only its length changes, by a factor called the eigenvalue:

\[A\mathbf{v} = \lambda \mathbf{v}\]

Most vectors get rotated by a matrix. Eigenvectors are the special directions that do not, which makes them the natural axes to describe what the matrix does.

2.5.1 Why this is PCA

Chapter 7.5 derives PCA properly; the mechanism is already visible here. The covariance matrix of a dataset (2.8.2) records how features vary together. Its eigenvectors are the directions along which the data varies, and each eigenvalue is how much variance lies along its eigenvector. Sorting eigenvalues largest-first and keeping the top few is the whole idea of PCA.

Almost 98% of the variance lies along a single direction, so projecting this two-dimensional data down to one dimension discards about 2% of it. That trade — fewer dimensions, with the information loss quantified rather than guessed at — is exactly what 7.5 formalizes.

2.5.2 The singular value decomposition

Eigendecomposition only applies to square matrices. The SVD applies to any matrix, which is why it is the workhorse underneath PCA, LinearRegression's solver, and truncated-SVD text methods:

\[X = U \Sigma V^\top\]

NumPy returns \(\Sigma\) as a 1-D array of singular values rather than a matrix.

Keeping only the largest singular values gives the best possible low-rank approximation of the matrix — the formal statement is the Eckart–Young theorem, and it is the reason "keep the top \(k\) components" is a principled operation rather than a heuristic.

The stability row is why scikit-learn's PCA uses the SVD of the centred data rather than eigendecomposing the covariance matrix, even though the two are mathematically equivalent.

2.5.3 Determinism warning: eigenvector signs

An eigenvector multiplied by \(-1\) is still an eigenvector — the definition constrains direction, not sign. Different library versions, and even different runs on different hardware, can legitimately return different signs. Anything downstream that depends on the sign (a scatter plot's orientation, the sign of a PCA loading) can flip without anything being wrong. sklearn.decomposition.PCA applies a deterministic sign convention for this reason; hand-rolled eigendecompositions do not.

2.6 Derivatives, Gradients and the Chain Rule

Training a model means minimizing a loss. Minimizing means moving downhill. Derivatives are how a program knows which way is downhill.

The derivative \(f'(x)\) is the slope of \(f\) at \(x\) — how much the output changes per unit change in the input. Its sign is the operationally important part: positive means the function is rising, so decreasing \(x\) decreases \(f\).

Whenever a derivative is derived by hand there is a cheap way to check it: compare against a numerical approximation of the same slope, \(\frac{f(x+h) - f(x-h)}{2h}\) for small \(h\). This is worth doing every time — a sign error in a gradient is silent, and produces a model that trains steadily in the wrong direction.

2.6.1 Partial derivatives and the gradient

A loss depends on many parameters at once, not one. The partial derivative \(\partial f / \partial w_j\) is the slope in the direction of \(w_j\) alone, holding every other parameter fixed. Stacking all of them into a vector gives the gradient:

\[\nabla f = \left[ \frac{\partial f}{\partial w_1}, \frac{\partial f}{\partial w_2}, \dots, \frac{\partial f}{\partial w_p} \right]\]

The gradient points in the direction of steepest increase. Gradient descent (10.4) therefore steps in the direction of \(-\nabla f\).

2.6.2 The gradient of mean squared error

This is the one gradient the material actually needs, and Chapters 10 and 11 both use it. For \(\hat{y} = X\mathbf{w}\) and

\[J(\mathbf{w}) = \frac{1}{n}\sum_{i=1}^{n}\left(\hat{y}^{(i)} - y^{(i)}\right)^2\]

the gradient is

\[\nabla J = \frac{2}{n} X^\top (X\mathbf{w} - y)\]

Deriving that is 11.2's job. Verifying it numerically takes four lines and is the single best habit in this whole section.

2.7 Probability for Machine Learning

Classification models do not really output classes — they output probabilities, and a threshold turns those into classes (8.6). Naive Bayes (Chapter 15) is built directly out of the rules below.

2.7.1 Random variables, expectation and variance

A random variable takes a value determined by a random process. Two summaries characterize it:

Expectation is not "the most likely value" and need not be attainable — a fair die has expectation 3.5.

2.7.2 The distributions that actually come up

The normal distribution is disproportionately common for a specific reason covered in 2.8.3, not because real data is usually bell-shaped — it usually is not (4.4).

2.7.3 Bayes' theorem

Bayes' theorem inverts a conditional probability — it converts \(P(\text{evidence} \mid \text{cause})\), which is usually measurable, into \(P(\text{cause} \mid \text{evidence})\), which is usually what you want:

\[P(A \mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}\]

The classic demonstration is a medical test, and the result is genuinely counter-intuitive.

A test that is 99% sensitive returns a positive result that is right less than 2% of the time. Nothing is wrong with the test: the disease is so rare that the 5% of false positives drawn from the 999 healthy people vastly outnumber the true positives drawn from the 1 sick person.

Common mistake — base rate neglect. Reading a 99% sensitivity as "99% chance I have it" ignores \(P(A)\) entirely. This is not a curiosity; it is the arithmetic behind every rare-event classifier in Chapter 22. A fraud detector with excellent sensitivity, applied to a population where fraud is 0.1% of transactions, will produce alert queues that are overwhelmingly false alarms — and the fix is a threshold decision (8.6), not a better model.

2.8 Statistics for Machine Learning

2.8.1 Descriptive statistics

The robustness column decides real choices: it is why RobustScaler exists (5.5), why median imputation is the safer default (5.2), and why the IQR rule is the standard outlier test (5.3).

One bad value moves the mean from 11.6 to 93.0 and the standard deviation from 1.0 to 182.0. The median does not move at all, and the IQR barely. That single row is the entire argument for robust statistics.

2.8.2 Covariance and correlation

Covariance measures whether two variables move together, but its units are the product of theirs, so its magnitude is uninterpretable. Correlation is covariance divided by both standard deviations, which normalizes it to \([-1, 1]\):

\[r = \frac{\mathrm{Cov}(a, b)}{\sigma_a \sigma_b}\]

Pearson correlation measures linear association only. Spearman correlates the ranks instead, so it detects any monotonic relationship — which is why 4.6 recommends looking at both.

The U-shaped case is the warning: t determines curved exactly, and both correlations report essentially zero. A correlation of 0 means "no monotonic relationship", never "no relationship". Any feature-selection method built on correlation alone (7.2) inherits this blind spot.

Correlation is not causation, and in ML it does not need to be. A predictive model exploits association regardless of its cause, and that is legitimate — until someone acts on it. "Customers who contact support churn more" supports predicting churn; it does not support closing the support line. 24.4 returns to this distinction, which is where interpretability and causal claims part company.

2.8.3 Sampling, the central limit theorem and hypothesis tests

Any dataset is a sample from a larger population, so every statistic computed from it is itself a random quantity that would come out differently on a different sample. The central limit theorem says that the sample mean is approximately normally distributed regardless of the population's shape, provided the sample is large enough.

The quantity \(\sigma / \sqrt{n}\) is the standard error — the typical distance between a sample mean and the truth. Its \(\sqrt{n}\) shrinks slowly, which is a fact with direct consequences: quadrupling a test set only halves the uncertainty in its score. That is why 8.3 prefers cross-validation to a single large hold-out, and why comparing two models whose scores differ by less than their standard error is not a comparison at all.

A hypothesis test formalizes that comparison. The p-value is the probability of seeing a difference at least this large if there were no real difference. It is not the probability that the models are the same.

2.9 In Practice

2.9.1 Where this shows up: a linear model returning nonsense

The most common way this chapter's material surfaces in real work is a fitted model whose coefficients are absurd — enormous, wildly signed, and unstable between runs. The maths above diagnoses it in three lines.

The scenario: a colleague's feature table has total_spend, and also total_spend_usd — the same quantity after a currency conversion that was later reverted, so the two columns now differ only by floating-point dust. Nobody noticed, because nothing errors.

Two things to read here. First, rank says the matrix is fine and the condition number says it is not. matrix_rank answers "are any columns exactly dependent?"; the condition number answers "are any columns nearly dependent?" — and on real data only the second question matters. A rank check alone would have cleared this design matrix.

Second, forming \(X^\top X\) squares the condition number, from \(10^{9}\) to \(10^{15}\). That is the stability row of 2.5.2's table made concrete, and it is why the textbook normal equation is the worst way to solve this in practice.

Now solve it three ways — the textbook formula, NumPy's least-squares routine, and scikit-learn.

Three solvers, three completely different answers to the same problem. This is the moment the mathematics stops being decoration:

The explicit inverse is the only one that corrupts the good coefficients too — 2.4's "if you are writing inv, you want solve" is not a style preference, it is the difference between 0.643 and 0.499 on a coefficient someone is going to quote in a meeting.

But look at lstsq: it recovers monthly and tenure perfectly and still reports a spend coefficient of about ±70,000. It is not broken. The problem genuinely has no unique answer, and lstsq returns the minimum-norm one among infinitely many; scikit-learn returns a different one because it discards singular values below a cutoff. Both are defensible; neither is the answer, because there isn't one.

Proving the problem is non-identifiable. If two coefficient vectors that differ enormously make the same predictions, the data cannot distinguish between them. Add 1000 to one spend coefficient and subtract 1000 from the other:

The coefficients differ by 2000 and the predictions differ by three thousandths; the two models are indistinguishable to ten decimal places of RMSE. No amount of data, no better solver and no evaluation metric can tell these two models apart, because the information needed to separate the two columns is not in the data. The question "what is the effect of total_spend?" has no answer here — not an uncertain answer, no answer.

The fixes.

Dropping the duplicate restores a unique solution and the true effects come straight back. Ridge takes a different route: the \(\lambda I\) term makes the problem identifiable again (2.4.2) and splits the shared effect evenly between the two duplicate columns — 0.114 each — which is a principled, reproducible choice rather than an arbitrary one. Note that ridge's coefficients are on standardized features, so they are in standard-deviation units and are not comparable to the raw 0.5 and 0.2.

Dropping the column is the right fix here because the duplication is a data bug. Ridge is the right fix when features are genuinely correlated and none of them can be discarded.

The finding, in one sentence for a colleague: total_spend_usd duplicates total_spend, making the design matrix ill-conditioned (\(\kappa \approx 5\times10^9\)); the spend coefficients are non-identifiable — values 2000 apart fit the data identically — so no conclusion may be drawn from them, and the numbers differ depending on which solver was used. Every claim in that sentence was supported by three lines of NumPy. 11.7 turns the check into a routine one using the variance inflation factor.

2.9.2 When you need the maths — and when the library is enough

The pattern: the maths is not needed to run things. It is needed to explain a result that looks wrong, and to know whether a result that looks right actually is.

2.9.3 What goes wrong in production

Numerical failures are quiet. None of these raises an exception:

The underflow row is the one that most often reaches production undetected, so it is worth seeing happen.

In normal space the answer is exactly 0.0, and comparing two classes both scoring 0.0 is a coin flip. In log space the same quantity is an ordinary number and the comparison is exact. Every serious implementation of Naive Bayes works this way — sklearn's exposes it as predict_log_proba — and 15.6 covers it as a first-class technique rather than an optimization.

The general rule this chapter leaves you with: when a numerical result is surprising, check the conditioning and the scale before questioning the model. Most inexplicable machine learning results are explicable arithmetic.