← Contents Volume II · Lecture 6

Volume II · Generative Learning

6

What Independence Buys You

Ask an economist whether income and neighbourhood are independent, given profession, and the honest answer is no. Naive Bayes assumes yes anyway — for every pair of features, given the class — and the resulting classifier is simple enough to fit by counting, and works far better than the assumption deserves.

The mental model

Assume every feature is independent of every other feature, given the class — an assumption that is almost always false in the real world — and get, in exchange, an estimator so simple it can be fit by counting.

The assumption, stated precisely

Lecture 5 built a generative classifier around one assumption about P(x|y): that it was a multivariate Gaussian, shared covariance across classes. Naive Bayes keeps the generative framing — model P(x|y) and P(y), then apply Bayes' rule — but replaces the Gaussian assumption with a completely different one: that the individual features of x are conditionally independent of one another, given the class.

Naive Bayes (conditional independence) assumption P(x1, ..., xn | y) = ∏i=1n P(xi | y)

Written in words: once you know the class, the joint distribution of all n features collapses to a simple product of n one-dimensional distributions, each estimated on its own. That collapse is the entire reason the model is tractable — instead of estimating one joint distribution over an exponentially large space of feature combinations, you estimate n small, separate ones.

y x₁ x₂ x₃ x₄ x₅ no edges between the feature nodes — that absence IS the assumption
Figure 6a Every feature is a child of the class and nothing else. There is no arrow from x₁ to x₂: whatever correlation those two features have in the real world is thrown away entirely once the class is known.

It is worth naming the assumption honestly rather than dressing it up. It is called naive because it is essentially never exactly true — features are usually correlated with one another even after conditioning on the class. Word choices in a sentence are not independent given the topic; symptoms in a diagnosis are not independent given the disease. And yet the resulting classifier performs surprisingly well across a wide range of real problems. That is worth dwelling on as a genuine, still slightly surprising empirical fact, rather than something to explain away — a wrong assumption, used honestly, can still produce a useful model.

The one real failure mode: an unseen value zeroes everything

Naive Bayes has exactly one sharp, well-understood way to fail, and it follows directly from the product in the equation above. Because prediction multiplies probabilities across every feature, a single feature value that never once appeared with a given class in the training set gets an estimated probability of exactly zero for that class. Multiply anything by zero and the product is zero — so one unseen feature value at test time can force the entire prediction to zero, no matter how strongly every other feature points the other way.

That is not a rare edge case; it is the default outcome for any feature space large enough that most possible values were never seen during training. The fix is to stop estimating probabilities as bare frequency counts and instead assume every possible value has already been seen at least once, before any training data arrives at all.

Laplace smoothing φj|y=1 = ( ∑i 1{xj(i)=1 ∧ y(i)=1} + 1 ) / ( ∑i 1{y(i)=1} + 2 )

Compare this to the raw frequency estimate it replaces — the same ratio without the added +1 in the numerator and +2 in the denominator. The added terms are exactly equivalent to having already observed one extra training example of each possible outcome (1 and 0) before counting the real data. As the real sample size grows, the +1 and +2 become negligible and the estimate converges to the ordinary frequency count; at small sample sizes, they are precisely what keeps a probability away from a hard zero.

WITHOUT SMOOTHING φ = 0 / 12 = 0 unseen value ⇒ P = 0 ⇒ whole product collapses to 0 WITH LAPLACE SMOOTHING φ = (0+1) / (12+2) ≈ 0.07 small, but nonzero other features still get a say
Figure 6b The same unseen feature value under both estimators. Without smoothing it is a hard zero that overrules every other feature; with the +1/+2 correction it is small but survives, letting the rest of the evidence through.

A second generative model, not a rival to the first

It is worth placing Naive Bayes next to Lecture 5's GDA rather than treating them as competitors. Both are generative: both model P(x|y) and P(y) and reach a prediction via Bayes' rule. They differ only in what they assume about P(x|y) — GDA assumes a joint Gaussian shared across features; Naive Bayes assumes conditional independence between individual features instead. Neither assumption is more "generative" than the other; they are two different bets about the same quantity.

That difference in assumption has a practical consequence worth stating plainly: Naive Bayes does not require x to be continuous, or Gaussian-shaped, or even numeric in any conventional sense. Each P(xi|y) is estimated on its own, so it can just as easily be a distribution over a discrete or binary value as over a real number. That is precisely why Naive Bayes remains the standard choice for discrete or categorical features — word-count-style features being the canonical example — where GDA's Gaussian assumption would not even make sense to write down.

Volume II closes here, on that contrast. Volume III turns the question around entirely: rather than assuming a shape for the data and deriving a boundary, it asks what the widest possible boundary looks like when the data is left unmodelled — and starts building toward the support vector machine.

Read the primary source

❦