The Information Domain: Entropy as a Codelength, and Which of Your Objectives Are One by Theorem

Start with the one sentence everything else hangs on, stated with the asymptote that is usually dropped. The entropy of a memoryless source is the number of bits per symbol you need to describe its draws, if you code optimally, in the limit of long blocks. For a single draw the best prefix code pays between $H$ and $H+1$ bits, and the gap closes only when every probability is a power of two. When the symbols are dependent the quantity that owns the compression limit is the entropy rate, $\bar H = \lim_{n\to\infty}\tfrac1n H(X_1,\dots,X_n)$, not the entropy of one symbol: draw one fair bit $B$ and set $X_t = B$ for every $t$, and each symbol has $H(X_t) = 1$ bit while the whole sequence has one bit of uncertainty, so $\bar H = 0$. §0 works with the memoryless case; §8 is about the processes where the two differ. Every result in this piece is a consequence of taking that sentence seriously — including the cases where a quantity that looks like an entropy is not one, and where a number that looks like a measurement is a property of the ruler.

Here is the claim, at the width the mathematics supports, because the tidy version is wrong in both directions. We are not claiming that machine learning is information theory wearing a hat, and we are not claiming that every budget you allocate is secretly a rate–distortion problem — §5 and Appendix D say exactly where that fails. The claim is this:

Information theory supplies a common language for several objectives you already optimise, but its connections operate at different levels. Cross-entropy and KL are related by an exact identity. Variational objectives split exactly into a distortion and an upper bound on rate — a decomposition of rate–distortion shape, whose rate meets Shannon's only where the family contains the optimum, as the linear-Gaussian example of §6 does. A stationary Gaussian process has an entropy rate determined by its power spectrum. Other connections — where a diffusion model's compute goes, what a fixed-size state can carry — are hypotheses whose consequences have to be established separately. The value of the lens is not that everything is secretly information theory. It is that it tells you which constraints are theorems, which quantities are measurements, and which analogies are bets.

Every connection in this piece carries one of four labels. If you are here for the floor under your loss, skip straight to §1 and come back to this table the first time a label stops you:

Three running examples carry the same numbers through the whole piece, so that when two sections seem to disagree you can see which one is wrong. A $K$-class classifier trained with label smoothing ($K = 10$ and $K = 128{,}000$, $\varepsilon = 0.1$): its floor in §1, its temperature in §1.1, its softmax as a variational identity in §9.8. A one-dimensional linear-Gaussian VAE ($z\sim\mathcal N(0,1)$, $x\mid z\sim\mathcal N(z, 1/10)$, encoder $\mathcal N(ax, s^2)$): its exact posterior, its ELBO level set and its rate–distortion frontier in §6, every number in closed form. An AR(1) process with unit innovation variance: the source of §5's real quantiser at $\phi = 0.9$, both spectral functionals in §8, and the length-scale callback in §11.3.

Prerequisites. Probability at the level of expectations and conditional distributions; logarithms; one linear-algebra fact (a log-determinant is a sum of log-eigenvalues); gradient descent. No prior information theory.

Two reading paths, and two sections neither may skip. The practitioner path is §0 → §1 → §2 → §3 → §5 → §6 → §7 → §12, doing the five checkpoint exercises that lie on it: exercise 1 closes §1, exercises 2 and 3 close §2.1 and §2.2, exercise 4 closes §6 and exercise 5 closes §7. The theorist path is §0 → §1 → §2 → §3 → §5 → §6 → §8 → §9 → §10 → §11, and adds the sixth exercise, which closes §8. §1 and §3 are on both paths: §1 holds the empirical-versus-population distinction and the one table that stops a floor being mis-subtracted, §3 is what §6 and §7 are built on, and §5–§6 are on the theorist's because they are the thesis. §4 (the information plane) is the one advanced branch neither path requires; take it by interest, and take §8 by interest if you are on the practitioner path. The history is Appendix A, the cross-field audit Appendix B, two toy models of ours Appendix C, and the graded ledger of how this piece bears on our earlier posts is Appendix D. This piece stands alone: where an earlier post of ours is relevant it is linked as a citation, the way a paper would be, and nothing below requires having read it.

Companion notebooks. 00_information_theory_from_scratch.ipynb builds entropy, cross-entropy, KL, mutual information, the information plane and a rate–distortion curve in pure NumPy; 01_entropy_across_the_series.ipynb carries the longer experiments — the entropy-rate identity, where a diffusion model's forward information goes and the I-MMSE check, the ELBO level set on the rate–distortion frontier, a real quantiser against its bound, a verification-noise toy, and what recursive training does to tails. Both ship fully executed, and every number in a code block below is copied from their printed output; the few remaining worked-example numbers come from worked_numbers.py (27 numbered sections), with the arithmetic shown inline. Training is float64 throughout, and the float32 and float16 numbers of §4.2 are readout casts of the same activations, not networks trained at lower precision. Environment, seeds and the reproducibility scope — every printed number within the environment; the closed-form checks across platforms; the trained-network collision counts not — are printed by each notebook's first cell and discussed where they matter, in §4.2.

Units, once. We write $\log_2$ and say bits unless a formula says otherwise; several results below are native to nats because the literature states them with $\ln$, and mixing the two inside one bound is the most common way to get a wrong number out of a right theorem. The ledger:

0. One quantity, four costumes

Start with the definition and then immediately do something useful with it. For a discrete distribution $p$ over outcomes $x$:

The standard gloss is "average surprise," which is true and slightly useless. The operational reading is better, and it has to be stated with its asymptote attached, because it is almost always stated without one:

$H(p)$ is the number of bits per symbol you need to describe independent draws from $p$, if you code optimally, in the limit of long blocks. For a single draw the number of yes/no questions is an integer, and the best you can do is bounded by $H(p) \le L^\ast < H(p)+1$ — Shannon's bound for prefix codes, which Huffman's construction attains. The $p=0.9$ coin has entropy 0.469 bits and still costs one whole question per draw; block a hundred draws together and the cost per draw approaches 0.469. Entropy is the per-symbol cost of describing the source, which is Shannon's source-coding theorem (§9.1 sketches the proof), not the cost of describing one symbol — and for a source with memory that per-symbol cost is the entropy rate of the introduction, never more than $H$ of a single symbol and usually less.

Three sanity checks from the notebook, worth running once by hand so the numbers stop being abstract:

H(uniform over 8)   = 3.000000 bits   (exact: log2(8) = 3)
H(point mass)       = 0.000000 bits   (exact: 0)
H(dyadic skewed)    = 1.875000 bits   (exact: 1.875)

Worked example — the one case where a single-symbol code is exact. The third distribution is $p = (\tfrac{1}{2}, \tfrac{1}{4}, \tfrac{1}{8}, \tfrac{1}{16}, \tfrac{1}{16})$ padded to eight slots with three zeros. It is called dyadic because every probability is a power of $\tfrac{1}{2}$, so the ideal code length $-\log_2 p(x)$ is a whole number of bits for every symbol: $1, 2, 3, 4, 4$. Assign the codewords 0, 10, 110, 1110, 1111 — a prefix code, so no codeword begins another and the stream decodes without separators. The expected length is

which equals $H(p)$ exactly. A dyadic distribution is precisely when the prefix-code gap closes; every non-dyadic distribution pays a strictly positive penalty per symbol for rounding fractional bits up to whole codewords, and that penalty is what block coding and arithmetic coding (§9.2) remove. And the claim that uniform is the maximum has a two-line proof rather than a vote: $H(p) = \mathbb{E}_p[\log_2(1/p(x))] \le \log_2\mathbb{E}_p[1/p(x)] = \log_2|\mathrm{supp}\,p| \le \log_2 K$ by Jensen's inequality, since $\log$ is concave. The expectation of $1/p(x)$ counts the outcomes with positive probability, which is why the dyadic example padded with three zeros is bounded by $\log_2 5$ and not only by $\log_2 8$; equality throughout needs $1/p(x)$ constant on a support of size $K$, i.e. $p$ uniform. (The notebook's 200,000 random distributions on eight outcomes, none above 3 bits, are a picture of the inequality, not the argument for it.)

The single most useful intuition pump is the biased coin:

A fair coin is exactly 1 bit. A coin with $p = 0.9$ is 0.4690 bits — nine-to-one odds have already destroyed more than half the uncertainty of a fair coin.

Remember that shape. Entropy falls off fast as a distribution concentrates, which is why a model that is "only somewhat confident" has already given away most of its uncertainty, and why calibration errors in the confident regime cost more than they look like they should.

The four costumes

Everything below is built from $H$ and two relatives:

• Cross-entropy $H(p,q) = -\sum_x p(x)\log_2 q(x)$ — the cost of coding $p$'s data with $q$'s codebook.
• KL divergence $D_{\mathrm{KL}}(p \Vert q) = \sum_x p(x)\log_2 \frac{p(x)}{q(x)}$ — the excess cost of that mistake.
• Mutual information $I(X;Y) = H(X) - H(X\mid Y)$ — how much learning $Y$ shrinks your uncertainty about $X$.

And the identity that makes the first two the same object:

We verified it over 20,000 random pairs — maximum absolute deviation 8.882e-16, which is floating-point noise. Be clear about what such a check is: it validates that our implementation agrees with an algebraic identity, and nothing else. It is not evidence for the identity (the proof is one line: expand the log of a ratio) and it does not vouch for any other experiment in the notebooks. We say this once here so that no later "agrees to $10^{-9}$" is read as more than the software test it is.

What the identity buys you is the most under-appreciated line in applied machine learning:

Minimising cross-entropy against a fixed target is exactly minimising KL divergence to that target, because $H(p)$ does not depend on your parameters. Minimising the population cross-entropy is a KL projection of the data distribution onto your model class. Training minimises the cross-entropy of the empirical distribution — a KL projection of the training sample — and the two coincide only in the large-sample limit, under conditions that are usually left unstated (a fixed model class, independent draws, a minimiser that exists and is found). We say "empirical" once here and mean it wherever a training loss appears below.

It does not follow that every classifier is doing variational inference. That claim needs a latent variable and an approximate posterior, and it is made — correctly, with its assumptions — in §6.

1. Your loss has a floor, and you built it yourself

Here is the first place the theory pays rent. Practitioners routinely watch a training loss plateau above zero and reach for the usual explanations — underfitting, a bad learning rate, insufficient capacity. Often the correct explanation is arithmetic: you added label smoothing, and label smoothing installs a floor.

If your targets are smoothed, the target distribution $p$ is no longer a point mass, so $H(p) > 0$. By the identity in §0 the best achievable cross-entropy is not zero — it is $H(p)$, reached when $q = p$ exactly. But "smoothed" is not one distribution, and a floor table that does not say which one it used is useless. There are two conventions in common use:

• Convention A — mix with uniform over all $K$ classes: $p = (1-\varepsilon)\,\mathbf{1}_y + \varepsilon/K$, so the true class gets $1-\varepsilon+\varepsilon/K$ and every other class gets $\varepsilon/K$. This is what label_smoothing= does in PyTorch's CrossEntropyLoss and in most frameworks that copied it (Szegedy et al., 2016).
• Convention B — hold the true class at $1-\varepsilon$ and spread $\varepsilon$ over the other $K-1$ classes, $\varepsilon/(K-1)$ each. The companion notebook prints both conventions; the table below is its convention-B table, and the convention-A row worked by hand after it is reproduced by the notebook to the same digits.

The notebook's table, convention B:

Irreducible cross-entropy floor H(p) induced by label smoothing
   K    eps   floor [bits]   floor [nats]
  10   0.00       0.000000       0.000000
  10   0.10       0.785988       0.544805
1000   0.10       1.465430       1.015758
128000   0.00       0.000000       0.000000
128000   0.05       1.134686       0.786504
128000   0.10       2.165573       1.501061
128000   0.20       4.115083       2.852358

Worked example — the same row under convention A, by hand. $K=10$, $\varepsilon=0.1$: the true class gets $0.9 + 0.01 = 0.91$ and each of the other nine gets $0.01$.

with four-decimal intermediates — the hand line is the rounded one. Unrounded it is 0.721763 bits (0.500288 nats), which is the notebook's convention-A row and the checkpoint value at the end of this section.

So the two conventions give 0.722 bits (0.500 nats) and 0.786 bits (0.545 nats) for the same nominal $\varepsilon$ at $K=10$ — a 9% difference in the floor, large enough to misdiagnose a run. At $K = 128{,}000$ the difference vanishes: convention A gives 2.165557 bits / 1.501050 nats against B's 2.165573 / 1.501061, because $\varepsilon/K$ on the true class is negligible when $K$ is six figures. The large-vocabulary warning therefore survives either way:

A vocabulary of 128,000 tokens with $\varepsilon = 0.1$ cannot go below 1.501 nats under either convention. A model sitting at 1.52 nats is not underfitting; it is within 0.02 nats of the per-example floor of its own smoothed objective on clean labels, and it will be reported as broken by anyone comparing against a run that used $\varepsilon = 0$. The same $\varepsilon = 0.1$ costs about 0.5 nats at $K=10$ and 1.5 nats at $K=128{,}000$ — the floor scales with vocabulary, which catches people moving between model families.

Procedure 1 — read the floor off the tensor. (i) Materialise one target row exactly as it reaches the loss: after smoothing, after any mixing or masking. (ii) Compute $-\sum_c t_c\log t_c$ in the log base your loss uses. (iii) Apply the same reduction your loss applies (mean over which tokens, sum, non-padding only). (iv) That number — not a table, not the convention you believe your framework uses — is your floor on clean labels. On noisy labels there are two more floors, and rule 3's table says which loss each belongs to.

Practical rules, in order:

1. Read $H(p)$ off the tensor you actually pass to the loss — Procedure 1 above. It is a one-line check and the only one that cannot be wrong about the convention.

2. Check the reduction. mean over tokens gives a per-token floor in the units above; sum scales it by the token count; a loss averaged over non-padding tokens only has yet another denominator. The floor must be computed in the same units as the loss you are reading.

3. Training loss and held-out loss have different floors, and there are two held-out losses. On training data with hard labels the floor is zero and a memorising model can reach it. On held-out data the labels are uncertain — $P(Y = c\mid X = x)$ is not a point mass — and which floor applies depends on which loss you score. Under convention A the expected smoothed target at $x$ is $r_\varepsilon(c\mid x) = (1-\varepsilon)P(Y = c\mid x) + \varepsilon/K$, and the four cases at one point, $P(Y = 1\mid x) = 0.9$, $K = 2$, $\varepsilon = 0.1$, are these (worked_numbers.py §11):

   the target you score against	minimiser $q^\ast$	floor at that point	what it is
   smoothed $r_\varepsilon$ — the loss you trained	$q = r_\varepsilon = (0.86, 0.14)$	$H(r_\varepsilon) = 0.584$ bits	the held-out floor of the smoothed objective
   hard label — what most validation code computes	$q = P(Y\mid X) = (0.9, 0.1)$	$H(Y\mid X) = H_2(0.9) = 0.469$ bits	the held-out floor of the hard-label objective
   hard label, but scored at the smoothed optimum $q = r_\varepsilon$	—	$-0.9\log_2 0.86 - 0.1\log_2 0.14 = 0.479$ bits	attains neither floor
   "$H(Y\mid X)$ plus the table's $H(t_y)$"	—	$0.469 + 0.286 = 0.755$ bits	wrong — the two entropies do not add

   The additive formula errs in the dangerous direction: a floor is a minimum, so overstating it makes a model at 0.6 bits look as if it had beaten its floor, and that is the misdiagnosis. The fair-label counter-example ($Y$ independent of $X$) hides the first three rows, because then $r_\varepsilon = (0.5, 0.5)$ for every $\varepsilon$ and every floor is exactly 1 bit — while the additive formula says 1.286. Subtract 0.584 from a hard-label curve and you have subtracted the wrong floor. Both noisy-label floors need $P(Y\mid X)$, so neither is readable from the targets alone; §9.4 (Fano) is the tool for bounding $H(Y\mid X)$.

   The restriction, and what $\varepsilon$ is not. Convention A is the target you would get from uniform label noise at rate $\varepsilon$, but $\varepsilon$ is a regulariser you chose, not an estimate of a noise rate: a model on its smoothed floor is optimal for the objective you wrote down, is not evidence that the labels are noisy at rate $\varepsilon$, and by construction reports 0.86 where $P(Y = 1\mid x) = 0.9$ — miscalibrated relative to the truth while sitting exactly on its floor. Excess over the floor is a statement about the objective you trained; calibration is a separate measurement, and §1.1 says how to make it.

4. Compare excess loss, never raw loss, across runs that differ in smoothing, vocabulary or tokenizer. And across tokenizers, per-token loss is not comparable at all: a tokenizer that produces more tokens per byte spreads the same information over more loss terms. Report bits per byte (total log-loss over the byte count of the text) when the vocabularies differ; it is the one unit that survives the change.

5. Distillation has the same floor. Training a student by cross-entropy against a teacher's soft distribution is KL to a target with its own entropy; the student's loss cannot go below the teacher's mean per-token entropy, and a student that reaches it has matched the teacher, not failed to beat it.

6. Every floor, and every number in this piece, assumes a distribution. On shifted data the identity in §0 reads $H(p_{\text{test}}, q) = H(p_{\text{test}}) + D_{\mathrm{KL}}(p_{\text{test}}\Vert q)$: a model that sat at its floor on the training distribution pays the full KL to the new one, and the floor itself moves, because $H(Y\mid X)$ is a property of the distribution rather than of the task's name. A perplexity measured on text from a different source than the training set is a codelength under the wrong code; a mutual-information estimate (§2.2) or a Fano floor (§9.4) computed under one distribution says nothing under another. Say which distribution every information quantity was computed under, and re-measure after a shift rather than carrying the number across.

1.1 Perplexity, temperature, and what a proper scoring rule does and does not promise

Perplexity is $2^{H}$ when $H$ is in bits, $e^{H}$ in nats — the effective number of equally likely options. For a language model the quantity reported is the exponential of the held-out cross-entropy per token, $\exp(-\tfrac{1}{N}\sum_i \log q(x_i\mid x_{<i}))$, which is the model's codelength on text it did not see. It is not the entropy of the model's own next-token distribution; those agree only when the model is the true distribution. A perplexity of 17 means the model's held-out surprise matches a uniform choice among 17 tokens.

Temperature is a direct entropy dial on that next-token distribution. From the notebook, for a single distribution over $K=50$ with logits drawn from $\mathcal{N}(0, 2^2)$:

      T   H [bits]   perplexity   p(top-1)  90% mass in
    0.1     0.1817         1.13     0.9725         1 tokens
    0.3     1.1949         2.29     0.7264         2 tokens
    0.7     3.3020         9.86     0.3431        11 tokens
    1.0     4.0870        16.99     0.2176        17 tokens
    1.5     4.7595        27.09     0.1289        25 tokens
    2.0     5.0904        34.07     0.0914        33 tokens
  100.0     5.6436        49.99     0.0208        45 tokens

Two things are worth extracting. First, the "90% mass in" column is the one that predicts sampling behaviour, and it is wildly nonlinear in $T$: going from $T=0.3$ to $T=0.7$ takes the nucleus from 2 tokens to 11. That is the range where most production sampling lives, and it is where small temperature changes have the largest effect on what can come out. Second, the limit: uniform over $K=50$ is $H = 5.6439$ bits, and at $T=100$ we measured 5.6436 — the logits have been thrown away. High temperature does not make a model creative; past a point it makes the model absent.

Now the point about scoring rules, which this table is the right place for. Cross-entropy is a strictly proper scoring rule: its expectation under the true distribution is uniquely minimised by reporting the true distribution. That is why it is the right training loss for probabilities. It is not a promise that a low loss means a calibrated model, and the table shows why in one column: every row has the same $\arg\max$, so every row has the same accuracy, while the loss and the calibration change with $T$. Temperature scaling (Guo et al., 2017) fits exactly this one parameter on held-out data to minimise cross-entropy, and it typically fixes the calibration of an over-confident network without touching a single prediction. So: a loss gap between two models can be a calibration gap rather than an accuracy gap, and the way to tell is to measure accuracy and a reliability diagram separately rather than to read either off the loss. (The Brier score is the other proper scoring rule you will meet. It is bounded, which makes it gentler on confident mistakes and less informative about them; choosing between the two is choosing how to price confidence, not choosing which one is correct.)

1.2 KL is not a distance, and the asymmetry is a design decision

$D_{\mathrm{KL}}(p\Vert q) \neq D_{\mathrm{KL}}(q\Vert p)$, and the gap is not a technicality — it determines what your model does when it cannot fit the target. Measured:

p = uniform(4), q = nearly a point mass
KL(p||q) = 5.475422 bits   (mass-covering: punishes q for ignoring p's support)
KL(q||p) = 1.965781 bits   (mode-seeking: q only has to sit somewhere p allows)
ratio    = 2.79x  — 'KL distance' is not a distance.

Forward KL $D(p\Vert q)$ is mass-covering. It integrates against $p$, so wherever the data has mass and your model does not, you pay. Under a model family too simple to fit every mode, the minimiser spreads itself across the modes. This is maximum likelihood. Reverse KL $D(q\Vert p)$ is mode-seeking. It integrates against $q$, so the model is only penalised where it puts mass, and it can ignore an entire mode for free. This is the variational objective, and it is the standard explanation for why mean-field variational inference under-estimates posterior variance.

Hold the generative-modelling corollary loosely. "Forward KL makes blurry images, reverse KL makes sharp ones" is a tendency under a misspecified unimodal family, not a law. Whether a maximum-likelihood model's samples look blurry depends on the likelihood you chose — a per-pixel Gaussian likelihood is what produces averaged images, and a model with the same KL direction but an autoregressive or flow likelihood does not — on whether the family can reach the modes at all, and on how you sample at generation time. The asymmetry tells you which mistakes the objective forgives. It does not tell you what the pictures look like.

The degenerate case matters more than the ratio:

p with a zero where q is positive: KL(p||q) = 1.000000 bits (finite)
the same pair the other way round: KL(q||p) = inf

An infinite KL is not a numerical problem to be clipped away — it is the objective telling you that $q$ places mass where $p$ says the event is impossible. If you are clamping KL terms to keep training stable, you are suppressing a signal about support mismatch. Look at the support first. (When supports genuinely differ by design — two disjoint datasets, two generators — KL is the wrong instrument altogether; Jensen–Shannon, total variation or a Wasserstein distance are the ones that stay finite, and §10.4 needs total variation for exactly this reason.)

1.3 The numerical box: how to actually compute it

One concrete engineering section, because cross-entropy's failure modes are silent and its cost profile surprises people.

Take logits, not probabilities. The cross-entropy of a softmax is $-z_y + \log\sum_j e^{z_j}$, and the second term must be computed as a shifted log-sum-exp, $m + \log\sum_j e^{z_j - m}$ with $m = \max_j z_j$. The notebook shows what happens otherwise:

naive softmax vs shifted softmax
  dtype=float64  max logit    100: naive finite?  True   naive[0]=0.66524095  shifted[0]=0.665241
  dtype=float64  max logit    800: naive finite? False   naive[0]=nan         shifted[0]=0.665241
  dtype=float32  max logit    100: naive finite? False   naive[0]=nan         shifted[0]=0.665241
 
float32 overflows at exp(~88). Models train in float32 or bfloat16. Shift, always.

A logit of 100 is not exotic in a large model, and in bfloat16 the usable range is the same as float32 with far fewer mantissa bits. Keep the logit tensor in at least float32 for the loss even when the rest of the model is in half precision; a cross-entropy kernel that accepts probabilities has already lost the game before it starts.

Mask zeros, do not clip them. The term $p\log p$ at $p=0$ is $0\cdot(-\infty)$ and must be defined as zero. The tempting alternative — clipping probabilities to $\epsilon$ — does not error, does not warn, and changes the answer by an amount that depends on $\epsilon$ and on how many zeros the distribution has:

masked  H(p) = 0.8812908992 bits   (correct)
clipped H(p) at eps=0.001  = 0.9127844694 bits   error +3.15e-02
clipped H(p) at eps=1e-06  = 0.8813523780 bits   error +6.15e-05

The loss layer can be the most expensive thing you run. The logit tensor is $\text{batch}\times\text{sequence}\times\text{vocabulary}$, and vocabulary is now six figures. Cut Your Losses in Large-Vocabulary Language Models (arXiv:2411.09009) reports that for small models the cross-entropy layer consumes an order of magnitude more memory than the rest of the model combined, and reduces the loss computation's footprint from 24 GB to 1 MB on Gemma 2 (2B) by never materialising the full logit matrix. The last conceptual step of your model is often the most expensive thing in the training loop. That is worth knowing before you buy more accelerators.

Checkpoint exercise 1 (floors). Materialise one row of your own training targets under your framework's label smoothing and compute its entropy in the units of your loss. Then prove that under convention B the floor is exactly $H_2(\varepsilon) + \varepsilon\log_2(K-1)$, check it against the $K=10$ and $K = 128{,}000$ rows, and show that under either convention it grows as $\varepsilon\log_2 K$ plus a constant. Finally, for a fair label independent of the input, show that the held-out floor under convention A is 1 bit at every $\varepsilon$. Checkpoints: $K=10$, $\varepsilon=0.1$: 0.785988 bits (B), 0.721763 bits (A); the independent-label floor is $H(r_\varepsilon) = 1$, not $1 + H(t_y)$.

2. Mutual information, and the law that says post-processing cannot help

Mutual information is the natural measure of "how much does knowing this tell me about that," and its most useful form is the third one below:

Mutual information is the KL divergence between the joint and the product of the marginals — a measurement of how far from independent two variables are, in the same units as everything else. For the binary symmetric channel — $X$ uniform, $Y$ equal to $X$ flipped with probability $f$ — the closed form is $I(X;Y) = 1 - H_2(f)$, with $H_2$ the binary entropy. Measured against it:

     f    I(X;Y) measured   1 - H(f) exact
  0.00           1.000000         1.000000
  0.01           0.919207         0.919207
  0.10           0.531004         0.531004
  0.25           0.188722         0.188722
  0.50           0.000000         0.000000

Note the shape: a 10% error rate costs you nearly half the bit. Channel capacity degrades much faster than error rate suggests, which is the same lesson the biased coin taught in §0, and which is why "97% accurate" and "99% accurate" are not a 2% difference in information terms.

2.1 The data-processing inequality, with its proof

If $X \to Y \to Z$ is a Markov chain — $Z$ depends on $X$ only through $Y$ — then

Proof sketch (three lines). Expand $I(X;Y,Z)$ by the chain rule in both orders: $I(X;Y,Z) = I(X;Z) + I(X;Y\mid Z) = I(X;Y) + I(X;Z\mid Y)$. The Markov property says $I(X;Z\mid Y) = 0$ — given $Y$, $Z$ tells you nothing further about $X$. Conditional mutual information is non-negative (it is a KL divergence), so $I(X;Y\mid Z)\ge 0$, and therefore $I(X;Z) = I(X;Y) - I(X;Y\mid Z) \le I(X;Y)$. Equality holds exactly when $I(X;Y\mid Z)=0$, i.e. when $Z$ is a sufficient statistic of $Y$ for $X$ — the processing threw away only what was irrelevant. $\square$

Worked example — a cascade of two noisy stages, by hand. Let $X$ be a fair bit, let $Y$ be $X$ passed through a binary symmetric channel with flip probability $f = 0.1$, and let $Z$ be $Y$ passed through a second identical channel. The first stage leaves $I(X;Y) = 1 - H_2(0.1) = 0.531$ bits (the table above). Two independent flips compose into one flip with probability $f' = 2f(1-f) = 2(0.1)(0.9) = 0.18$, so $I(X;Z) = 1 - H_2(0.18)$. Now $H_2(0.18) = -0.18\log_2 0.18 - 0.82\log_2 0.82 = 0.18(2.474) + 0.82(0.286) = 0.445 + 0.235 = 0.680$, giving $I(X;Z) = 0.320$ bits. The second stage destroyed $0.531 - 0.320 = 0.211$ bits, and no decoder placed after $Z$ — however large, however well trained — sees more than 0.320 bits about $X$. That is the whole inequality in one pipeline.

(The notebook also runs the inequality on 5,000 random Markov chains and finds no violation — a software test of our implementation, not evidence for the theorem.)

What the DPI does and does not say about your pipeline. It says your preprocessing is a hard ceiling on the information available to the Bayes-optimal predictor downstream: a frozen encoder bounds every head you attach, an early quantisation step bounds every later architecture, and nothing recovers what was removed. It does not say that a step which loses information will hurt a model you can actually train, and it says nothing at all about steps that lose none — keep the two kinds of preprocessing apart. Standardising a feature, $Z = (Y-\mu)/\sigma$ with fixed $\mu$ and $\sigma > 0$, is invertible, so $I(X;Z) = I(X;Y)$ exactly: mutual information is invariant under invertible maps of either variable, and whatever standardisation does for optimisation it does without moving the ceiling. Cropping, quantising and discarding nuisance variation are non-invertible; they lower the ceiling, and a finite-sample learner — which is not the Bayes-optimal predictor — routinely gains held-out accuracy from them anyway, because they shrink the hypothesis space it has to search with the data it has. That is the practitioner's version of the inequality: an operation can make information easier for a finite model to use without changing, or while reducing, what an ideal predictor could extract. So when a bigger model fails to beat a smaller one on the same features, the DPI is one hypothesis — the information is not in the features — and it is a hypothesis you can test rather than assume: estimate how much information is there (§2.2, carefully) and turn it into a floor on error with Fano's inequality (§9.4). Learnability and information are different ceilings, and only the second one is what this theorem bounds.

Checkpoint exercise 2 (data processing). Three BSC($f$) stages in series: derive the composite flip probability and compute $I(X;Z)$ for $f = 0.1$. Then exhibit a Markov chain $X\to Y\to Z$ with $I(X;Z) = I(X;Y)$ and $Z\ne Y$, and say what that makes $Z$. Checkpoints: composite flip $\tfrac12\big(1-(1-2f)^3\big) = 0.244$; $I(X;Z) = 1 - H_2(0.244) = 0.198$ bits, below the two-stage 0.320; equality needs $Z$ to be a sufficient statistic of $Y$ for $X$.

2.2 Your mutual-information estimate is a measurement with an error bar you have to earn

Here is the part that should change how you read papers. The formulas above take a distribution as input. In practice you have samples, and the obvious move is to build a histogram and put it through the formula. That plug-in estimator has a known leading-order bias: on a fixed $K\times L$ table with every cell probability positive, in the large-$N$ regime,

(the Miller–Madow correction applied to the three entropies in $I$). The notebook measures this against a ground truth of exactly zero — $X$ and $Y$ drawn independently on an $8\times 8$ table:

TRUE mutual information is 0.000000 bits (X and Y drawn independently, 8x8 table)
       N   mean I_hat       std   predicted bias   ratio
      20     1.388577  0.193090         1.767301   0.786
      50     0.803480  0.108236         0.706921   1.137
     200     0.196598  0.038791         0.176730   1.112
    1000     0.035860  0.007079         0.035346   1.015
    5000     0.007059  0.001517         0.007069   0.999
  100000     0.000347  0.000068         0.000353   0.981

At $N=200$ on an $8\times 8$ table, the estimator confidently reports 0.197 bits between two independent variables. For scale, a perfectly informative binary variable carries 1.000 bits. The correction formula tracks well in the regime it was derived for and fails where you would most want it: at $N=20$ it predicts 1.767 bits while the estimator reports 1.389. The asymptotic expansion is unreliable when the table has fewer samples than cells, which is the regime it exists to describe. The correction is a diagnostic, not a rescue. And the bias does not cancel when there is a real signal:

       N    true I     I_hat     error
      50   0.53100   0.54576  +0.01476
     200   0.53100   0.53917  +0.00817
    1000   0.53100   0.53184  +0.00084
   10000   0.53100   0.53096  -0.00004

Now a correction to a rule you will see stated often: treat every mutual-information estimate as an upper bound. That is not generally true. The leading-order term is positive, so under independence and in the large-sample regime the estimate is an over-estimate. But the bias of the plug-in estimator is not positive for every distribution and sample size, and a one-line counter-example settles it: let $X = Y$ be a fair bit, so $I(X;Y) = 1$ bit exactly, and draw one sample. The empirical joint is a single cell, $\hat H(X) = \hat H(Y) = \hat H(X,Y) = 0$, and $\hat I = 0$ — an under-estimate by a full bit. More generally, a strongly dependent pair observed on a sparse table under-counts its own dependence, while an independent pair on a sparse table over-counts. The sign of the error depends on the distribution, the table and $N$, and no single correction knows which case you are in.

Procedure 2 — how to report a mutual-information estimate. Treat it as an uncertain, estimator-dependent measurement, not as a bound in either direction, and report four things with the number. (i) $N$ and the estimator — bins and their count, or the nearest-neighbour $k$, or the critic and batch size. (ii) A permutation null: shuffle $Y$, re-estimate, repeat a thousand times — that is the distribution of $\hat I$ under independence at your $N$ and your bins, the only null that matters — and report its 95th percentile beside your estimate. (iii) A bootstrap interval for the spread. (iv) For a difference between two setups, in this order. First name the estimand both numbers estimate — the same quantity, on the same variables, under the same discretisation — because two plug-in values at different $N$ or different bins are estimates of different things until you have said why they are not: the bias term above moves with both, and so does the regime in which it is valid. Raw numbers across unmatched settings are not a comparison; they become one only once the bias of each estimate has been accounted for and the uncertainty of each carried through, which is possible and is work. Second, if the settings are matched, the comparison is controlled, not corrected: matching removes the sample-size and binning contributions and leaves the part of the bias that depends on the underlying distribution, so two matched estimates can still carry different biases. Matching is a useful experimental control — neither a requirement nor a sufficient correction. Third, keep the test and the interval apart. The permutation null of (ii) tests one stated null, independence under exchangeability, and that is all it tests; it is not an interval for a difference. The interval comes from a resampling procedure built for the difference — paired when the two estimates share observations, independent-sample otherwise. If a paper reports that layer 5 carries more information than layer 3, this is the list to check before believing it.

Histograms are not the only estimator, and in more than a few dimensions they are the wrong one. Nearest-neighbour estimators (Kraskov, Stögbauer & Grassberger, 2004) avoid binning and have their own small-sample failure modes; neural estimators such as MINE (arXiv:1801.04062) exist because both fail in high dimensions, and they carry variance pathologies of their own. §3 states the variational bounds that make the neural ones principled, and the ceiling that caps the most popular of them. There is no free estimator.

Checkpoint exercise 3 (estimation). Reproduce the $8\times 8$ independence experiment at $N = 200$ with a 1,000-shuffle permutation null, and report the 95th percentile of $\hat I$ under the null. Then construct a dependent distribution and a sample size at which the plug-in estimate is biased downward, and state why. Checkpoints: the notebook's null has its 95th percentile at 0.265 bits, with the single observed estimate at 0.229 inside it; $X = Y$ at $N = 1$ gives $\hat I = 0$ against a truth of 1 bit.

3. Discrete and differential entropy are different objects, and you need both before the next five sections

Everything so far was discrete. The next five sections involve Gaussian noise, continuous latents, diffusion processes and kernels, and every one of them is a trap if you carry the discrete intuition across unchanged. Here is the boundary.

For a density $f$ on $\mathbb{R}^d$ the differential entropy is $h(X) = -\int f\log_2 f$, and it is not a count of anything:

• It can be negative. A uniform distribution on $[0,\tfrac12]$ has $h = \log_2\tfrac12 = -1$ bit. A Gaussian with variance $\sigma^2$ has $h = \tfrac12\log_2(2\pi e\sigma^2)$, which is 2.047 bits at $\sigma = 1$ and negative once $\sigma < 0.242$.
• It is not a file size. A continuous value has infinitely many bits; there is no lossless code for it.
• It is invariant under translation and not under scaling: for a scalar $a$ acting on $X\in\mathbb{R}^d$, $h(aX) = h(X) + d\log_2|a|$, and for an invertible matrix $A$, $h(AX) = h(X) + \log_2|\det A|$. Rescale your features and your "entropy" moves by a constant that grows with the dimension.

What does connect it to bits is quantisation, and the correction carries the dimension. Discretise $X\in\mathbb{R}^d$ on a cubic grid of side $\Delta$ and the entropy of the discrete variable obeys

each cell has volume $\Delta^d$, not $\Delta$, so the correction is one $\log_2(1/\Delta)$ per dimension.

Worked example (worked_numbers.py §5 and §15). A unit Gaussian in one dimension at $\Delta = 0.1$: $H(X^\Delta) \approx 2.047 + \log_2 10 = 2.047 + 3.322 = 5.369$ bits, and the binned entropy computed exactly is 5.3696. In two dimensions, with independent standard coordinates, $h(X) = \log_2(2\pi e) = 4.094$ bits and the grid entropy at $\Delta = 0.1$ is $4.094 + 2\log_2 10 = 10.738$ bits (exact: 10.7392); the one-dimensional formula would say 7.416 and be wrong by a full $\log_2 10$. Halve $\Delta$ and you add one bit per dimension, forever; the differential entropy is the part that does not depend on your ruler. §6's remark that a continuous reconstruction term becomes a codelength only after fixing a precision is this formula with $d$ equal to the data dimension.

That last sentence is why mutual information survives the passage to continuous variables and entropy does not. Write $I(X;Y) = h(X) + h(Y) - h(X,Y)$ and the $d\log\Delta$ terms cancel: each variable's own dimension and bin width appears once with each sign. So $I$ is well-defined, finite under mild conditions, and invariant under smooth invertible transformations of either variable — which is why it is the right currency for representations (§4) and channels (§7), and why differential entropy on its own almost never is.

Two consequences you need before going on:

1. A deterministic continuous map has infinite mutual information with its input. If $T = g(X)$ with $X$ continuous and $g$ smooth and non-degenerate, then $h(T\mid X) = -\infty$ and $I(X;T) = \infty$. If instead $X$ takes finitely many values, $I(X;T) = H(T) \le H(X)$ is finite and bounded by the input's entropy. These are two different probability models and they give different answers; §4 shows what goes wrong when you quote the first in an experiment built on the second.
2. The Gaussian is the maximum-entropy density at a given variance (and at a given covariance in $d$ dimensions), so for a channel $Y = X + N$ with Gaussian noise of variance $\sigma^2$ and a Gaussian input of variance $P$, $I(X;Y) = \tfrac12\log_2(1 + P/\sigma^2)$ bits per use. In $d$ isotropic dimensions, multiply by $d$. That formula is what §7's diagnostic evaluates for a Gaussian source; the relation that owns the floor there is I-MMSE, $dI_{\mathrm{nats}}/d\,\mathrm{snr} = \tfrac12\,\mathrm{mmse}$ for any finite-power input, and §7 displays it with its factor of one half in the same line as the number it produces.

3.1 The variational bounds, and the ceiling on the most popular one

Estimating $I(X;Y)$ in high dimensions means bounding it with something you can train. Three bounds carry most of modern representation learning, and they are worth stating because each says exactly what its estimate means.

All three are stated in nats, because that is how every implementation computes them: the critic's cross-entropy is a natural log, and the bound has to be read in the same unit.

Barber–Agakov (2003). For any conditional model $q(x\mid y)$,

with both terms in nats: the entropy of $X$ has to be converted from §0's bits before it enters ($H_{\mathrm{nats}} = H_{\mathrm{bits}}\ln 2$), and a bits-valued $H$ added to a nats-valued log-likelihood is not a bound on anything. For continuous $X$ replace $H_{\mathrm{nats}}(X)$ by the differential entropy $h_{\mathrm{nats}}(X)$ and read $q$ as a density; under the usual finiteness conditions the bound is still exact, and it is tight when $q(x\mid y)$ equals the true conditional almost everywhere. Check on the BSC(0.1) of §2 with $X$ a uniform bit and $q$ the true conditional (worked_numbers.py §19): $H_{\mathrm{nats}}(X) = \ln 2 = 0.693$, $\mathbb{E}\ln p(x\mid y) = -H_{\mathrm{nats}}(X\mid Y) = -0.325$, sum $0.368$ nats $= 0.531$ bits, which is $I(X;Y)$ exactly; with $H$ left in bits the sum would be $0.675$, a number with no meaning. A decoder that predicts $X$ from $Y$ gives a lower bound; a better decoder gives a tighter one; the bound needs the entropy of $X$, which you have when $X$ is something you constructed and have to estimate otherwise.

Donsker–Varadhan. $I_{\mathrm{nats}}(X;Y) = \sup_T \big(\mathbb{E}_{p(x,y)}[T] - \ln\mathbb{E}_{p(x)p(y)}[e^{T}]\big)$ over all functions $T$; divide the whole expression by $\ln 2$ for bits. Restrict $T$ to a neural network and you have MINE. Be careful about what is a bound: the population quantity, for any fixed $T$, is a lower bound on $I$; the number a finite-sample, optimised MINE run prints is not, because the supremum is taken over the same sample it is evaluated on and can overshoot the true $I$ — and its variance grows exponentially with the true $I$, which is the pathology §2.2 mentioned.

InfoNCE (van den Oord, Li & Vinyals, 2018). Draw one positive pair and $N-1$ negatives, train a critic to pick the positive, and with $\mathcal{L}_{\mathrm{NCE}}$ the critic's cross-entropy in nats, $I_{\mathrm{nats}}(X;Y) \ge \ln N - \mathcal{L}_{\mathrm{NCE}}$. The two terms must be in the same unit — a $\log_2 N$ ceiling subtracted from a nat-valued loss is the mistake to avoid. This is the objective under contrastive representation learning, and its ceiling is the thing to remember: the bound can never exceed $\ln N$ nats, i.e. $\log_2 N$ bits (Poole et al., 2019). With $N = 4{,}096$ candidates per positive — one positive and 4,095 negatives — the estimate saturates at $\ln 4096 = 8.318$ nats $= 12$ bits regardless of how much information the representation actually carries (a batch of 4,096 negatives has $N = 4{,}097$; the count that enters the ceiling is the total). A contrastive loss that has stopped improving may have hit the batch, not the data.

The pattern across all three is the same one as §2.2: an estimate of mutual information comes with the estimator's name attached, and the estimator's ceiling or its variance is part of the number.

4. The information plane, and the most contested measurement in the field

Saxe et al. (ICLR 2018) showed that the compression phase of the information plane is a property of double-sided saturating nonlinearities read through a binned estimator, not of stochastic gradient descent: it appears with tanh, it does not appear with ReLU or with linear units, and where it appears it has no evident causal link to generalisation. That is the sentence this section starts from, and everything below is an addition to it rather than a rediscovery of it. The claim it answers is the information bottleneck (Tishby, Pereira & Bialek, 1999; Tishby & Zaslavsky, 2015), which proposes that a good representation $T$ of input $X$ for predicting $Y$ solves

keeping what predicts $Y$ and discarding everything else, together with the famous empirical claim (Shwartz-Ziv & Tishby, 2017) that training has two phases — a short fitting phase where $I(T;Y)$ rises, then a long compression phase where $I(T;X)$ falls while accuracy holds. Goldfeld et al. (2019) supplied the other half of the objection: once noise makes $I(X;T)$ finite and estimable, what the estimate tracks is geometric clustering of the representations. Neither is in dispute here. What our experiment adds is not a fourth estimator but two things an estimator cannot give: an input with finite support and exactly known entropy, so that $I(X;T) = H(T)$ is a quantity you count rather than estimate, at every checkpoint; and a noisy readout whose information is an integral over a known mixture, with a standard error, no bins, and both absolute and scale-normalised noise. The first is a lemma about the quantiser. The second is the measurement that generalises, and it comes first.

We built the task — all 4,096 twelve-bit patterns, uniform, so $H(X) = 12$ bits exactly — trained a tanh network and a ReLU network (hidden widths 10, 7, 5, 4, 3) to convergence, and measured. Both learned the task:

 tanh: final epoch 4000  loss 0.00008 nats  train accuracy 1.0000
 relu: final epoch 4000  loss 0.03921 nats  train accuracy 0.9890

4.1 The measurement that generalises: a noisy readout, at two scales

A deterministic layer read at the precision it was computed in — float64 here, NumPy's default — is one readout channel among several, and not the one most consumers of the layer have: a deterministic forward pass at that precision does read it that way, while a lower-precision serving path, a sensor, or the next layer's own tolerance read through something coarser — and whether a given precision actually merges any of the 4,096 inputs depends on the activation values and the format, which is why §4.2 recounts the same weights after a cast to float32 and to float16, and why the measurement below varies the channel explicitly rather than assuming one. So the question with operational content is how much of the input's identity survives a readout through a stated noise channel: $I(X;\,T+N)$ with $N\sim\mathcal N(0,\sigma^2 I)$ added to the activations. Because $X$ is uniform over a known set of 4,096 points, the density of $T+N$ is a 4,096-component Gaussian mixture with known components, and $I(X;T+N)$ is an ordinary Monte Carlo expectation of $\log p(\tilde t\mid x) - \log p(\tilde t)$ — no learned estimator, no bins, none of the $\ln N$ ceiling of §3.1, and an error bar. (The printed label's "exact" refers to the density being known exactly; the integral is Monte Carlo, and the standard error shown is the one that matches the design: every input is present by construction with four independent noise draws each, so the variance of the estimate is the sum of the per-input variances over $n^2 m$ — a stratified error bar — not the pooled spread of all the values, which also counts the between-input spread of the conditional means and is conservative. The notebook prints both and a 16-draw rerun of two cells, and the 4-draw bars hold.) The column headed H(T) exact is the $\sigma\to 0$ limit, and §4.2 says why it is a count rather than an estimate. First the table with the same absolute $\sigma$ on both networks:

Stochastic representation: I(X; T + N) in bits, N ~ N(0, sigma^2 I) on the activations,
computed as an exact Monte Carlo expectation over the known 4,096-component mixture

Three things to read off it. First, at $\sigma = 0.01$ — a hundredth of the activations' RMS — tanh layer 3 carries 4.2 bits rather than its float-precision 12, and layer 5 carries 1.6: whatever the deterministic layer holds, a whisper of noise removes most of it, which is what "tightly clustered" means in bits. Second, this is Goldfeld et al.'s mechanism on our own numbers: once noise makes the quantity finite, what it measures is how far apart the activation vectors sit relative to $\sigma$, and the tanh network's deep layers cluster far more tightly than the ReLU network's. Third, the ordering of the two networks flips between the exact and the noisy columns: exactly, tanh carries more (12 against 7.22 bits at layer 5); at each of the three noise levels, ReLU carries more in every hidden layer but the first.

But the absolute table does not control for scale, and the flip depends on it. The noise has the same absolute standard deviation in both networks while the activation scales differ — RMS 0.978 at the deepest tanh layer against 3.881 at the deepest ReLU one — so $\sigma = 0.1$ is a four-times-smaller relative perturbation of the ReLU representation, and a downstream affine layer can absorb any rescaling of its inputs for free. The comparison that survives a change of scale is the same measurement on standardised activations, $I\big(X;\,T/\mathrm{RMS}(T) + \sigma Z\big)$, with $\sigma$ now a fraction of each layer's own RMS. One layer-wide RMS removes a global scalar and nothing more: for an invertible matrix $A$, $I(X;AT) = I(X;T)$ but in general $I(X;AT + \sigma Z) \ne I(X;T + \sigma Z)$, so both tables are statements about robustness under isotropic Gaussian noise in the coordinates the network produced, not a representation-invariant notion of intrinsic robustness:

Scale-normalised stochastic representation: I(X; T/RMS(T) + N) in bits, N ~ N(0, sigma^2 I),
so sigma is now a fraction of the layer's own RMS activation (same Monte Carlo integration, same seeds).
  act  layer  act.RMS  H(T) exact |   rel sigma=0.01 |    rel sigma=0.1 |    rel sigma=0.5
 
 tanh      1    0.719     12.0000 |   12.000 +/-0.000 |   11.242 +/-0.008 |    5.227 +/-0.016
 
 tanh      2    0.847     12.0000 |    9.617 +/-0.009 |    4.943 +/-0.011 |    1.944 +/-0.009
 
 tanh      3    0.947     12.0000 |    4.268 +/-0.008 |    2.117 +/-0.006 |    1.262 +/-0.005
 
 tanh      4    0.897     12.0000 |    2.479 +/-0.006 |    1.433 +/-0.004 |    1.088 +/-0.003
 
 tanh      5    0.978     12.0000 |    1.606 +/-0.004 |    1.199 +/-0.002 |    1.043 +/-0.002
 
 relu      1    1.013     12.0000 |   11.868 +/-0.003 |    8.407 +/-0.013 |    3.003 +/-0.012
 
 relu      2    1.847     12.0000 |    9.358 +/-0.009 |    4.696 +/-0.009 |    2.211 +/-0.008
 
 relu      3    2.598     11.9609 |    7.664 +/-0.009 |    4.046 +/-0.008 |    1.835 +/-0.008
 
 relu      4    3.396     11.9609 |    7.226 +/-0.008 |    3.951 +/-0.008 |    1.683 +/-0.008
 
 relu      5    3.881      7.2237 |    4.580 +/-0.006 |    2.696 +/-0.006 |    1.270 +/-0.007

Now the flip can be attributed. In the three deepest layers it survives normalisation, so it is shape and not scale: at a relative $\sigma$ of 0.01 the ReLU network carries 7.66, 7.23 and 4.58 bits in layers 3, 4 and 5 against the tanh network's 4.27, 2.48 and 1.61 — and the ReLU figures fell when scale was removed (5.61 → 4.58 at layer 5, because its RMS of 3.881 had made the absolute $\sigma = 0.01$ a quarter-percent perturbation), while the tanh figures barely moved. The ReLU layers keep more information under matched relative noise because their activation vectors are spread more evenly relative to their own scale — the geometry the effective-rank column of §4.2 hints at for layers 3 and 4 (1.31 and 1.54 for ReLU against 1.05 and 1.03 for tanh), while at layer 5 both networks have an effective rank near 1 and the difference is how far apart the values sit along that one direction (2,271 distinct levels of a single live rectifier against 4,096 vectors packed into a cloud a hundredth of their RMS wide). In layer 2 the flip was scale: absolutely, ReLU led (10.25 against 9.34 bits at $\sigma = 0.01$); relatively, tanh leads at the two finer levels (9.62 against 9.36, and 4.94 against 4.70 at $\sigma = 0.1$), because layer 2's ReLU RMS of 1.847 against tanh's 0.847 had bought it a factor of two in relative noise — and at the coarsest relative level, 0.5, ReLU edges ahead again (2.21 against 1.94), so layer 2 is the one place where the ordering genuinely depends on the noise level. Layer 1 is tanh's under both. So the honest version of "ReLU is the more robust representation" is narrower than the absolute table suggested and still true where it matters: in the layers where the tanh network has collapsed onto one direction, ReLU carries more about the input through noise at any scale we measured; one layer up, the ordering is a scale artefact.

What the two tables separate is two questions that the single word "robust" runs together: which representation keeps more information at the scale the network produced (the absolute table), and which has the more information-preserving geometry once scale is factored out (the relative one). Where they disagree, the disagreement is scale, not shape. Both are measured under two controlled channels — absolute and RMS-normalised isotropic Gaussian noise — and the orderings are findings about those channels on these two networks, not an architectural ranking independent of readout noise or parametrisation. Three noise levels are three points, not a curve, and we claim neither ordering for any $\sigma$ between or beyond them. The notebook draws all of it on one figure — the exact count, the binned value at three bin counts, and the noisy readout at three absolute and three relative levels, per layer and per network — so that the three rulers can be seen disagreeing on the same axes.

4.2 The lemma under it: on a finite input, $I(X;T) = H(T)$, and you can count it

$X$ has finite support and $T$ is a deterministic function of $X$, so by §3, $I(X;T) = H(T) \le H(X) = 12$ bits. There is no infinity anywhere in this setup; the familiar claim that "$I(T;X)$ is infinite for a deterministic network" is true for a continuous input and false for ours. And the exact value needs no estimator: count the distinct activation vectors over the 4,096 inputs and take the entropy of their empirical distribution. The notebook does this for every layer at the end of training, beside the 30-bin value, the effective rank of the layer's activation covariance (the exponential of the entropy of its normalised eigenvalues) and the fraction of units that are constant on the whole input set:

Exact I(X;T) = H(T) at float64 precision (the dtype the network was trained and read in), final epoch, vs the 30-bin value
  act  layer  width  distinct  H(T) exact   30-bin  eff.rank  const units
 
 tanh      1     10      4096     12.0000  11.9893     2.441          0%
 tanh      2      7      4096     12.0000   8.4147     1.186          0%
 tanh      3      5      4096     12.0000   3.6580     1.053          0%
 tanh      4      4      4096     12.0000   2.0854     1.030          0%
 tanh      5      3      4096     12.0000   1.5376     1.026          0%
 relu      1     10      4096     12.0000   8.3856     1.717          0%
 relu      2      7      4096     12.0000   5.6741     1.572          0%
 
 relu      3      5      4065     11.9609   4.9885     1.313         40%
 relu      4      4      4065     11.9609   5.1328     1.537         25%
 
 relu      5      3      2271      7.2237   3.3971     1.000         67%

Every tanh layer carries all 12 bits about the input at the end of training — at float64 precision, the dtype the notebook trains and reads in, which is the quantiser this sentence depends on, and §4.1 has already shown how little of it survives a coarser one. Read it as a statement about a readout channel: a downstream layer that reads these activations at the same float64 precision can in principle recover the input; what a float32 or a float16 consumer of the layer sees is a different map, and the next table counts it. Layer 3 — whose binned value fell from 11.85 to 3.66, the "8.19 bits of compression" of §4.3 — maps every input to its own activation vector. Its effective rank of 1.05 across five units says what actually happened: the activations collapsed onto essentially one direction, tightly clustered, so a 30-level quantiser merges most of them. The ReLU network is the one that genuinely loses information, and every loss is a collision — two inputs sent to the same activation vector: layer 5, with 67% dead units and a single live rectifier, maps the 4,096 inputs onto 2,271 vectors, $H(T) = 7.22$ bits, and layers 3 and 4 each merge a handful. Which of those collisions the dead units explain is a separate question, and the per-checkpoint table answers it.

The precision is part of the measurement, so the notebook varies it. Every array in the notebook is float64 — NumPy's default; nothing in it is float32 — so "distinct" above means distinct as float64. A consumer that reads the layer at lower precision reads a different map, and the exact count has to be redone for that map. The notebook casts the same float64 activations — same weights, same forward pass — to float32 and to float16 before counting. These are readout casts, not networks trained at lower precision, which would move the weights as well as the readout:

Readout-precision ablation: distinct activation vectors and H(T) [bits] when the float64 activations
are CAST to a narrower dtype before counting (same trained weights; the cast is at the readout only).
  act  layer |            float64 |       float32 cast |       float16 cast
 tanh      1 |   4096    12.0000 |   4096    12.0000 |   4096    12.0000
 tanh      2 |   4096    12.0000 |   4096    12.0000 |   4095    11.9995
 tanh      3 |   4096    12.0000 |   4096    12.0000 |   2990    11.1031
 tanh      4 |   4096    12.0000 |   4096    12.0000 |   1234     8.1140
 tanh      5 |   4096    12.0000 |   4090    11.9971 |    445     4.2910
 
 relu      1 |   4096    12.0000 |   4096    12.0000 |   4095    11.9995
 
 relu      2 |   4096    12.0000 |   4096    12.0000 |   3630    11.7501
 relu      3 |   4065    11.9609 |   4065    11.9609 |   3565    11.6983
 relu      4 |   4065    11.9609 |   4065    11.9609 |   3154    11.4652
 relu      5 |   2271     7.2237 |   2271     7.2237 |   1473     6.7696

The float32 cast merges 0, 0, 0, 0, 6 of the 4,096 inputs in tanh layers 1–5, and the float16 cast leaves 4,096, 4,095, 2,990, 1,234, 445 distinct vectors in those layers — 4.29 bits at the deepest one against the 12.0000 that float64 reports, the layer whose activations §4.1 found packed a hundredth of their RMS apart. The ReLU network's layer 5 goes from 2,271 distinct vectors at float64 to 2,271 at float32 and 1,473 at float16 (6.77 bits), and its layer 3 from 4,065 to 4,065 and 3,565. For a deterministic map on a finite input set, $I(X;T) = H(T)$ is exact at every precision; what the cast changes is the map whose entropy is being counted. So the 12.0000 of the first table is a float64 number and should be quoted as one, and the operational reading of §4.1 — a readout coarser than the one the layer was computed in loses what a tightly clustered layer holds — has a dtype column now as well as a noise column.

And the exact counts are a property of this run. The notebook prints the environment it ran under — Python 3.14.5, NumPy 2.5.3, macOS on Accelerate BLAS, float64 — fixes its seeds, and within that environment reproduces every number above on re-execution. An independent re-execution on Linux (Python 3.13.5, NumPy 2.3.5), reported to us during review, reproduced every closed-form check in this notebook and every number in the second one, and moved the trained-network numbers: the ReLU network reached a final accuracy of 0.9946 rather than 0.9890, with 4,017 distinct vectors at layer 3 rather than 4,065 and 2,252 at layer 5 rather than 2,271, from the same seed. Nothing is wrong with either run. Four thousand epochs of full-batch gradient descent amplify last-bit differences between two BLAS implementations into different weights, and a count of exact floating-point collisions is the most sensitive statistic one could build on those weights. What is portable is the set of findings — every tanh layer at 4,096 distinct vectors at float64, the ReLU network's deepest layer dead from initialisation and gaining exact information through training, the orderings under noise — and what is not is the third digit of a collision count. The trained weights ship beside the notebooks as the reference file nb00_trained_params.npz, so the representation these tables describe can be read without retraining it; a forward pass from saved weights does not carry the trajectory's sensitivity. A re-execution never writes that file: it saves its own weights as nb00_trained_params.local.npz, checks the reference against its released SHA-256 (nb00_trained_params.sha256), reports whether its own weights match, and if they do not, prints the reference weights' collision counts beside its own. If the reference is absent it says so and continues on the local file — a locally trained file promoted to "reference" by a notebook run would make a later match self-confirmation rather than reproduction — and the reference is produced only by the explicit release step make_reference_weights.py --write.

A final-epoch table cannot show that the exact plane was static — the compression claim is about a trajectory — so the notebook records the exact collision count and the dead-unit fraction at every probe epoch of the same training pass that produced the final-epoch table, from the same forward pass as the binned value, and asserts that the final probe equals a fresh forward pass of the final weights. Seven of those probes:

Exact H(T) [bits] of the three deepest layers over training (the per-checkpoint collision count),
beside the dead-unit fraction of the same layer. 12.0000 = no two inputs collide.
 
tanh
 epoch     acc |  L3 H(T)  dead |  L4 H(T)  dead |  L5 H(T)  dead | L5 distinct
     1  0.6267 |  12.0000    0% |  12.0000    0% |  12.0000    0% |        4096
     2  0.6389 |  12.0000    0% |  12.0000    0% |  12.0000    0% |        4096
     5  0.7156 |  12.0000    0% |  12.0000    0% |  12.0000    0% |        4096
    10  0.8547 |  12.0000    0% |  12.0000    0% |  12.0000    0% |        4096
    20  0.8589 |  12.0000    0% |  12.0000    0% |  12.0000    0% |        4096
    50  0.8706 |  12.0000    0% |  12.0000    0% |  12.0000    0% |        4096
  4000  1.0000 |  12.0000    0% |  12.0000    0% |  12.0000    0% |        4096
 
relu
 epoch     acc |  L3 H(T)  dead |  L4 H(T)  dead |  L5 H(T)  dead | L5 distinct
     1  0.4880 |  11.7914    0% |  11.7554   25% |   0.7395   67% |         227
     2  0.4871 |  11.7914    0% |  11.7554   25% |   0.7232   67% |         222
     5  0.4883 |  11.7900    0% |  11.7494   25% |   0.6710   67% |         206
    10  0.4910 |  11.7900    0% |  11.7413   25% |   0.5991   67% |         184
    20  0.4919 |  11.7900    0% |  11.7347   25% |   0.5305   67% |         163
    50  0.4932 |  11.7927    0% |  11.6916   25% |   0.3343   67% |         103
  4000  0.9890 |  11.9609   40% |  11.9609   25% |   7.2237   67% |        2271
  layer 3: first collision at probe epoch 1, first dead unit at probe epoch 3644
  layer 5: first collision at probe epoch 1, first dead unit at probe epoch 1

The tanh rows are the trajectory the final-epoch table implied: 4,096 distinct vectors at all seven probes, 12.0000 bits in every deep layer at every checkpoint we measured. At the measured checkpoints the exact information plane of the tanh network is a point that does not move — seven probes do not rule out an excursion between them, but a compression phase invisible at epochs 1, 2, 5, 10, 20, 50 and 4,000 is not the one the two-phase story describes — and the binned curve's steep descent (§4.3) is entirely the ruler. The ReLU rows say something the final-epoch table hid, and it runs against the compression story: layer 5's 67% dead units are dead at the first probe epoch, and the layer's exact information rises during training, from 0.74 bits (227 distinct vectors) at epoch 1 to 7.22 bits (2,271) at epoch 4,000. Two of its three units never fire on any input, and training spent 4,000 epochs teaching the one live rectifier to separate inputs it had initially merged. Layers 3 and 4 collided a handful of inputs from epoch 1 and finished with fewer collisions, not more.

Now attribute the loss correctly, because the table refutes the tidy version. What removes exact information is a collision in the representation map, and dead units are one way to produce collisions, not the only one. Layer 3 is the counter-example in our own data: it collides inputs at epoch 1 (11.79 bits) with 0% dead units, and its first globally dead unit appears at probe epoch 3,644, by which time its collision count has fallen. A rectifier does not have to be dead on every input to merge two of them; it only has to be off on the inputs where they differ — rectification over a subset of the input set — and two float64 values can also coincide outright. So the dead-unit fraction is a diagnostic that explains some collisions (layer 5) and is silent on others (layer 3); the collision count is the measurement. In this experiment the exact $I(X;T)$ of the ReLU network goes up wherever it moves at all, and every exact loss is a collision present from initialisation.

The one measurement in the section that needs no estimator and no quantiser at all:

Permanently dead ReLU units by layer: ['0%', '0%', '40%', '25%', '67%']

67% of the deepest ReLU layer's units output zero for all 4,096 inputs. A unit that is constant on the whole input set carries exactly zero bits about any of it — that is a statement about $H$ of a constant, not an estimate. What it does not say is how many bits the layer lost: a layer with two dead units and one live unit that takes a distinct value on every input still carries all twelve (one neuron computing $\mathrm{ReLU}(\sum_i 2^{i-1}X_i)$ does exactly that), so the dead-unit count neither proves nor quantifies a loss. The loss is the collision count above; the dead-unit fraction is one place to look for it — the place that explains layer 5 and says nothing about layer 3. Together with the effective rank, these are the two numbers in the section whose values do not depend on a choice we made.

4.3 What the binned estimator measures

So every number below 12 in the next table is a measurement of a different channel, and an exact one. For a deterministic $T = f(X)$ and a quantiser $Q$, $I(X;Q(T)) = H(Q(T)) \le I(X;T) = H(T)$, and because the notebook enumerates the whole input set the binned value is that quantity computed exactly for the 30-level readout — no sampling error, no estimator. (Uniform $X$ on four values, $T = X$, $Q(T) = \lfloor T/2\rfloor$: $I(X;T) = 2$ bits and $I(X;Q(T)) = 1$ bit, both exact, two channels.) It is a valid information measurement of how much of the input's identity survives a 30-level quantisation of each unit — a statement about how tightly the activations cluster, which is a geometric property of the layer — and it is not a measurement of the float64 layer's $I(X;T)$. That is precisely the Saxe et al. and Goldfeld et al. objection, and with it said, here is what the 30-bin quantiser reports:

Per-layer I(T;X): value at its peak, and at the end of training (30 bins)
  act  layer  peak I(T;X)     final    change
 tanh      1      12.0000   11.9893   -0.0107
 tanh      2      12.0000    8.4147   -3.5853
 tanh      3      11.8529    3.6580   -8.1948
 tanh      4       8.9857    2.0854   -6.9003
 tanh      5       7.9751    1.5376   -6.4375
 relu      1      11.7029    8.3856   -3.3173
 relu      2       8.0739    5.6741   -2.3998
 relu      3       8.0004    4.9885   -3.0119
 relu      4       5.2580    5.1328   -0.1252
 relu      5       3.4599    3.3971   -0.0628

And the same final-epoch measurement at different bin counts, which is the §2.2 discipline applied to our own numbers:

Final-epoch I(T;X) of the deepest hidden layer, as a function of bin count
  bins       tanh       relu
     8     1.2124     2.1748
    30     1.5376     3.3971
   120     1.7661     4.5322
   256     2.3442     5.1362

The tanh number roughly doubles from 8 bins to 256 and the ReLU number more than doubles, and they would keep climbing toward 12 as the bins shrink, because that is where the exact quantity sits. So the bit values in the first table are findings about their quantiser and not about the layer: each is the exact $I(X;Q(T))$ of its own readout, a different $Q$ gives a different exact number, and none of them is $I(X;T)$. What survives every bin count is qualitative: the tanh network's binned quantity falls steeply in its later layers during training and the ReLU network's barely moves. That is the Saxe et al. point made concrete — saturating units cluster their activations as they saturate; ReLU units do not — and it is a finding about the nonlinearity, not about learning in general.

4.4 The status of the claim

One controlled notebook is an illustration, and the field's position on the underlying claim should be stated rather than implied. The fitting-then-compression story is not the current default explanation of deep training. Outside saturating nonlinearities the compression phase is generally not observed (Saxe et al.); with noise injected so that $I(X;T)$ is finite and estimable, the measured quantity tracks geometric clustering (Goldfeld et al.); and the live explanatory alternatives — neural collapse of the last-layer features (Papyan, Han & Donoho, 2020), grokking (Power et al., 2022), and the geometric accounts of representation learning — do not require an information-theoretic phase at all. The information bottleneck as an objective is alive and useful, and its variational form (Deep VIB, Alemi et al., 2016) is the bridge to §6. The information bottleneck as a theory of SGD is contested, and our experiment does not settle it.

The legs of this experiment — exact at float64, binned, cast to float32 and float16, and noisy at two scales, all on the same trained networks — are four observation channels on one map, each reporting the information of its own channel, exactly where the channel is a deterministic quantiser and with an error bar where it is noise; what they separate is "the representation compressed" from "our ruler got coarser". For the tanh network it was the ruler, binned or noisy. For the ReLU network the only exact losses were collisions present from initialisation, partly recovered by training. On the question the field actually argues about — whether SGD has a compression phase on image models — a 12-bit input, five layers and two nonlinearities are silent, and training to epoch 4,000 does not make the counting argument say more than it says. The three results travel as takeaway 5's three items, and no further.

5. Rate–distortion: the floor under every compression scheme

Rate–distortion theory asks the question every lossy system actually faces: given that I will spend only $R$ bits per source symbol, what is the least distortion I can possibly achieve? The answer is a curve, and the theorem has assumptions worth stating once:

for an i.i.d. source $X$, a per-symbol distortion $d$, and rate in bits per source symbol. The theorem says two things. Converse: no code at any blocklength achieves distortion $D$ at rate below $R(D)$. Achievability: codes exist that approach $R(D)$ — in the limit of long blocks, with vector codes that look nothing like a scalar quantiser. Both halves matter below.

The two closed forms, with the domains that are usually left implicit:

• Bernoulli($p$) source, Hamming distortion: $R(D) = H_2(p) - H_2(D)$ for $0 \le D \le \min(p, 1-p)$, and $R(D) = 0$ beyond that — once you tolerate more error than always guessing the majority symbol produces, you need no bits at all.
• Gaussian source of variance $\sigma^2$, squared error: $R(D) = \max\{0, \tfrac12\log_2(\sigma^2/D)\}$ — zero bits once $D \ge \sigma^2$, because sending the mean already achieves distortion $\sigma^2$.

We computed both with Blahut–Arimoto (Blahut, 1972; Arimoto, 1972) and checked against the closed forms. Binary, $p = 0.3$:

       D   R measured    H(p)-H(D) exact    abs err
  0.2689     0.041349           0.041349   0.00e+00
  0.1192     0.354226           0.354226   2.22e-16
  0.0474     0.605931           0.605931   0.00e+00
  0.0000     0.881284           0.881284   1.11e-16

Unit-variance Gaussian, $0 < D \le 1$:

       D   R measured   0.5*log2(1/D)      diff
  0.4688      0.54650         0.54651   -0.00001
  0.1114      1.58294         1.58295   -0.00001
  0.0129      3.13760         3.13761   -0.00001
 
mean |difference| = 0.00007 bits over 25 points

The residual is the 241-point grid, and it grows at low distortion where the code needs finer resolution than the grid has — a discretisation artefact, correctly identified. (As in §0: this verifies the implementation against a closed form. It says nothing about any other experiment.)

5.1 How close does a real codec get, and why "not achievable" needs a regime

The bound is approached by long vector codes, not by the quantiser you would actually write, and the theory predicts the gap for that quantiser. In the high-resolution limit — fine quantisation of a smooth density — an entropy-coded uniform scalar quantiser sits $\tfrac12\log_2\tfrac{2\pi e}{12} \approx 0.2546$ bits per sample above the Shannon bound (Gish & Pierce, 1968): the space-filling loss of a cubic lattice against an optimal one, the famous quarter-bit. We built the quantiser half of a codec on a correlated Gaussian source ($d=16$, AR(1) with $\phi=0.9$, 200,000 samples) — a KLT, then a uniform scalar quantiser — and charged ourselves the sum of the empirical marginal entropies of the quantiser indices, $R_{\text{est}} = \tfrac1d\sum_j H(\hat Z_j)$ bits per sample. No bitstream is written: that column is the rate an ideal entropy coder of those marginals approaches from above, with a finite-length overhead on top — an estimated entropy-coded rate, not a measured file size. Its gap against the bound at matched distortion:

   step    R est.         D   R bound at same D  gap [bits/sample]
        (R est. = sum of empirical marginal entropies of the quantiser indices; no entropy coder is run)
   2.00    0.4889   0.16359              0.3506             0.1384
   1.00    1.1185   0.06934              0.8069             0.3116
   0.50    2.0374   0.02075              1.6727             0.3647
   0.25    2.9572   0.00521              2.6692             0.2880
   0.12    3.9909   0.00120              3.7278             0.2631
 
theoretical space-filling gap (high-resolution limit) = 0.2546 bits/sample
  rates >= 2.5 bits/sample: mean gap = 0.2850, spread 0.0203 over the 4 operating points (predicted 0.2546)
    — the spread is variation across step sizes, not a confidence interval; the block repeat below is the uncertainty
  rates <  1.5 bits/sample: gap = 0.2199  — the high-resolution approximation does not hold here
 
Block repeat at step 0.12 (five disjoint 40,000-sample blocks): gap = 0.2621 +/- 0.0011 bits/sample
  (mean +/- SD ACROSS the five 40,000-sample block estimates — an across-block dispersion, not a confidence
   interval for the 200,000-sample value 0.2631; per block 0.2615, 0.2639, 0.2619, 0.2623, 0.2609;
   a 40,000-sample plug-in entropy reads slightly low, which is the direction of the small difference)

At the highest rate measured the gap is 0.2631 against a prediction of 0.2546 — an error of 0.0084 bits, still converging downward. The 0.0203 printed beside the high-rate mean is the spread across four operating points — variation over step sizes, not a confidence interval — and the dispersion of one operating point over data is the block repeat: 0.2621 ± 0.0011 bits per sample, the mean and standard deviation across five disjoint 40,000-sample block estimates at the finest step — a repetition over data rather than over settings, and an across-block spread, not a confidence interval for the 200,000-sample value 0.2631. The low-rate points are not evidence against the quarter-bit; they are outside the regime the formula describes, and at low rate a scalar quantiser can sit closer to the bound than the asymptotic gap, as the 0.1384 row shows. The general lesson: when a prediction comes with a regime, test it inside the regime and say so. Reporting the mean over all rates would have blurred a clean confirmation into a vague one.

5.2 Reverse water-filling, and exactly when a threshold is the answer

For a Gaussian vector source with covariance eigenvalues $\lambda_1 \ge \lambda_2 \ge \dots$, the rate–distortion function is reverse water-filling: choose a water level $\theta$, set each component's distortion to $D_i = \min(\theta, \lambda_i)$, and spend $\tfrac12\log_2(\lambda_i/\theta)$ bits on components above the level and nothing on components below it. Our source's spectrum:

Eigenvalue spectrum of the source:
  9.927  2.949  1.128  0.568  0.341  0.229  0.167  0.129  ...
  top 4 of 16 components hold 91.1% of the variance

That is the entire reason transform coding works: spend bits where the variance is. And it is worth being precise about why the optimum is a threshold, because the shape is easily promoted into a law about every budget. The problem is $\min\sum_i \tfrac12\log(\lambda_i/D_i)$ subject to $\sum_i D_i \le D$ and $0 \le D_i \le \lambda_i$. The objective is separable and convex, each term's marginal value $\partial/\partial D_i = -1/(2D_i)$ is unbounded as $D_i \to 0$, and the constraint $D_i \le \lambda_i$ is a hard cap. The KKT conditions then equalise marginal value across the components that are interior — $D_i = \theta$ — and clamp the rest at their cap, which is the threshold. Change any of those ingredients and the threshold disappears: a proportional-fairness allocation ($\max\sum_i w_i\log x_i$ with $\sum_i x_i = B$) is interior, with every $x_i$ proportional to its weight; an entropy-regularised allocation is a softmax; any problem whose non-negativity constraints never bind has a smooth interior optimum. Water-filling is L1 for parallel Gaussian channels and for Gaussian sources under squared error. "If your allocation is smoothly proportional you have solved the wrong problem" is false, and would mislead anyone tuning a sampler, a learning-rate schedule or a mixture weight. The right instruction is: write down your constraint set, and check whether it is one of the two that produce a threshold.

With that fence, the callback. "How many Gaussians does a scene need?" — the question our Gaussian-splatting piece asked — resembles "where do you put the water level?" The resemblance is L3: a splat count is a model-complexity budget, and it becomes a rate only once you fix a representation and an entropy code for the splats, which the splatting literature does in its compression papers and the rendering papers do not. We are deliberately not attaching a published compression number to the analogy, because the specific results in that area are not ones we have verified.

6. The ELBO splits exactly into a distortion and an upper bound on rate — and what that does and does not say about your collapsed latent

Write the negative ELBO for a VAE with encoder $q_\phi(z\mid x)$, decoder $p_\theta(x\mid z)$ and prior $p(z)$:

The reconstruction term is distortion. The KL term is rate, and the claim that it is a rate has a proof, which is short enough to show.

Proof sketch (the aggregate-posterior decomposition). Average the KL term over the data distribution $p_{\mathcal D}(x)$ and let $q(z) = \mathbb{E}_{p_{\mathcal D}}[q(z\mid x)]$ be the aggregate posterior. Then

by adding and subtracting $\log q(z)$ inside the expectation; the first term is the mutual information between input and latent under the encoder, the second is non-negative. $\square$ So the average rate is an upper bound on the information the latent carries about the input, exact when the aggregate posterior matches the prior (Hoffman & Johnson, 2016; Alemi et al., Fixing a Broken ELBO, 2018). Averaged over data,

and that has the form of a rate–distortion Lagrangian with the price of a bit fixed at 1, with $R$ an upper bound on the rate rather than Shannon's minimum over all channels — L2, a variational correspondence with every term defined, and the one place in this piece where the "it's rate–distortion" reading is earned rather than asserted, at exactly that width. Alemi et al. are the source of the level-set observation that follows; it is not a discovery of this series, and the $\beta$-VAE literature had already treated the KL coefficient as a rate–distortion price.

Posterior collapse is three different failures, and the decomposition is what keeps them apart.

• (a) A dimension's per-dimension KL, $D_{\mathrm{KL}}(q(z_j\mid x)\Vert p(z_j))$ averaged over data, is near zero: it spends no rate.
• (b) Its activity, $A_j = \mathrm{Var}_x\big[\mathbb{E}_{q(z_j\mid x)} z_j\big]$ (Burda, Grosse & Salakhutdinov, 2016, active when $A_j > 0.01$), is near zero: its posterior mean does not move with the input.
• (c) The decoder ignores $z$: $p_\theta(x\mid z)$ is insensitive to it.

These are not the same event, and (a) is not a count of active units. The worked point: a dimension with $q(z_j\mid x) = \mathcal N(0, 0.25)$ for every $x$ carries no information about the input — $I(X;Z_j) = 0$, $A_j = 0$ — yet pays $D_{\mathrm{KL}}(\mathcal N(0,0.25)\Vert\mathcal N(0,1)) = \tfrac12(0.25 - 1 - \ln 0.25) = 0.318$ nats of rate, and a KL threshold would call it active. Rate can be spent on a posterior that is merely narrower than the prior; information requires the posterior to depend on $x$, which is exactly the $I_q(X;Z)$ term of the decomposition, with the 0.318 nats landing in the aggregate-posterior gap. The converse failure exists too, and it is the activity statistic that misses it: let $X$ be a fair bit with $q(z_j\mid X{=}0) = \mathcal N(0, 0.25)$ and $q(z_j\mid X{=}1) = \mathcal N(0, 1)$. The posterior mean is zero for both inputs, so $A_j = 0$ and the 0.01 threshold calls the dimension inactive, yet $I(X;Z_j) = 0.1338$ bits (worked_numbers.py §26): the two conditionals differ, $|z_j|$ is informative about $X$, and a decoder can use it. Activity is a statement about the posterior mean; information can ride the posterior variance. So neither the KL column nor the activity column characterises the latent's mutual information — the first over-counts a narrowed posterior, the second misses a variance-coded one — which is why Procedure 3 asks for all three. Changing $\beta$ moves the operating point on the frontier below; it does not say which of (a), (b) or (c) you have.

One unit caveat before the numbers. For a continuous $x$ the distortion is a negative log density and can be negative; it becomes a codelength only after you fix a precision $\Delta$ per dimension, which adds $-d\log_2\Delta$ (§3). The negative distortions in the tables below are that: densities, not bits of a file.

6.1 What the level set shows

Three encoders on a linear-Gaussian task ($x = z + \text{noise}$, $z\sim\mathcal N(0,1)$, $q(z\mid x) = \mathcal N(ax, s^2)$):

                     encoder  rate R [bits]  distortion D [bits]   -ELBO = R+D
       collapsed (a=0, s2=1)         0.0000              14.8131       14.8131
    balanced (a=0.6, s2=0.3)         0.6492               3.0984        3.7476
 high-rate (a=0.95, s2=0.05)         2.1918               0.0453        2.2371

These have different ELBOs; they are the axes. The notebook then searches the encoder family for a level set of $R+D$:

Encoders whose -ELBO equals 2.3131 bits to within 0.002:
  count                : 725
  rate R spans         : 1.2578 to 2.3398 bits  (a spread of 1.0820 bits)
  distortion D spans   : -0.0251 to 1.0571 bits
Exact-posterior minimum: R=1.729716, D=0.386132, -ELBO=2.115847 bits

725 encoders lie within 0.002 bits of one objective value, with rates from 1.26 to 2.34 bits. Here is exactly what that establishes: a scalar ELBO hides different rate–distortion splits. Two models reporting the same bound can be carrying a bit more or a bit less in the latent, and you cannot tell from the number. Here is what it does not establish, and it is tempting to pretend otherwise: these 725 encoders are not minima, not stationary points, and not where an optimiser ends up. For a smooth objective in two parameters, a curve of near-equal values is unremarkable; the true minimum (the exact posterior, $a = 10/11$, $s^2 = 1/11$) sits 0.2 bits below the band and the gradient on the band is non-zero. The band has no zero-rate member; the collapsed encoder is 12.7 bits worse. The level set is a statement about score ambiguity, nothing more.

The $\beta$-VAE sweep in the same family, minimising $D + \beta R$:

beta=15.85 lands at R=0.191 bits  (latent nearly unused)
beta=0.06 lands at R=3.731 bits  (latent carrying real information)

The curve $\beta$ traces is not Shannon's $R(D)$. Shannon's function minimises $I(X;\hat X)$ over every conditional with $\mathbb{E}[d]\le D$. A $\beta$-sweep traces the achievable curve of one encoder family and one decoder family: changing $\beta$ moves the operating point of that family, and the point lands on $R(D)$ only if the optimum lies inside the family. The running example is the case where it does — at exactly one $\beta$. The reproduction the decoder sees is $z$ and the distortion $-\log\mathcal N(x; z, 1/10)$ is affine in $(x-z)^2$, so Shannon's curve is the Gaussian one, $R(\delta) = \tfrac12\log_2(1.1/\delta)$ at squared error $\delta$; the family's optimum of $D + \beta R$ is $a = 10/(\beta+10)$, $s^2 = \beta/(\beta+10)$ (worked_numbers.py §22):

At $\beta = 1$ the family holds the exact posterior, $a = 10/11$, $s^2 = 1/11$, the aggregate posterior is $\mathcal N(0,1) = p(z)$, and $R_{\text{VAE}} = I_q = R(\delta)$: the curve touches $R(D)$. Everywhere else it sits above it, by the aggregate-posterior gap plus a smaller term from the family's $\beta$-optimum not being the channel that minimises $I$ at that $\delta$. Now restrict the decoder so that the optimum is not in the family — give it mean $z/2$ instead of $z$, so it cannot use the latent at full gain — and the best encoder at $\beta = 1$ ($a = 1.4286$, $s^2 = 0.2857$) lands at $\delta = 0.161$ with $R_{\text{VAE}} = 2.008$ bits against Shannon's $1.385$ at that distortion: a gap of $0.62$ bits, and a $-\mathrm{ELBO}$ of $2.836$ bits against $h(x) = 2.116$. That picture has the right inclusion. The family's curve lies inside the region $R(D)$ bounds, it reaches the boundary only where the family contains the optimum, and a $\beta$ sweep explores the family, not the bound.