Posts tagged “fisher-information”

3 posts found

The Information Domain: Entropy as a Codelength, and Which of Your Objectives Are One by Theorem
ai-tutorialstutorialinformation-theory

The Information Domain: Entropy as a Codelength, and Which of Your Objectives Are One by Theorem

A standalone tutorial, not a survey: entropy as a coding quantity with its asymptote attached, the cross-entropy floor under both label-smoothing conventions and the table of which held-out loss owns which floor, the plug-in mutual-information estimator as an uncertain measurement with a report format, discrete versus differential entropy in d dimensions before any VAE or kernel claim, the information plane measured three ways (exact count at float64 with a readout cast to float32 and float16, binned, noisy readout at absolute and scale-normalised noise), the ELBO as an exact split into a distortion and an upper bound on rate, with the three failures called posterior collapse and the gap between a β-VAE's curve and Shannon's R(D), the I-MMSE floor under a diffusion loss with both sides integrated, Kolmogorov–Szegő with its convention and hypotheses displayed and its circulant check kept apart from the Toeplitz limit and from the jitter it rides on, hypothesis-testing exponents (Stein, Chernoff, Sanov kept apart) checked against exact finite-sample errors, the Cover & Thomas and MacKay results with proof sketches, modern consumers from InfoNCE to speculative decoding, six checkpoint exercises, and an appendix that grades every connection to our earlier posts by how much it actually proves.

Oct 8, 2026•158 min read
Bayesian Rendering Explained: When a 3D Model Admits What It Doesn't Know
ai-tutorialstutorialbeginner

Bayesian Rendering Explained: When a 3D Model Admits What It Doesn't Know

A beginner-friendly tour of Bayesian 3D reconstruction: why an ordinary 3D scan gives one confident answer everywhere, how turning geometry into a distribution produces honest confidence maps, and where the 'prior' that fills in unseen regions comes from.

Jul 18, 2026•7 min read
When Gaussians Get Uncertain: Probabilistic 3D Reconstruction
ai-tutorialstutorialgaussian-splatting

When Gaussians Get Uncertain: Probabilistic 3D Reconstruction

3D Gaussian Splatting gives us a single best-fit scene — but no sense of where that geometry is trustworthy. This week we turn point-estimate splats into distributions over geometry, connecting Bayesian rendering, Fisher information, and diffusion priors back to last week's GP framework.

Jul 16, 2026•8 min read