Posts tagged “numpy”

22 posts found

Adam, Muon and Friends Explained: What a Normalised Step Buys, What It Costs, and Which Claims Are Theorems
ai-tutorialstutorialbasics

Adam, Muon and Friends Explained: What a Normalised Step Buys, What It Costs, and Which Claims Are Theorems

Adaptive optimizers for practitioners who use them every day: why Adam's first step is a sign step of length about η√d whatever the gradient's size, why 'Adam moves every coordinate by η' is true for a constant gradient and false in general (3.16η after a quiet stretch, 7.27η in the worst case), why a per-coordinate scale is exactly right for a block-heterogeneous problem and why a constant-step Adam is not monotone even on a noiseless quadratic, what the Reddi counter-example does and does not say, why weight decay under AdamW is an equilibrium norm and under coupled L2 is a layer-dependent accident, what Muon's Newton–Schulz iteration computes and by how much it misses, and how to read a benchmark number. Every number is from a seeded NumPy script that ships with the piece.

Nov 14, 2026•39 min read
Gradient Descent Explained: The Step, the Condition Number, and Why Your Loss Curve Goes Flat
ai-tutorialstutorialbasics

Gradient Descent Explained: The Step, the Condition Number, and Why Your Loss Curve Goes Flat

Gradient descent for practitioners who train models with it: why the step size has a hard ceiling of 2/L and what happens on either side of it, why the condition number is an exponent and not a constant, what the descent lemma actually promises and how a line search collects more, what momentum's square root buys and the one effective-learning-rate fact that is a theorem without qualification, why a flat loss under noise is a floor and not convergence, why the right schedule depends on whether the run is transient-limited or floor-limited, and what the two non-convex pictures — a PL function and a double well — do to every one of those statements. Every number is from a seeded NumPy script that ships with the piece.

Nov 12, 2026•37 min read
The Optimization Domain: Convexity, Rates, and Which of Your Training Claims Are Theorems
ai-tutorialstutorialoptimization

The Optimization Domain: Convexity, Rates, and Which of Your Training Claims Are Theorems

A standalone tutorial, not a survey: an optimization problem as a function and a set and a method as a sequence of oracle queries, what convexity buys and the one pathology it leaves, duality and KKT with the SVM's support vectors named by complementary slackness and the log-partition's conjugate as negative entropy, the rates of gradient descent, heavy ball and Nesterov against their theorems and against Nesterov's worst-case function, mirror descent and natural gradient as steepest descent under a different geometry, Adam as a diagonal preconditioner with the counter-example on which it fails and the block-heterogeneous quadratic on which it wins, Shampoo, SOAP and Muon as steepest descent under the spectral norm with the 2025–26 scoreboard quoted from the neutral benchmarks, the stochastic regime from the Robbins–Monro hand-off through the noise floor, the gradient noise scale, variance reduction, schedules on the noisy quadratic model, warmup and the equilibrium norm that weight decay actually sets, double descent with the bias–variance decomposition exact on the linear model and benign overfitting as the condition under which interpolating is harmless, what a stationary-point guarantee promises on a non-convex function and the lower bounds that say nothing more can be promised, saddles, Polyak–Łojasiewicz, the conservative-gradient theorem under autodiff on a ReLU net, benign landscapes, lazy versus feature learning, the edge of stability, implicit bias and the sharpness that is not a property of the function, deterministic global optimization from branch-and-bound through αBB and Lasserre to the ReLU verifiers where it is load-bearing today, Langevin, Bayesian optimization and quantization as the stochastic and discrete alternatives, minimax as not-descent, seventeen checkpoint exercises, a list of the common misreadings, and an appendix that grades every connection to our earlier posts by how much it actually proves.

Nov 5, 2026•200 min read
Concentration and the Central Limit Theorem Explained: How Many Samples Your Number Needs
ai-tutorialsbasicsprobability-theory

Concentration and the Central Limit Theorem Explained: How Many Samples Your Number Needs

The error bar for practitioners: why an average of independent, identically distributed draws has a Gaussian error of size sigma over root n and how fast that becomes true, why a mean of rare events needs a hundred times more data than a symmetric one, what Chebyshev, Hoeffding and the exact binomial each know, how many test examples resolve a one-point gap (about 12,000 unpaired, a few hundred disagreements paired), why picking the best of a hundred checkpoints adds three points you did not earn, why random vectors in 4,096 dimensions are nearly orthogonal and attention divides by root d, and why 10,000 correlated samples can be worth 50 independent ones — with the batch-means fix.

Oct 31, 2026•56 min read
Probability Theory Explained: Sample Spaces, Random Variables, and the Expectation You Already Optimise
ai-tutorialsbasicsprobability-theory

Probability Theory Explained: Sample Spaces, Random Variables, and the Expectation You Already Optimise

Probability for practitioners who need it for machine learning: why a probability is a function on questions rather than a property of things, why a density can be 40 and is still not a probability, why expectation is linear whether or not anything is independent, why the conditional mean is the best predictor by theorem rather than by training, why cross-entropy's minimiser is calibrated for the same reason, and the seven places — a density mistaken for a probability, base rates, Simpson's paradox, uncorrelated-but-dependent, infinite means, conditioning on a zero-probability event, and 'random' without a measure — where the intuition breaks and the definition does not.

Oct 29, 2026•47 min read
The Probability Domain: Measure, Convergence, and Which of Your Training Claims Are Theorems
ai-tutorialstutorialprobability-theory

The Probability Domain: Measure, Convergence, and Which of Your Training Claims Are Theorems

A standalone tutorial, not a survey: probability as a measure and the σ-algebra as the list of questions you may ask, why that is forced (Vitali) and what it buys (conditioning, filtrations, independence as statements about σ-algebras), a random variable as a measurable function and a density as a Radon–Nikodym derivative with KL's infinities and the GAN support problem falling out, the three integral-limit theorems that license every gradient of an expectation and the one loss where they fail, conditional expectation as the L2 projection that makes the regression function optimal by theorem and the Borel–Kolmogorov paradox computed both ways, the four modes of convergence with the typewriter and escaping-mass counter-examples and the law of the iterated logarithm as the envelope a running mean cannot escape, the central limit theorem as a Fourier theorem with its Berry–Esseen rate and the infinite-width network as a CLT, concentration of measure from Markov to Gaussian Lipschitz with the geometry of d = 4096 and the eval-set error bar table, martingales under SGD, peeking and anytime-valid tests, ergodic theorems under MCMC, Langevin and diffusion as SDE reversal, minimax risk and the price of a density in high dimension, conditional independence as architecture and de Finetti as the licence for Bayesian inference, eleven checkpoint and three diagnostic exercises, and an appendix that grades every connection to our earlier posts by how much it actually proves.

Oct 22, 2026•328 min read
The Information Bottleneck Explained: What a Representation Is For
ai-tutorialsbasicsinformation-bottleneck

The Information Bottleneck Explained: What a Representation Is For

The information bottleneck for practitioners: what it means to keep only the bits that predict, why the binned compression phase in our experiment tracks the activation function rather than learning, why every information-plane number depends on its binning while the exact one for a finite input does not, and how the same equation — not the same explanation, and with β multiplying the opposite term — shows up in a VAE's collapsed latent.

Oct 16, 2026•67 min read
Information Theory Explained: Entropy, Cross-Entropy, and the Loss You Already Use
ai-tutorialsbasicsinformation-theory

Information Theory Explained: Entropy, Cross-Entropy, and the Loss You Already Use

Information theory for practitioners who need it for machine learning: what a bit actually measures, why minimising cross-entropy is minimising KL divergence, the floor label smoothing installs under your loss (and which smoothing convention your framework uses), and why a mutual-information estimate is a measurement with a sign that depends on the regime, not a number.

Oct 15, 2026•52 min read
The Information Domain: Entropy as a Codelength, and Which of Your Objectives Are One by Theorem
ai-tutorialstutorialinformation-theory

The Information Domain: Entropy as a Codelength, and Which of Your Objectives Are One by Theorem

A standalone tutorial, not a survey: entropy as a coding quantity with its asymptote attached, the cross-entropy floor under both label-smoothing conventions and the table of which held-out loss owns which floor, the plug-in mutual-information estimator as an uncertain measurement with a report format, discrete versus differential entropy in d dimensions before any VAE or kernel claim, the information plane measured three ways (exact count at float64 with a readout cast to float32 and float16, binned, noisy readout at absolute and scale-normalised noise), the ELBO as an exact split into a distortion and an upper bound on rate, with the three failures called posterior collapse and the gap between a β-VAE's curve and Shannon's R(D), the I-MMSE floor under a diffusion loss with both sides integrated, Kolmogorov–Szegő with its convention and hypotheses displayed and its circulant check kept apart from the Toeplitz limit and from the jitter it rides on, hypothesis-testing exponents (Stein, Chernoff, Sanov kept apart) checked against exact finite-sample errors, the Cover & Thomas and MacKay results with proof sketches, modern consumers from InfoNCE to speculative decoding, six checkpoint exercises, and an appendix that grades every connection to our earlier posts by how much it actually proves.

Oct 8, 2026•158 min read
Bochner's Theorem and the Kernel–Fourier Bridge
ai-tutorialstutorialbochner-theorem

Bochner's Theorem and the Kernel–Fourier Bridge

For continuous stationary positive-definite kernels, Bochner’s theorem turns kernel design into a choice of spectral measure. We derive random Fourier features, connect positional rotations to frequency, and explain where the bridge to attention and spectral transformers needs extra assumptions.

Sep 27, 2026•28 min read
Fourier Transforms Explained: From Signals to Spectral Bias
ai-tutorialstutorialfourier-transform

Fourier Transforms Explained: From Signals to Spectral Bias

A practical guide to Fourier transforms, spectral bias, and choosing a useful representation: why audio and images need different spectral tools, how Fourier features change learning, and where FFT mixing helps or loses information.

Sep 26, 2026•29 min read
The Frequency Domain: How Fourier Transforms Secretly Powered Everything We Built
ai-tutorialstutorialfourier-transform

The Frequency Domain: How Fourier Transforms Secretly Powered Everything We Built

A practitioner-level synthesis of twelve weeks of tutorials through one lens — spectral bias, Fourier features, FFT-based token mixing, Bochner's theorem, and neural operators — showing that positional encodings, kernels, state-space models and diffusion schedules are the same idea in different clothes.

Sep 17, 2026•82 min read
Game Theory for ML Practitioners: Payoffs, Regret, and Equilibrium
ai-tutorialstutorialgame-theory

Game Theory for ML Practitioners: Payoffs, Regret, and Equilibrium

A hands-on guide to mixed strategies, regret, exploitability, and the measurements that distinguish equilibrium from a merely stable-looking neural policy.

Sep 2, 2026•26 min read
From Optical Flow to Neural Scene Flow: Video Understanding for 3D
ai-tutorialstutorialvideo-understanding

From Optical Flow to Neural Scene Flow: Video Understanding for 3D

How motion estimation evolved from pixel-level optical flow to neural scene flow fields — and why it matters for reconstructing dynamic 3D worlds from video.

Apr 19, 2026•16 min read
4D Gaussian Splatting Explained: Extending 3DGS to the Time Domain
ai-tutorialstutorial4d-gaussian-splatting

4D Gaussian Splatting Explained: Extending 3DGS to the Time Domain

A step-by-step introduction to 4D Gaussian Splatting — how adding a time dimension to 3DGS enables real-time dynamic scene rendering, from deformation fields to temporal regularization.

Apr 17, 2026•14 min read
From Static Splats to Dynamic Worlds: The 4D Gaussian Frontier
ai-tutorialstutorial4d-gaussian-splatting

From Static Splats to Dynamic Worlds: The 4D Gaussian Frontier

How 4D Gaussian Splatting extends real-time radiance fields into the time domain — deformation fields, temporal regularization, neural scene flow, and the path to dynamic world models.

Apr 17, 2026•15 min read
Spatial Architectures: From Capsule Networks to Equivariant Neural Networks
ai-tutorialstutorialequivariant-networks

Spatial Architectures: From Capsule Networks to Equivariant Neural Networks

A tour of the architectures that bake 3D geometry into the network itself—from Hinton's capsules and the geometric deep learning framework to modern E(n)-equivariant graph networks powering physical AI.

Apr 12, 2026•11 min read
How Vision Transformers Work (and Why Patch Tokenization Is a Hack)
ai-tutorialstutorialvision-transformer

How Vision Transformers Work (and Why Patch Tokenization Is a Hack)

Understand the elegant trick that let transformers process images, why it surprisingly works, and why its fundamental limitations drive the next generation of vision architectures.

Apr 10, 2026•11 min read
Beyond Attention: The Post-Transformer Architecture Landscape for Physical AI
ai-tutorialstutorialpost-transformer

Beyond Attention: The Post-Transformer Architecture Landscape for Physical AI

Survey the post-transformer frontier—state space models, recurrent revivals, long convolutions, and equivariant networks—and see why the architectures that will power physical AI look nothing like a stack of attention layers.

Apr 9, 2026•17 min read
Inside Marble-Like Architectures: From Pixels to 3D Worlds
ai-tutorialstutorialmachine-learning

Inside Marble-Like Architectures: From Pixels to 3D Worlds

Trend tutorial on World Models -- focusing on Worldlabs' Marble

Apr 2, 2026•12 min read
The Evolution of World Models: From RNN Dreams to Persistent 3D Worlds (2018–2026)
ai-tutorialstutorialmachine-learning

The Evolution of World Models: From RNN Dreams to Persistent 3D Worlds (2018–2026)

World Models: evolution and connection to RSI

Mar 26, 2026•13 min read
Energy-Based World Models vs. Transformer-Based Generation: Two Competing Visions for Machine Intelligence
ai-tutorialstutorialmachine-learning

Energy-Based World Models vs. Transformer-Based Generation: Two Competing Visions for Machine Intelligence

Trends of Energy-based World Models

Mar 25, 2026•18 min read