Back to Blog

Adam, Muon and Friends Explained: What a Normalised Step Buys, What It Costs, and Which Claims Are Theorems

artifocial•November 14, 2026•39 min read

Adaptive optimizers for practitioners who use them every day: why Adam's first step is a sign step of length about η√d whatever the gradient's size, why 'Adam moves every coordinate by η' is true for a constant gradient and false in general (3.16η after a quiet stretch, 7.27η in the worst case), why a per-coordinate scale is exactly right for a block-heterogeneous problem and why a constant-step Adam is not monotone even on a noiseless quadratic, what the Reddi counter-example does and does not say, why weight decay under AdamW is an equilibrium norm and under coupled L2 is a layer-dependent accident, what Muon's Newton–Schulz iteration computes and by how much it misses, and how to read a benchmark number. Every number is from a seeded NumPy script that ships with the piece.

Adam, Muon and Friends Explained: What a Normalised Step Buys, What It Costs, and Which Claims Are Theorems

Prerequisites: the one-coordinate picture of gradient descent — a step multiplies each curvature direction by (1−ηh)(1 - \eta h), the step is capped by the stiffest direction, the slowest sets the clock — which is §1–§2 of this package's first companion, Gradient Descent Explained. Singular values appear in §7; the one fact needed is that every matrix is a rotation, a scaling, and a rotation.

Reading path. §1–§3 are Adam, and they carry one idea three times: the normaliser makes the step's size independent of the gradient's, exactly in one case and approximately in every other, and each "approximately" has a number. §4 is why that is the right thing on a transformer-shaped problem and what it costs on a clean one. §5 is the counter-example, and what it does and does not say. §6 is weight decay, which under Adam is not a penalty but a norm. §7 is Muon, where the normalisation is per matrix instead of per coordinate. §8 is the scoreboard and how to read it. §9 is three traps.

Companion Notebook

This article pairs with the companion script worked_numbers_companions.py — NumPy only, float64, CPU-only, one seed, about sixteen seconds — which ships with this piece beside the flagship's worked_numbers.py; its B sections print every number quoted below and its A sections are the first companion's. A notebook, 01_adaptive_methods.ipynb, is built from the B sections cell for cell once the prose is final. A section number in the form §B3 beside a block names the section that printed it; the numbers are verbatim. Where a section re-computes a quantity of the flagship's script (its §12, §13, §14, §20, §21, §46, §48) it does so on its own instance under its own seed, so that the deterministic quantities agree to the printed digit and the random ones agree to sampling error.

One convention, once. Adam (Kingma & Ba 2015) keeps two exponential moving averages per coordinate, mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1 - \beta_1)g_t of the gradient and vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1 - \beta_2)g_t^2 of its square, corrects both for starting at zero, m^t=mt/(1−β1t)\hat m_t = m_t/(1 - \beta_1^t) and v^t=vt/(1−β2t)\hat v_t = v_t/(1 - \beta_2^t), and steps xt+1=xt−η m^t/(v^t+ϵ)x_{t+1} = x_t - \eta\,\hat m_t/(\sqrt{\hat v_t} + \epsilon). The defaults are β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8}, and every block below uses them unless it says otherwise. Where plain momentum appears for comparison its base step is written α\alpha, as in the first companion, so that η\eta always means Adam's step.

Three kinds of sentence. Every claim below is one of three things, and the text says which at the moment it is made: a theorem, true under the hypotheses stated; an identity under a model, exact for the toy it is computed on and only a guide elsewhere; or an observation at one seed, a number from one instance of one problem. The flagship grades the same claims L1–L3; here the words do the grading, and a reader who keeps the three apart will not turn a table into a theorem.


1. The first step is a sign step

Set t=1t = 1 in the formula. Then m1=(1−β1)g1m_1 = (1 - \beta_1)g_1 and m^1=g1\hat m_1 = g_1; v1=(1−β2)g12v_1 = (1 - \beta_2)g_1^2 and v^1=g12\hat v_1 = g_1^2; the update is −η g1/(∣g1∣+ϵ)-\eta\,g_1/(|g_1| + \epsilon) in every coordinate — at ϵ=0\epsilon = 0 exactly −ηsign⁡(g1)-\eta\operatorname{sign}(g_1), a theorem; at the default ϵ=10−8\epsilon = 10^{-8} a sign step wherever ∣g1∣≫ϵ|g_1| \gg \epsilon, which is every coordinate whose gradient is not within a few ϵ\epsilon of zero. The first update moves each such coordinate by η\eta, in the direction of its gradient's sign, and knows nothing about the gradient's size:

§B1 Adam's first step is a sign step: ||x1 - x0||_2 is approximately eta sqrt(d) whatever the gradient's size — exactly at eps = 0, and wherever every |g_i| >> eps at the default
  d =       10: ||update||_2 / eta =      3.162   sqrt(d) =      3.162   (||g||_2 was 0.0025)
  d =     1000: ||update||_2 / eta =     31.621   sqrt(d) =     31.623   (||g||_2 was 0.0325)
  d =  1000000: ||update||_2 / eta =    999.910   sqrt(d) =   1000.000   (||g||_2 was 1.0005)
  with eps = 1e-8 the first step is -eta g / (|g| + eps), a sign step only where |g| >> eps: |step| / eta at |g| = 1, 1e-6, 1e-8, 1e-10, 0 = 1.00000, 0.99010, 0.50000, 0.00990, 0.00000; the 999.910 against 1000 above is that shortfall: each coordinate steps |g|/(|g| + eps) of a full sign step, short by eps/(|g| + eps) (about eps/|g| where |g| >> eps)
  the per-coordinate shortfall summed over the million coordinates is 94.4, and sqrt(d) - ||update||/eta = 0.090 is that sum / sqrt(d) = 0.094 to first order; the median shortfall is 1.5e-05 (|g| ~ 1e-3), so the sum is carried by the coordinates whose |g| is far below the RMS — the 842 coordinates with |g| < 1e-6 alone contribute 39.0 of it

A gradient of Euclidean length 0.00250.0025 produced an update of length 3.16 η3.16\,\eta; a gradient of length 11 in a million dimensions produced one of length 999.9 η999.9\,\eta — the 0.09 η0.09\,\eta short of d\sqrt d is ϵ\epsilon: each coordinate steps ∣g∣/(∣g∣+ϵ)|g|/(|g| + \epsilon) of a full sign step, a shortfall of ϵ/(∣g∣+ϵ)≈ϵ/∣g∣\epsilon/(|g| + \epsilon) \approx \epsilon/|g|, which is 10−510^{-5} at the typical ∣g∣=10−3|g| = 10^{-3} here and far more on the many coordinates whose gradient happens to be near zero — the block's second line shows those carry the sum — and ϵ\epsilon halves the step of a coordinate whose gradient is exactly 10−810^{-8} and leaves a zero gradient where it is. At a billion parameters the first step has length η⋅31,623\eta \cdot 31{,}623. This is the oldest reason warmup exists: the first steps are sign steps, the sign of a noisy gradient at initialisation is close to random, and a small η\eta for the first few thousand steps bounds the damage while vtv_t accumulates enough history to be a scale (the flagship's §5.5 has two further mechanisms). It is also the cleanest statement of what Adam is: at β1=β2=0\beta_1 = \beta_2 = 0 it is exactly sign descent, the steepest-descent direction when distance is measured by the largest coordinate rather than the Euclidean length (the flagship's §4.1), and the two averages are what make it sign descent with memory.

2. "Every coordinate moves by η\eta" — exactly when, and by how much otherwise

The intuition everyone carries about Adam is that it moves each coordinate by about η\eta per step, which is why one learning rate works across layers whose gradients differ by orders of magnitude. Here is exactly when that is true:

§B2 One coordinate under Adam: a constant gradient steps exactly eta; a gradient of constant size and alternating sign steps at most eta; a random sign steps far less
  constant g = 0.3:          |step| / eta at t = 1..6 = 1.00000, 1.00000, 1.00000, 1.00000, 1.00000, 1.00000
  g = +1, -1, +1, -1, ...:   |step| / eta at t = 1..6 = 1.00000, 0.05263, 0.33579, 0.05263, 0.20442, 0.05263   (t = 2 is 1/19 = 0.05263: m_hat = (0.09 - 0.1)/0.19, v_hat = 1)
  g = random +-1 (stationary): mean |step| / eta over steps 1000..20000 = 0.1848, max = 0.8367   (the EMA of a random sign is near zero; its RMS is 1)

While the gradient is constant — size and sign — m^t=g\hat m_t = g and v^t=∣g∣\sqrt{\hat v_t} = |g| and the step is exactly η\eta, at every tt, at ϵ=0\epsilon = 0. That is the first line and it is a theorem. The second line keeps the magnitude constant and flips the sign: now v^t=1\hat v_t = 1 exactly but m^t\hat m_t is a weighted average of ±1\pm1, and the step is at most η\eta and often far less — η/19\eta/19 at t=2t = 2, where the bias-corrected average m^2=m2/(1−β12)\hat m_2 = m_2/(1 - \beta_1^2) is (0.9⋅0.1−0.1)/0.19=−1/19(0.9 \cdot 0.1 - 0.1)/0.19 = -1/19 (the correction still matters at t=2t = 2; for large tt it has faded and m^t=mt\hat m_t = m_t, which is the regime every later bound is stated in). The third line is the case a training loop actually sees — a gradient whose sign is close to random from step to step — and the mean step is 0.18 η0.18\,\eta: the moving average of a random sign is small, its moving RMS is one, and the ratio is small. So the honest version of the intuition is: Adam's step is at most η\eta when the gradient's magnitude is steady, equal to η\eta only when its sign is too, and well below η\eta in noise.

The other direction fails as well. "At most η\eta" needed the magnitude to be steady; let it change, and the step can exceed η\eta:

§B3 Adam's per-coordinate step is not capped at eta: a quiet stretch then one unit gradient gives up to (1 - b1)/sqrt(1 - b2) = 3.16 eta; the worst history gives 7.27 eta
  b1 = 0.9, b2 = 0.999: single-spike bound (1-b1)/sqrt(1-b2) = 3.1623; supremum over all histories (Cauchy-Schwarz, t -> inf) = 7.2703
  the supremum factorises: m = (1-b1) sum_j b1^j g_(t-j), v = (1-b2) sum_j b2^j g_(t-j)^2; write b1^j = (b1^j / b2^(j/2)) b2^(j/2) and Cauchy-Schwarz gives |m|/sqrt(v) <= (1-b1)/sqrt(1-b2) x 1/sqrt(1 - b1^2/b2) = 3.1623 x 2.2991 = 7.2703, with equality iff g_(t-j) is proportional to (b1/b2)^j
      t0 quiet steps, then one unit gradient    |step| / eta at t0    geometric history g_{t-j} = (b1/b2)^j, run through Adam, |step| / eta at t = t0
         1                                          1.0000                    1.0000
        10                                          0.4844                    1.0431
       100                                          0.9758                    2.2434
      1000                                          2.5146                    5.7812
     10000                                          3.1622                    7.2701

After a long stretch of zero gradients, one unit gradient arrives: mm is (1−β1)(1 - \beta_1), vv is (1−β2)(1 - \beta_2), and the step is η(1−β1)/1−β2=3.16 η\eta(1 - \beta_1)/\sqrt{1 - \beta_2} = 3.16\,\eta — the figure Kingma & Ba give in their §2.1 for exactly this case, reached here once the quiet stretch is long enough for the bias corrections to have faded (t0=10,000t_0 = 10{,}000). And a single spike is not the worst history. Write mm and vv as the geometric sums they are, mt=(1−β1)∑jβ1jgt−jm_t = (1 - \beta_1)\sum_j \beta_1^j g_{t-j} and vt=(1−β2)∑jβ2jgt−j2v_t = (1 - \beta_2)\sum_j \beta_2^j g_{t-j}^2, split each weight as β1j=(β1j/β2j/2)⋅β2j/2\beta_1^j = (\beta_1^j/\beta_2^{j/2})\cdot\beta_2^{j/2}, and Cauchy–Schwarz gives ∣mt∣≤(1−β1)∑j(β12/β2)j ∑jβ2jgt−j2|m_t| \le (1 - \beta_1)\sqrt{\sum_j (\beta_1^2/\beta_2)^j}\,\sqrt{\sum_j \beta_2^j g_{t-j}^2}; so for large tt, once the bias corrections have faded, ∣mt∣/vt≤1−β11−β2⋅11−β12/β2=3.162×2.299=7.270|m_t|/\sqrt{v_t} \le \frac{1 - \beta_1}{\sqrt{1 - \beta_2}}\cdot\frac{1}{\sqrt{1 - \beta_1^2/\beta_2}} = 3.162 \times 2.299 = 7.270, with equality exactly when gt−j∝(β1/β2)jg_{t-j} \propto (\beta_1/\beta_2)^j. That is a bound over every gradient history, and the right-hand column runs the attaining history — a gradient that decays into the past at the rate β1/β2\beta_1/\beta_2 — through Adam and reaches 7.2701 η7.2701\,\eta at t=10,000t = 10{,}000 (the flagship's §4.4 has the algebra with the finite-tt corrections). This is the shape of a rare token's embedding row: quiet for thousands of steps, then not. It is the reason gradient clipping and a non-negligible ϵ\epsilon stay in the recipe, and the reason "η\eta is the per-coordinate step" is a trust-region intuition and not a bound. The bound is 7.3 η7.3\,\eta.

3. The two memories

The whole of §2 is a statement about two time constants. mm forgets at the rate β1=0.9\beta_1 = 0.9: after ten steps it has absorbed 65%65\% of a new gradient, after a hundred all of it. vv forgets at the rate β2=0.999\beta_2 = 0.999: after a hundred steps it has absorbed under 10%10\% of a new scale, after a thousand 63%63\%:

§B9 The two memories: how much of a new gradient scale v has absorbed after t steps, 1 - b2^t, and the effective learning rate the momentum gives a constant gradient, 1/(1 - b1)
        t      1 - 0.999^t (share of a new scale in v)     1 - 0.9^t (share in m)
        1              0.0010                                  0.1000
       10              0.0100                                  0.6513
      100              0.0952                                  1.0000
     1000              0.6323                                  1.0000
     3000              0.9503                                  1.0000
    10000              1.0000                                  1.0000
  time constant of v: 1/(1 - b2) = 1000 steps; of m: 1/(1 - b1) = 10 steps; a constant gradient's effective step under plain momentum (no normaliser) is alpha/(1 - beta) = 10 alpha at beta = 0.9

So m^/v^\hat m/\sqrt{\hat v} is a ten-step average divided by a thousand-step RMS, and every number in §2 is a consequence of that mismatch: the spike exceeds η\eta because mm has already caught the new gradient while vv still remembers the old quiet; the random sign steps below η\eta because a ten-step average of ±1\pm1 is small while the thousand-step RMS is one. β2\beta_2 is the knob that sets how long the scale is trusted, and it is a knob with a wrong setting (§9.1), not merely a noisy one. The last line is the one fact about momentum that is pure algebra and that every reader should carry: without a normaliser, a constant gradient's effective step under momentum β\beta is α/(1−β)\alpha/(1 - \beta), ten times α\alpha at β=0.9\beta = 0.9. Adam's normaliser removes exactly that factor — m^/v^→1\hat m/\sqrt{\hat v} \to 1 on a constant gradient — which is one more way to say that Adam's η\eta and SGD-with-momentum's α\alpha are not the same kind of number.

4. The problem Adam is right for, and the one it is not

Why does Adam beat SGD on transformers by a margin it does not show on convolutional nets? The explanation with the most controlled evidence is about curvature, and the smallest model of it is a quadratic with blocks:

§B4 Block heterogeneity: a 4-block quadratic (25 coordinates each, curvatures 1, 10, 100, 1000) — one learning rate for all against one per block, and Adam at a constant step
  f(x0) = 15858.56; GD with the one safe step 1/L (L = 1000): 10529 iterations to f < 1e-8 (the curvature-1 block contracts by (1 - 1/1000)^2 per step)
  per-block step 1/L_block: f after 1 step = 6.9e-30  (each block is a scalar quadratic; a diagonal preconditioner is exact)
  Adam eta = 0.1   constant        : f at t = 100, 500, 1000, 2000, 3000 = 2.746e-01, 2.151e-19, 2.826e-42, 1.178e-04, 4.042e-03;  max f over t in [1000, 3000] = 1.482e+00
  Adam eta = 0.01  constant        : f at t = 100, 500, 1000, 2000, 3000 = 3.133e+03, 1.151e+00, 2.752e-08, 1.904e-39, 1.651e-09;  max f over t in [1000, 3000] = 1.631e-02
  Adam eta = 0.1   decayed x0.995/step: f at t = 100, 500, 1000, 2000, 3000 = 1.740e-01, 8.172e-20, 1.907e-28, 3.233e-29, 3.164e-29;  max f over t in [1000, 3000] = 1.907e-28
  at a constant step Adam is not monotone even on a noiseless quadratic: once the gradient has been tiny for ~1/(1-b2) steps v forgets the old scale, m/sqrt(v) grows back toward one and a step of ~eta per coordinate kicks the iterate out of the minimum it had reached; a decaying step removes the re-excitation

Four blocks of twenty-five coordinates with curvatures 11, 1010, 100100, 1,0001{,}000. One learning rate for everything must be 1/10001/1000 for the stiffest block, and then the softest block converges at (1−1/1000)2t(1 - 1/1000)^{2t}: ten and a half thousand steps. One learning rate per block, set from the block's known curvature, solves the whole problem in one step, because each block is a scalar quadratic and a diagonal preconditioner built from the Hessian is exact on a diagonal Hessian. That row is the ideal benchmark, not Adam: Adam estimates a per-coordinate gradient scale from its moments, which tracks the curvature only in the sense that a stiff block's gradients are larger, and it needs the vv memory of §3 to form the estimate — the three Adam rows are what the estimate actually does. Zhang et al. (2024) measured the Hessian of transformers blockwise — embedding, each attention projection, each MLP matrix — and found exactly this picture: block spectra differing by orders of magnitude, where a CNN's do not; SGD on par with Adam without the heterogeneity and worse with it. On the quadratic the claim is a theorem; as the explanation for transformers it is the best-evidenced of three mechanisms that are not exclusive of each other (the flagship's §4.4 grades all three).

Now read the three Adam rows, because they show the cost of the same normaliser on a clean problem. At η=0.1\eta = 0.1 Adam is at 10−1910^{-19} by step 500500 and 10−4210^{-42} by step 1,0001{,}000 — far faster than gradient descent could be — and then at step 2,0002{,}000 the loss is 10−410^{-4}, and the largest value it took between steps 1,0001{,}000 and 3,0003{,}000 is 1.481.48, above where it was at step 100100. Adam at a constant step is not monotone, even here, with no noise and a convex quadratic.

The mechanism is §3: once the gradient has been tiny for about 1/(1−β2)1/(1 - \beta_2) steps, vv has forgotten the old scale; m^/v^\hat m/\sqrt{\hat v} drifts back toward one; and the step, which is again ∼η\sim\eta per coordinate, kicks the iterate out of a minimum it had reached to forty digits. At η=0.01\eta = 0.01 the same thing happens later and smaller. A decaying step removes the re-excitation entirely — the third row sits at 10−2810^{-28} and stays. This is one seeded instance, not a theorem: it shows that a constant-step Adam near a minimum can re-excite itself and that the mechanism is the vv memory. Whether the re-excitation settles into a periodic cycle, and at what size, depends on the problem and the step, and we claim no general limit-cycle result. What the instance does establish is that monotone descent is not a property of the method, and that the schedule is not optional.

5. The counter-example, and what it does and does not say

Kingma & Ba's convergence proof had a gap, and Reddi, Kale & Kumar (2018) closed it in the wrong direction: there is a convex problem on which Adam does not converge. Online linear losses on [−1,1][-1, 1]: the gradient is CC every third round and −1-1 otherwise, with C>2C > 2, so the loss over one period is (C−2)x(C - 2)x and the best point is x=−1x = -1.

§B5 Reddi, Kale & Kumar (2018): online linear losses on [-1, 1], gradient C every third round and -1 otherwise — Adam drifts to the wrong end, AMSGrad does not
  C = 3.0, beta2 = 1/(1+C^2) = 0.100, beta1 = 0, eta_t = 0.1/sqrt(t) (the setting of their Theorem 1): after 5000 rounds Adam sits at x = +1.000 (average regret +0.6536);  AMSGrad at x = -1.000 (average regret +0.0381)
  same sequence with the default first-moment coefficient beta1 = 0.9 and bias correction, beta2 kept at 0.1 (not the full defaults): Adam sits at x = -1.000 (average regret +0.0098) — the momentum averages the three gradients to (C - 2)/3 > 0 before the normaliser sees them, and the drift reverses; Theorem 2 of the paper builds a different sequence for beta1 > 0
  same sequence at the full defaults (beta1, beta2) = (0.9, 0.999) with bias correction: Adam sits at x = -1.000 (average regret +0.0140); the displayed three-period example uses beta2 = 0.1 — the theorem is not restricted to that value, but the sequence that defeats a long beta2 is a different and longer one
  the sum of losses over one period of 3 is (C - 2) x = 1.0 x, minimised at x = -1; with beta1 = 0 the step after the C-gradient is small (v just jumped) and after each -1 gradient it is large: net drift is toward +1

The mechanism is §3 once more, with a short β2\beta_2: the large gradient CC inflates vv, so the step that follows it is small; the two gradients of −1-1 that follow let vv decay, so their steps are large; the net motion per period is toward +1+1, the wrong end, and Adam sits there with a regret that grows linearly. AMSGrad — replace v^t\hat v_t by its running maximum, so the effective step never increases — converges to −1-1. The theorem is real: Adam's regret can be Ω(T)\Omega(T) on a convex problem for any constant β1<β2\beta_1 < \sqrt{\beta_2}.

Now the second line, which the paper's readers often miss. Run the same gradient sequence through Adam with the default first-moment coefficient β1=0.9\beta_1 = 0.9 (bias-corrected), keeping β2=0.1\beta_2 = 0.1, and it goes to −1-1, the right end, with an average regret of 0.00980.0098; at the full defaults (0.9,0.999)(0.9, 0.999) it goes to −1-1 as well, regret 0.01400.0140: the momentum averages the three gradients to (C−2)/3>0(C - 2)/3 > 0 before the normaliser sees them, and the drift reverses. The theorem for β1>0\beta_1 > 0 is true, but it needs a different sequence (their Theorem 2 builds it), and the sequence above is the β1=0\beta_1 = 0 one. Two honest conclusions. The failure is narrower than the theorem reads: the displayed three-period example uses β2=0.1\beta_2 = 0.1 and an adversarial order — the theorem is not restricted to that β2\beta_2, but the sequence that defeats a long one is a different and longer construction — and nobody runs a β2\beta_2 of 0.10.1. And it is not empty: the paper's stochastic version needs a rare large gradient against many small ones — the shape of a rare token's gradient, §2 again — and that one is not exotic. The AMSGrad fix ships as an off-by-default flag in every framework; what practice does instead is run β2\beta_2 close to one, which is precisely the hypothesis the positive convergence results need (Défossez et al. 2022).

6. Weight decay under Adam is a norm, not a penalty

For a weight followed by a normalisation layer — which is most of a modern network — the function does not depend on ∥w∥\|w\|, and an ℓ2\ell_2 penalty "regularises" nothing about the function. What it does is set the norm, and through the norm, the angle by which each update turns the weight. The computation is the AR(1) of the first companion's §5 in another costume: under AdamW's decoupled update w←w−η(u+λw)w \leftarrow w - \eta(u + \lambda w), with uu an update of unit RMS per coordinate that is uncorrelated with ww, the squared norm obeys E∥w∥2←(1−ηλ)2E∥w∥2+η2d\mathbb E\|w\|^2 \leftarrow (1 - \eta\lambda)^2\mathbb E\|w\|^2 + \eta^2 d, whose fixed point is ∥w∥/d→η/(2λ)\|w\|/\sqrt d \to \sqrt{\eta/(2\lambda)}, and the angular step is then ηd/∥w∥=2ηλ\eta\sqrt d/\|w\| = \sqrt{2\eta\lambda}:

§B6 Weight decay under Adam-style updates is an equilibrium: decoupled (AdamW) settles ||w||/sqrt(d) at sqrt(eta/(2 lambda)) and the angular step at sqrt(2 eta lambda); coupled L2 does not
  d = 1000, eta = 0.01, lambda = 0.1: equilibrium ||w||/sqrt(d) = sqrt(eta/(2 lambda)) = 0.2236, angular step sqrt(2 eta lambda) = 0.0447 rad, timescale 1/(eta lambda) = 1000 steps
  decoupled:  t      ||w||/sqrt(d)    angular step this update (rad)
                 1       2.9969           0.0033
               100       2.7144           0.0037
              1000       1.1243           0.0087
              3000       0.2757           0.0364
             10000       0.2273           0.0440
  two layers under Adam's normaliser: gradient noise RMS 1 (first half) and 0.01 (second half); ||w||/sqrt(d/2) per half at t = 1000, 3000, 10000
    decoupled (AdamW)                        : t=1000: 1.1178 / 1.1205   t=3000: 0.2639 / 0.2491   t=10000: 0.2123 / 0.2213
    coupled L2 (lambda w inside the gradient): t=1000: 1.1470 / 0.0077   t=3000: 0.2666 / 0.0133   t=10000: 0.2076 / 0.0218
  why the two-layer rows sit near 0.2226 and NOT at 0.230 x 0.2236 = 0.0515: Adam's update is a momentum average of the normalised noise, so its one-step RMS is only about sqrt((1-b1)/(1+b1)) = 0.2294 of the noise's (0.230 measured), but successive updates are correlated and the AR(1) recursion sums that correlation
  scalar proxy m_t = b1 m_(t-1) + (1-b1) g_t, w_(t+1) = (1 - eta lambda) w_t - eta m_t, unit Gaussian g: exact stationary RMS of w = eta sqrt( (1-b1)/(1+b1) * (1 + a b1) / ((1 - a^2)(1 - a b1)) ) = 0.2226 with a = 1 - eta lambda = 0.999; simulated over 1000 coordinates at t = 20000: 0.2227; the uncorrelated-noise formula sqrt(eta/(2 lambda)) = 0.2236 lands within 0.5% of it because the correlation factor (1 + a b1)/(1 - a b1) = 18.82 almost exactly cancels the variance factor (1-b1)/(1+b1) = 0.0526
  (so the same-norm-in-both-layers result stands — the normaliser hands both layers unit-RMS noise before the momentum — but it is a statement about a correlated AR(1), not 'the update RMS times the formula'; the coupled form's second layer, where lambda w dominates the gradient, is driven to a different norm by a decay the normaliser has rescaled)

Started at an RMS of 33 per coordinate, the norm takes about 1/(ηλ)=1,0001/(\eta\lambda) = 1{,}000 steps to decay and settles at 0.2270.227 against the predicted 0.2240.224; the angular step settles at 0.0440.044 radians against 0.04470.0447, about 2.5°2.5° per update, independent of the gradient's scale and of the start. The two consequences are the ones the flagship's §5.6 grades: η\eta and λ\lambda are not two knobs — their product sets the rotation rate and their ratio sets the norm, and the norm is invisible to the function; and the first 1/(ηλ)1/(\eta\lambda) steps are a transient in the norm, which is one reason warmup and weight decay interact.

The second block is why AdamW (Loshchilov & Hutter 2019) is not Adam with an ℓ2\ell_2 term. Take two "layers" whose gradient noise differs by a factor of a hundred. Under the decoupled form both settle at the same norm — 0.210.21 and 0.220.22, against 0.22260.2226 from the computation below — because the normaliser makes the update's statistics the same in both layers and the decay is applied outside it. Under the coupled form, where λw\lambda w is added to the gradient before the normaliser, the second layer collapses to 0.0080.008 by step 1,0001{,}000: there the decay term dominates the gradient, the normaliser rescales it to a unit-sized step, and the layer is driven toward zero by a force that has nothing to do with λ\lambda's nominal value. The coupled form gives every layer a different effective decay set by that layer's gradient scale; the decoupled form gives every layer the same equilibrium. That is the whole of the "AdamW is better" claim at the level of algebra; whether it generalises better is a measurement, and the flagship labels it one.

One correction to the first paragraph's recipe, because it is the kind of shortcut this piece exists to catch. The formula η/(2λ)\sqrt{\eta/(2\lambda)} assumed an update that is uncorrelated from step to step. Adam's update is a momentum average of the normalised noise, so its one-step RMS is only 0.230.23 — and the tempting fix, "multiply the formula by that RMS", gives 0.050.05, which the table refutes. The reason is temporal correlation: successive updates share the same mm, and the AR(1) recursion for E∥w∥2\mathbb E\|w\|^2 picks up the cross term between the weight and the update. For the scalar proxy mt=βmt−1+(1−β)gtm_t = \beta m_{t-1} + (1 - \beta)g_t, wt+1=(1−ηλ)wt−ηmtw_{t+1} = (1 - \eta\lambda)w_t - \eta m_t with unit Gaussian gtg_t, the exact stationary variance is η2 1−β1+β 1+aβ(1−a2)(1−aβ)\eta^2\,\frac{1 - \beta}{1 + \beta}\,\frac{1 + a\beta}{(1 - a^2)(1 - a\beta)} with a=1−ηλa = 1 - \eta\lambda: the variance factor 0.0530.053 and the correlation factor 18.818.8 nearly cancel, and the RMS is 0.22260.2226, within 0.5%0.5\% of the uncorrelated formula's 0.22360.2236 — the block's last lines compute it and simulate it. So the two-layer rows illustrate a scale-independent equilibrium whose value the correlated AR(1) predicts; the uncorrelated formula lands nearby at these constants by that near-cancellation, and it is the correlated computation, not the instantaneous RMS, that says why.

7. Muon: normalise the matrix, not the coordinate

A network's parameters are mostly matrices, and a per-coordinate normaliser treats a matrix as a long vector. Muon (Jordan et al. 2024) does not: it takes the momentum MM of a weight matrix's gradient and replaces it by its polar factor UV⊤UV^\top — the matrix with the same singular directions and every singular value set to one — and steps W←W−η UV⊤W \leftarrow W - \eta\,UV^\top. That is the steepest-descent direction when distance is measured by the spectral norm (the flagship's §4.5; Bernstein & Newhouse 2024 is the paper that puts Adam, Shampoo and Muon in one family by the norm each one descends under). Computing UV⊤UV^\top by an SVD each step is too expensive; Muon instead runs five steps of a Newton–Schulz iteration, X←aX+b XX⊤X+c (XX⊤)2XX \leftarrow aX + b\,XX^\top X + c\,(XX^\top)^2 X with (a,b,c)=(3.4445,−4.7750,2.0315)(a, b, c) = (3.4445, -4.7750, 2.0315), a polynomial tuned to lift small singular values fast — not, as the block shows, to converge to one. Here is what five steps do to an 8×48 \times 4 gradient whose smallest singular value we forced to be tiny:

§B7 Newton-Schulz orthogonalisation (Muon's coefficients 3.4445, -4.7750, 2.0315), five steps on an 8 x 4 Gaussian gradient with one singular value forced small
  singular values after Frobenius scaling: [0.8093 0.4446 0.3839 0.0008]
  NS step 1: singular values -> [1.1471 1.0691 0.9619 0.0028]
  NS step 2: singular values -> [0.7787 0.7364 0.685  0.0096]
  NS step 3: singular values -> [1.1311 1.0696 1.0093 0.0331]
  NS step 4: singular values -> [0.7472 0.6948 0.6852 0.1137]
  NS step 5: singular values -> [1.1309 1.1206 1.0548 0.3848]
  after 5 steps: ||X||_2 = 1.1309 (exact polar factor: 1), ||X||_F = 1.9482 (exact polar factor: sqrt(min(m, n)) = 2.0000); the five-step update is 2.6% short of the ideal Frobenius norm
  near zero p(sigma) = 3.4445 sigma - 4.775 sigma^3 + 2.0315 sigma^5 ~ 3.44 sigma: a value of 8.09e-04 needs about log(1/sigma)/log(3.44) = 5.8 steps to reach the band where the healthy values live
  p(1) = 0.7010: 1 is NOT a fixed point of these coefficients (the classical Newton-Schulz cubic 1.5 s - 0.5 s^3 has p(1) = 1, p'(1) = 0); the positive fixed points are s = 0.8680 and 1.2637 with |p'(s)| = 1.58 and 6.47, both > 1 and so both unstable — the iteration never converges to 1, it oscillates in a band (rows above), a trade the coefficients make for the x3.44 lift near zero
  per-entry RMS of the exact polar factor on m x n = 8 x 4: sqrt(min/(m n)) = 1/sqrt(max(m, n)) = 0.3536 (measured 0.3536); Moonshot's 0.2 sqrt(max(m, n)) = 0.5657 rescales it to RMS 0.200, AdamW's typical per-coordinate update RMS

Three things to read. First, the three healthy singular values, which started at 0.810.81, 0.440.44, 0.380.38, end at 1.131.13, 1.121.12, 1.051.05 — near one, not at one, and on this instance all above it. That is not a convergence cut short: these coefficients have no fixed point at one. Substitute and p(1)=3.4445−4.7750+2.0315=0.701p(1) = 3.4445 - 4.7750 + 2.0315 = 0.701; the polynomial's positive fixed points are 0.8680.868 and 1.2641.264, both unstable (∣p′∣=1.6|p'| = 1.6 and 6.56.5), so the iteration oscillates in a band — watch the rows alternate between about 0.70.7 and 1.151.15 — rather than settling. The classical Newton–Schulz cubic 1.5σ−0.5σ31.5\sigma - 0.5\sigma^3 does have p(1)=1p(1) = 1 and p′(1)=0p'(1) = 0 and converges quadratically; Muon's authors traded that for a steeper slope at zero, deliberately, because five steps of a fast lift are worth more to them than an exact orthogonalisation. Second, the tiny one: 0.0008→0.0028→0.0096→0.033→0.11→0.380.0008 \to 0.0028 \to 0.0096 \to 0.033 \to 0.11 \to 0.38, a factor of 3.443.44 per step, because near zero the polynomial is ≈3.44σ\approx 3.44\sigma. A singular value of 8×10−48 \times 10^{-4} needs about six steps of that lift to reach the band where the healthy values live, and gets five, so it ends at 0.380.38. Third, the measured consequence: the exact polar factor has spectral norm 11 and Frobenius norm min⁡(m,n)=2\sqrt{\min(m, n)} = 2; the five-step output has 1.131.13 and 1.951.95, so the actual update is 13%13\% over in spectral norm and 2.6%2.6\% short in Frobenius norm on this matrix (the flagship's 6×46 \times 4 instance is 3%3\% over and 26.5%26.5\% short — the gap depends on the singular-value spread, which is the point: here three of four values are healthy, there two of four were small). Every statement of the form "Muon's update has size exactly η\eta" is a statement about UV⊤UV^\top, true for the ideal and approximate for what the code does.

The last line is why one learning rate can serve Adam and Muon in the same run. The exact polar factor's per-entry RMS is 1/max⁡(m,n)1/\sqrt{\max(m, n)} — 0.3540.354 on 8×48 \times 4 — and AdamW's per-coordinate update RMS is typically around 0.20.2; Moonshot's recipe (Liu et al. 2025) multiplies the Muon update by 0.2max⁡(m,n)0.2\sqrt{\max(m, n)} so that both have RMS 0.20.2. What that matching buys is one thing exactly: the weight-decay equilibrium of §6 becomes transferable — the same η\eta and λ\lambda give both methods the same equilibrium norm and the same predicted rotation rate. What it does not buy is an identical step: the orthogonalised update spends its length equally across singular directions where Adam's spends it per coordinate, so the effective step per direction differs between the two methods, and a learning-rate sweep on each is sweeping a different thing. And the shared rotation rate is a prediction under §6's idealisation — a unit-RMS update uncorrelated with ww — so the actual angular rates of Muon and AdamW in a run are a measurement to take, not a number to assume.

Does the direction help? On a problem whose difficulty is in the input directions rather than the coordinates, it should, and here is an illustration — an ill-conditioned matrix regression, rotated so that the stiffness is not aligned with the coordinates Adam normalises:

§B8 Four methods on an ill-conditioned matrix regression (16 x 32 weights, input covariance condition number 1e4), each at the best step from a grid, step decayed x0.998 per iteration, input rotated so the stiffness is not axis-aligned — an illustration, not a theorem
  loss at W = 0: 3.3805e+01; largest eigenvalue of the input covariance 0.9394, smallest 6.65e-05
  GD                                   best step  1.9500: t=100: 6.083e-02  t=1000: 1.112e-02  t=2000: 9.715e-03
  GD + momentum 0.9                    best step  3.5000: t=100: 3.999e-03  t=1000: 1.476e-04  t=2000: 1.051e-04
  Adam                                 best step  0.3000: t=100: 2.654e-02  t=1000: 1.123e-03  t=2000: 6.301e-04
  Muon-like (orth of momentum 0.9)     best step  0.1000: t=100: 1.154e-01  t=1000: 3.357e-04  t=2000: 6.525e-06
  the problem was built to be ill-conditioned in exactly the way a per-direction step fixes; a fixed-length update in every singular direction overshoots the easy ones early and wins late
  the same instance unrotated (Qrot^T applied, no new draw; the stiffness is axis-aligned) — the pilot the prose compares against:
    GD                                   best step  1.9500: t=100: 6.889e-02  t=1000: 1.426e-02  t=2000: 1.243e-02
    GD + momentum 0.9                    best step  3.5000: t=100: 3.748e-03  t=1000: 9.864e-05  t=2000: 6.816e-05
    Adam                                 best step  0.3000: t=100: 2.826e-04  t=1000: 6.072e-31  t=2000: 5.417e-31
    Muon-like (orth of momentum 0.9)     best step  0.1000: t=100: 1.059e-01  t=1000: 2.497e-04  t=2000: 6.236e-06
    unrotated, Adam's t=2000 loss is 25.1 orders of magnitude below the best of the other three; rotated, it is an ordinary method
  but axis-aligned is not the same as exactly preconditioned: H = diag(1, 100), x0 = (2, 1), gradient (2, 100); the exact Newton step is (-2, -1) and Adam's first eps-free step is eta x (-1, -1) — component ratios 2:1 against 1:1, so no scalar eta makes them equal; 1/sqrt(v_hat) is a gradient scale, not the inverse Hessian, even on a diagonal quadratic

The orthogonalised update is the worst of the four at a hundred steps — a fixed-length step in every singular direction overshoots the easy ones before the step has decayed — and the best by a factor of sixteen over momentum and a hundred over Adam at two thousand, because the 10−210^{-2}-scaled input directions get the same step as the 11-scaled ones. Note what the rotation did to Adam. On the unrotated instance — the block's second table, the same draw with the rotation undone — the curvature directions line up with the parameter coordinates, a per-coordinate scale is the right shape, and Adam is at 5×10−315 \times 10^{-31} by a thousand steps, twenty-five orders of magnitude below the best of the other three at two thousand; rotate the inputs so the stiffness is spread across coordinates and Adam is an ordinary method again. The right shape is not exact preconditioning: Adam's 1/v^t1/\sqrt{\hat v_t} is a gradient scale, not the inverse Hessian, even on a diagonal quadratic — on H=diag⁡(1,100)H = \operatorname{diag}(1, 100) from x0=(2,1)x_0 = (2, 1) the Newton step is (−2,−1)(-2, -1) and Adam's first ϵ\epsilon-free step is η (−1,−1)\eta\,(-1, -1), and no scalar η\eta makes those equal (the block's last line). The exact one-step solver on a diagonal problem is §4's per-block 1/Lblock1/L_{\text{block}} row, built from the Hessian; Adam's coordinate-wise normalisation estimates that shape from gradient moments, which is very effective when the problem is diagonal and no better than momentum when it is not. This block is labelled an illustration because it is one: the problem was built to be ill-conditioned in exactly the way the spectral direction fixes, and the theorems that exist for Muon are stationarity rates under spectral-norm smoothness (the flagship's §4.5 lists them), not speedup theorems for a transformer.

8. The scoreboard, and how to read one number on it

The claim that sells Muon is "2×2\times the compute efficiency of AdamW". Here is how to grade it, and the flagship's §4.6 does so for the whole 2025–26 field. Liu et al. (2025) measured it in their own scaling-law runs, which makes it a vendor's number — not wrong, but to be quoted with that label. The neutral benchmark, AlgoPerf (Dahl et al. 2023; Kasimbeg et al. 2025), had Distributed Shampoo win its tuned track by about 28%28\% over the baseline and schedule-free AdamW win its self-tuning track by about 8%8\%. Wen et al. (2025) ran ten optimizers at 0.10.1–1.21.2B parameters with equal tuning and found the matrix methods' speedup over a tuned AdamW to be 1.4×1.4\times at 0.10.1B shrinking to about 1.1×1.1\times at 1.21.2B, and named the two traps that inflate a comparison: unequal tuning, and comparing runs before the learning-rate decay has finished — which, after §4 and the first companion's §5, you can now see is comparing one run's floor to another's transient. Khona et al. (2026) report SOAP and Muon ahead of AdamW at multi-billion scale and stable at batch sizes where AdamW degrades; that is a vendor's run too, and a recent one.

So the practitioner's reading: the direction is a theorem (steepest descent under the spectral norm), the approximation to it is measured (§7), the speedup is a measurement about a model family at a scale, it shrinks with scale in the one neutral multi-scale study, and the single most valuable thing an optimizer comparison can do is finish the cooldown before it reads the loss.

9. Where intuition breaks, and what to do instead (deeper reading)

9.1 — "β2\beta_2 is just smoothing; any value near one is fine." §5. A short β2\beta_2 is wrong, not noisy — it is the hypothesis of the counter-example — and a long one is the hypothesis of the convergence proofs. The symptom of a β2\beta_2 that is too short for your gradient statistics is a loss that spikes on rare inputs; the fix is a longer β2\beta_2, the AMSGrad flag, or clipping.

9.2 — "Warmup is always needed." It is needed for the reasons it exists, and not otherwise:

§B10 Warmup on the 4-block quadratic from a far start (RMS 10): Adam at eta = 0.1 with no warmup against a 100-step linear warmup
  no warmup           : f at t = 10, 50, 100, 200, 500 = 1.358e+06, 7.132e+05, 3.133e+05, 5.974e+04, 1.151e+02
  linear warmup 100 st.: f at t = 10, 50, 100, 200, 500 = 1.573e+06, 1.304e+06, 7.317e+05, 1.606e+05, 9.697e+02
  f(x0) = 1.586e+06, ||x0||_2 = 99.2; the first Adam step moves every coordinate by eta, ||x1 - x0||_2 = eta sqrt(d) = 1.0, whatever the gradient — harmless here, because this quadratic has no noise and no sharpening for warmup to protect against; the gain warmup shows in a transformer is a measurement about the transformer, not a theorem about descent

On a noiseless quadratic from a far start, warmup only costs: eight times the loss at step 500500. The sign step of §1 is harmless when the gradient's sign is right, and here it is; warmup protects against a noisy sign at initialisation, against a sharpening landscape, and against instabilities that grow with scale (the flagship's §5.5), none of which this problem has. The point is not that warmup is useless — it is close to free at pretraining scale and the divergence it prevents is not — but that its benefit is a measurement about the model, and a reader who sees no gain from it on a clean problem has not found a bug.

9.3 — "Adam's loss is flat, so it converged." Two floors are possible, and they need different fixes. Under noise, the floor is the first companion's §5 and the fix is a cooldown. With no noise, §4: a constant-step Adam near a minimum can re-excite itself as vv forgets, on the 1/(1−β2)1/(1 - \beta_2) timescale and at a size of order η\eta per coordinate — an observation on the block quadratic, not a theorem — and the fix is the same cooldown for a different reason. Either way, flat is not converged, and a decayed step tells you which floor you were on by how far it drops.


What to take away

  1. Adam's first step is a sign step of length ηd\eta\sqrt d, whatever the gradient — exactly at ϵ=0\epsilon = 0, to within ϵ/(∣g∣+ϵ)\epsilon/(|g| + \epsilon) per coordinate at the default. 31,623 η31{,}623\,\eta at a billion parameters; the oldest reason for warmup; at β1=β2=0\beta_1 = \beta_2 = 0 Adam is sign descent.
  2. "Every coordinate moves by η\eta" is a theorem for a constant gradient, an upper bound for a constant magnitude, and false in general. η/19\eta/19 at the second step of an alternating sign; 0.18 η0.18\,\eta on average in noise; 3.16 η3.16\,\eta after a quiet stretch; 7.27 η7.27\,\eta at the supremum, attained by a history a run can produce. The bound is 7.3 η7.3\,\eta, and clipping and ϵ\epsilon are in the recipe because of it.
  3. The two memories are ten steps and a thousand steps. Every excess and every shortfall in point 2 is the ten-step average disagreeing with the thousand-step RMS; β2\beta_2 is how long the scale is trusted.
  4. A per-coordinate scale is the right shape on a diagonal problem — exact when built from the Hessian, an estimate when Adam builds it from gradient moments — and ordinary on a rotated one. One step against 10,52910{,}529 on the block quadratic; the transformer's blockwise Hessian is the measured reason Adam wins there. At a constant step Adam is not monotone even on a noiseless quadratic — 10−4210^{-42} at step 1,0001{,}000, 1.481.48 somewhere before step 3,0003{,}000, on this instance — and a decayed step removes it.
  5. The displayed Reddi sequence is the β1=0\beta_1 = 0, short-β2\beta_2 construction; the theorem is broader — any constant β1<β2\beta_1 < \sqrt{\beta_2} fails on some sequence — but with the default β1=0.9\beta_1 = 0.9, at β2=0.1\beta_2 = 0.1 or at the full defaults, the displayed sequence converges and a different one is needed. Narrower than it reads, not empty: a rare large gradient against many small ones is the stochastic version, and that is what a rare token looks like.
  6. Weight decay under AdamW sets a norm, η/(2λ)\sqrt{\eta/(2\lambda)}, and an angular rate, 2ηλ\sqrt{2\eta\lambda}, the same in every layer. Measured 0.2270.227 and 0.0440.044 against 0.2240.224 and 0.04470.0447; under momentum the update is correlated in time and the exact AR(1) gives 0.22260.2226, not the update RMS times the formula. Coupled ℓ2\ell_2 gives each layer a decay rescaled by its own gradient: 0.210.21 in one layer and 0.0080.008 in another at the same λ\lambda.
  7. Muon's direction is a theorem; its five-step Newton–Schulz is an approximation whose error you can measure. 13%13\% over in spectral norm and 2.6%2.6\% short in Frobenius norm here, 3%3\% and 26.5%26.5\% on the flagship's instance; the coefficients have no fixed point at one (p(1)=0.70p(1) = 0.70) and oscillate in a band by design; a small singular value grows by 3.443.44 per step and needs six to reach that band. The 2×2\times is a vendor's measurement; the neutral multi-scale study has 1.4×1.4\times shrinking to 1.1×1.1\times; finish the cooldown before you compare.

Where to go next

The companion script worked_numbers_companions.py prints every number above in its B sections; its A sections carry the first companion, Gradient Descent Explained, which is where the step ceiling, the condition number, the noise floor and the schedule question that this piece keeps pointing back to are built from one coordinate up.

This package's flagship, The Optimization Domain, takes each section here one level down: §4.1 for steepest descent under a norm, which is the one idea that makes Adam, Shampoo and Muon one family; §4.4 for Adam with the Cauchy–Schwarz bound, the counter-example's theorem and the three mechanisms behind "Adam wins on transformers", each graded; §4.5 for Shampoo, SOAP and Muon with the stationarity theorems that exist; §4.6 for the 2025–26 scoreboard quoted from the neutral benchmarks; §5.5–§5.6 for warmup and the weight-decay equilibrium at full size; and §11 for the assumption map that says, claim by claim, which of these is a theorem and under what hypothesis. Our Information Domain is where mirror descent and the natural gradient — the two other geometries of the step — come from.

For deeper reading: Kingma & Ba (2015) is short and worth reading in the original, §2.1 in particular; Reddi, Kale & Kumar (2018) for the counter-example and AMSGrad; Loshchilov & Hutter (2019) for the decoupling; Bernstein & Newhouse (2024), Old Optimizer, New Norm, for the one-family view; Jordan et al. (2024) for Muon as it was first written up, and Liu et al. (2025) for the recipe that made it run at scale; Kasimbeg et al. (2025) and Wen et al. (2025) for the two comparisons that controlled for tuning; and Kosson, Messmer & Jaggi (2024) for the rotational-equilibrium view of weight decay that §6 is the smallest instance of.

We build agent systems and practitioner tooling at Artifocial — see our apps, including Alarmly.

Comments