Planning Under Uncertainty: From MPC to Active Inference

Level: Intermediate | Part of Artifocial W33 Basics

The Plan Is Wrong. Now What?

Suppose you have a world model: something that takes the current state and a proposed action and predicts what happens next. Maybe it's a Gaussian process fit to a few minutes of robot data, maybe a neural network trained on video. Either way it is wrong — a little wrong at one step, and increasingly wrong the further you push it.

The naive move is to search at the start of the episode for the best hundred-step plan your model can imagine, then execute all hundred steps. This fails because it is open-loop: every small prediction error accumulates with nothing to correct it. Ten steps in, your model thinks the arm is holding a cup it dropped on step three, and the rest is fiction.

This week's advanced companion covers what goes wrong even when you do everything right. This piece is the everything-right part.

Model-Predictive Control: Plan a Lot, Commit a Little

Model-predictive control (MPC) is the fix, and it's almost insultingly simple. At every timestep you:

Look at where you actually are, right now, measured — not predicted.
Search with the model for a good action sequence over a short horizon (say 10–30 steps).
Execute only the first action of that sequence.
Throw the rest away, and repeat.

The horizon slides forward one step each time you act — hence the other name, receding-horizon control.

The part practitioners under-appreciate is that throwing the plan away is the point. Re-measuring and re-planning every step is precisely what makes MPC robust to a bad model: you never act on the long tail of a rollout, so model error gets truncated each step instead of compounding across the episode.

That reframes what "good model" means — not one that imagines a hundred steps correctly, but one whose first step is good enough to rank candidate actions, evaluated fresh, many times a second. It's why MPC on a mediocre learned model often beats open-loop planning on a better one, and how it differs from DreamerV3, which trains an actor-critic inside the model's imagination and then just does a forward pass at run time. MPC thinks at decision time: costlier per action, but it adapts the moment the situation changes.

The Search Itself: Random Shooting and CEM

"Search for a good action sequence" hides the real question. That sequence lives in a continuous space of horizon × action-dimensions, and your model is a black box you may not be able to differentiate through cleanly. What do you actually run? Almost always one of two things.

Random shooting is the baseline: sample a few hundred action sequences at random, roll each through the model, sum the predicted reward, keep the best, execute its first action. It's trivially parallel and needs nothing from the model but forward passes.

The cross-entropy method (CEM) is random shooting that learns from its own samples, and it's the workhorse:

Sample N action sequences from a Gaussian over the action space (mean and variance per timestep).
Score them all through the model and keep the top k — the elites.
Refit the Gaussian's mean and variance to those elites.
Repeat a few times, then execute the first action of the final mean.

Each round pulls the sampling distribution toward whatever worked, and the variance narrows as the search converges. Budgets are larger than newcomers expect: PETS (Chua et al., NIPS 2018) runs five iterations of 500 candidate sequences, and TD-MPC2 — a sampling-based MPC of this family in the latent space of a decoder-free world model — runs six iterations of 512. V-JEPA 2's action-conditioned variant closes the loop on real Franka arms the same way, at about 16 seconds per action on a single RTX 4090, against roughly four minutes for Cosmos, a video-generative model on the same rig — and V-JEPA 2 is drawing 10× more samples per refinement step while it does it.

These samplers dominate not because they're clever but because they're indifferent: CEM doesn't care whether your model is a GP, an ensemble of MLPs or a transformer, or whether your cost is differentiable. You buy that generality with sample count — which is what a GPU sells cheaply.

Point Estimate vs. Posterior

Here's the fork that separates ordinary MPC from uncertainty-aware MPC. Rolling a candidate sequence through the model, you can propagate a point estimate — one predicted state per step, the model's single best guess — or a distribution over where you might end up.

Point-estimate planning has a specific failure mode: the planner exploits the model's errors. CEM's whole job is to find the sequence scoring highest under the model, and where your model has barely any data its predictions are unconstrained — exactly where an optimizer finds a suspiciously wonderful plan. The planner isn't malfunctioning; it's doing its job against a scoring function that lies loudest where you know least.

Propagating a posterior fixes this at the objective level rather than by patching, and PILCO (Deisenroth & Rasmussen, ICML 2011) is still the cleanest demonstration — though note it is not itself an MPC method. PILCO optimizes a feedback controller offline by analytic gradients rather than re-searching at each step, which is what makes it the purest test of the propagation idea in isolation. It fits a Gaussian-process model of the dynamics and pushes the model's posterior forward through time — approximating the state distribution at each step as a Gaussian and matching its mean and covariance, a technique called moment matching. Because every rollout carries its own doubt, a plan routed through unknown territory returns a wide, diffuse distribution instead of a narrow, fabulous point prediction — and near the goal, where PILCO's saturating cost punishes that spread, the policy optimizer stops being rewarded for wishful thinking. (Far from the goal the same cost can reward spread, which is where PILCO's exploration quietly comes from.)

The payoff wasn't marginal: PILCO solved cart-pole swing-up-and-balance on real hardware with 17.5 seconds of total interaction with the physical system, reporting at least an order-of-magnitude data-efficiency gain over the prior work it compared against (a mixed set, some of it balance-only). Knowing what you don't know doesn't only make failure graceful — it makes learning cheaper.

Two Kinds of Doubt, Stated Once

That machinery only pays off if you're clear about which uncertainty you're propagating. PETS is where the vocabulary became standard in deep model-based RL:

Aleatoric uncertainty is noise in the system itself — sensor jitter, contact chatter, a stochastic environment. It is irreducible: a thousand times more data does not shrink it. You capture it by having the model output the parameters of a distribution rather than a single number.
Epistemic uncertainty is your ignorance, from limited data. It is reducible, shrinking toward zero as data accumulates. You capture it with an ensemble — do independently-trained models disagree here? — or a Bayesian posterior.

Keep them separate because they imply opposite actions. Epistemic uncertainty is a reason to go look: the region is unknown, so visiting it is informative. Aleatoric uncertainty is a reason to be conservative — visiting it won't shrink the noise, and the right response is a wider margin. Conflate them and you get an agent that either avoids everything it hasn't seen or explores a coin flip forever. PETS planned with both and reached PPO's asymptotic performance in under 100 trials — per its abstract, 8× fewer samples than Soft Actor-Critic and 125× fewer than PPO on half-cheetah.

Expected Free Energy: One Objective for Acting and Knowing

Everything so far treats "achieve the goal" as the objective and "reduce uncertainty" as a bonus bolted on with a coefficient — VIME is the canonical version, adding information gain to the reward with a weight someone must tune. Active inference proposes the two were never separate: an agent scores each candidate policy with a single scalar, the expected free energy (EFE), and selects the policy that minimizes it.

Now the sign convention, because this is the most-flubbed point in secondary write-ups. You will constantly read "expected free energy is pragmatic value plus epistemic value." That has it backwards. EFE is minimized, so it cannot be a sum of two things you want more of. Flip the sign and it reads correctly: negative expected free energy decomposes into a pragmatic term (how well expected outcomes match your preferences) plus an epistemic term (how much the policy is expected to reveal). The cleanest open statement is Da Costa et al.'s synthesis paper: EFE is minimized, extrinsic value enters as the negative expected log evidence, and the epistemic contributions enter with negative signs — splitting further into salience (information gain about hidden states) and novelty (information gain about model parameters).

Read practically: minimizing EFE is maximizing pragmatic-plus-epistemic value. Say it that way and you'll never flip it.

This is more than notation. Both terms come out in the same units — nats — so nobody has to invent a coefficient to trade them off, and the framework subsumes the classics: Sajid et al. show that stripping outcome preferences out of EFE reduces active inference to optimal Bayesian experimental design, while removing the uncertainty terms reduces it to expected-utility maximization.

Two caveats belong right here, because active inference attracts more enthusiasm than it can currently cash. The explore/exploit knob is relocated, not eliminated — it reappears in how sharply you specify your preference distribution over outcomes, and choosing that sharpness is the same kind of design decision as tuning VIME's coefficient. And Da Costa et al.'s reward-maximization paper establishes that on partially observed MDPs the standard scheme yields Bellman-optimal actions only at a planning horizon of 1; longer horizons need the recursive "sophisticated inference" variant.

What You'll Build This Week

One warning to carry in: every technique here reasons about uncertainty one step at a time, and a well-calibrated one-step model does not give you a well-calibrated hundred-step imagination — Biased Dreams (Berger et al., RLC 2026) finds that ensemble disagreement captures local epistemic uncertainty without reliably reflecting the error compounding over long rollouts. That's the advanced companion's subject.

Two CPU-friendly notebooks make this concrete: an uncertainty-aware MPC that plans with the W28 Gaussian process as its transition model, propagating the posterior PILCO-style, then ablates it — same planner, same budget, point estimate instead of a distribution — so you can watch the point-estimate agent plan straight through a region the model has no data for; and a conformal action gate that calibrates a distribution-free error bound on held-out rollouts and compares it against gating on ensemble disagreement. Both open on GitHub as executed notebooks with their charts inline, and both run on CPU.

The through-line: a planner is only as honest as the uncertainty you hand it, and the cheapest way to make a wrong model useful is to stop trusting it further than one step at a time.

This tutorial is part of the Artifocial research-multimodal series. This week's advanced companion: The Full Loop: World Models That Act on What They Don't Know. Previous basics: Neuro-Symbolic AI Explained.

Build with AI — early access

Want to take planning loops like this from a notebook to real hardware? Build with AI is our early-access program for engineers moving from tutorials to production AI — hands-on guidance instead of guesswork.

Join the waitlist →