A Retrospective on Active Inference

Author: Beren Millidge Published: 27 July 2024 URL: https://www.beren.io/2024-07-27-A-Retrospective-on-Active-Inference/

Summary: A personal retrospective by Beren Millidge on his PhD research into active inference, covering the relationship between active inference and reinforcement learning, the expected free energy derivation, and implications for AI alignment.

Key Arguments

1. Active Inference ≈ Reinforcement Learning: Active inference and deep RL are essentially isomorphic. Both can be derived from the “control as inference” framework. The only difference is how reward is encoded into the probabilistic graphical model, which gives rise to slightly different objectives. Practically, algorithms from one can be ported directly to the other.

2. Expected Free Energy (EFE) Critique: The EFE is not directly derivable from a standard Bayesian treatment. Existing derivations either assume their conclusion or are incorrect. The distinction between “divergence objectives” (minimising divergence between desired and actual states → exploration via information gain) and “evidence objectives” (standard utility maximisation) is more fundamental.

3. No Practical Advantage: Despite theoretical elegance, active inference provides no “secret sauce” over standard RL methods for practical tasks. The Bayesian lens is theoretically illuminating but offers limited practical benefits because the best approach is still to parametrise densities with neural networks and use black-box variational inference.

4. Weak Neuroscientific Evidence: The claim that brain function is well-explained by active inference has weak empirical support. The author prefers a bottom-up empirical approach to understanding brain algorithms rather than fitting Bayesian algorithms to neural data.

5. Positive Aspects:

   • Provides a unified framework linking perception and action
   • Focuses on structured generative models for discrete systems
   • Drives research into more principled Bayesian inference methods beyond SGD
   • Bayesian perspective on action/decision-making is underappreciated in AI alignment discourse

6. Alignment Relevance: Problems of misspecified utility functions, extreme Goodharting, and squiggle maximisation ultimately stem from poor uncertainty calibration. Bayesian frameworks with sensible priors naturally address these issues. The distinction between divergence and evidence objectives is relevant for regularised/soft optimisation in alignment.

References

• Millidge’s PhD thesis — key results on scaling active inference and the connection to RL
• “Control as inference” framework — the general mapping of decision-making to Bayesian inference