mlsanityData to 7 Oct 2026

R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models

arXiv 2409.14216 ↗First posted 21 Sep 2024cs.RO cs.AI cs.CVcited by 10

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

Although research has produced promising results demonstrating the utility of active inference (AIF) in Markov decision processes (MDPs), there is relatively less work that builds AIF models in the context of environments and problems that take the form of partially observable Markov decision processes (POMDPs). In POMDP scenarios, the agent must infer the unobserved environmental state from raw sensory observations, e.g., pixels in an image. Additionally, less work exists in examining the most difficult form of POMDP-centered control: continuous action space POMDPs under sparse reward signals. In this work, we address issues facing the AIF modeling paradigm by introducing novel prior preference learning techniques and self-revision schedules to help the agent excel in sparse-reward, continuous action, goal-based robotic control POMDP environments. Empirically, we show that our agents offer improved performance over state-of-the-art models in terms of cumulative rewards, relative stability, and success rate. The code in support of this work can be found at https://github.com/NACLab/robust-active-inference.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 53 of its 89 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

53 of 89 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
38 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2018World Modelscited by 1,444
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2020Mastering Atari with Discrete World Modelscited by 990
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2019Model-Based Reinforcement Learning for Ataricited by 495
  12. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  13. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  14. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  15. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  16. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  17. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  18. 2018Model-Based Active Explorationcited by 105
  19. 2020Deep active inference agents using Monte-Carlo methodscited by 72
  20. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  21. 2018Probabilistic Recurrent State-Space Modelscited by 64
  22. 2020Sophisticated Inferencecited by 56
  23. 2019Deep Active Inference as Variational Policy Gradientscited by 52
  24. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  25. 2020Reinforcement Learning through Active Inferencecited by 47
  26. 2020Whence the Expected Free Energy?cited by 40
  27. 2020Action and Perception as Divergence Minimizationcited by 31
  28. 2019Scaling active inferencecited by 30
  29. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  30. 2021Contrastive Active Inferencecited by 21
  31. 2020Deep Active Inference for Partially Observable MDPscited by 20
  32. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 20
  33. 2023On efficient computation in active inferencecited by 15
  34. 2023Mortal Computation: A Foundation for Biomimetic Intelligencecited by 15
  35. 2022On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learningcited by 13
  36. 2019Bayesian policy selection using active inferencecited by 12
  37. 2023Neural representation in active inference: using generative models to interact with -- and understand -- the lived worldcited by 4
  38. 2021Online reinforcement learning with sparse rewards through an active inference capsulecited by 3

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 4.

10 papers.

  1. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  2. 2025Four Principles for Physically Interpretable World Modelscited by 10
  3. 2025Deep Active Inference Agents for Delayed and Long-Horizon Environmentscited by 3
  4. 2026Learning to unfold cloth: Scaling up world models to deformable object manipulationcited by 3
  5. 2024Free Energy Projective Simulation (FEPS): Active inference with interpretabilitycited by 1
  6. 2025Meta-Representational Predictive Coding: Neuroscience-Informed Self-Supervised Learningcited by 0
  7. 2025Enter the Void - Planning to Seek Entropy When Reward is Scarcecited by 0
  8. 2026Enhancing Eye Feature Estimation from Event Data Streams through Adaptive Inference State Space Modelingcited by 0
  9. 2026Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regretcited by 0
  10. 2026Active Inference with a Self-Prior in the Mirror-Mark Taskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Active Inference as a Convex Markov Decision Process
  2. 2020Active Reward Learning for Co-Robotic Vision Based Exploration in Bandwidth Limited Environments
  3. 2026The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective
  4. 2026Learning POMDP World Models from Observations with Language-Model Priors
  5. 2025Real-World Robot Control by Deep Active Inference With a Temporally Hierarchical World Model
  6. 2024Reward Machine Inference for Robotic Manipulation
  7. 2024Demonstrating the Continual Learning Capabilities and Practical Application of Discrete-Time Active Inference
  8. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferences