mlsanityData to 7 Oct 2026

Sense, Imagine, Act: Multimodal Perception Improves Model-Based Reinforcement Learning for Head-to-Head Autonomous Racing

arXiv 2305.04750 ↗First posted 8 May 2023cs.RO cs.AI cs.LGcited by 1

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Model-based reinforcement learning (MBRL) techniques have recently yielded promising results for real-world autonomous racing using high-dimensional observations. MBRL agents, such as Dreamer, solve long-horizon tasks by building a world model and planning actions by latent imagination. This approach involves explicitly learning a model of the system dynamics and using it to learn the optimal policy for continuous control over multiple timesteps. As a result, MBRL agents may converge to sub-optimal policies if the world model is inaccurate. To improve state estimation for autonomous racing, this paper proposes a self-supervised sensor fusion technique that combines egocentric LiDAR and RGB camera observations collected from the F1TENTH Gym. The zero-shot performance of MBRL agents is empirically evaluated on unseen tracks and against a dynamic obstacle. This paper illustrates that multimodal perception improves robustness of the world model without requiring additional training data. The resulting multimodal Dreamer agent safely avoided collisions and won the most races compared to other tested baselines in zero-shot head-to-head autonomous racing.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 21 of its 34 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

21 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022DayDreamer: World Models for Physical Robot Learning28 Jun
  2. 2021Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racing8 Mar
  3. 2020Mastering Atari with Discrete World Models5 Oct
  4. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec

Other references · 17

  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  4. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  5. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  6. 2020BADGR: An Autonomous Self-Supervised Learning-Based Navigation Systemcited by 97
  7. 2014Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictionscited by 66
  8. 2022Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racingcited by 56
  9. 2022TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Modelscited by 48
  10. 2020Regularizing Action Policies for Smooth Control with Reinforcement Learningcited by 47
  11. 2022A survey of multimodal deep generative modelscited by 47
  12. 2021Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learningcited by 41
5 more
  1. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
  2. 2020VMLoc: Variational Fusion For Learning-Based Multimodal Camera Localizationcited by 13
  3. 2020Learning to Drive Off Road on Smooth Terrain in Unstructured Environments Using an On-Board Camera and Sparse Aerial Imagescited by 9
  4. 2020Multimodal dynamics modeling for off-road autonomous vehiclescited by 6
  5. 2021How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agentscited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33

Similar, not cited

Close in wording, no citation either way

  1. 2021Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning
  2. 2024An Examination of Offline-Trained Encoders in Vision-Based Deep Reinforcement Learning for Autonomous Driving
  3. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
  4. 2026Toward the Cognitive--Physical Limits of Embodied Intelligence through a World-Model-Centric Autonomous Racing Agent
  5. 2021Autonomous Racing using a Hybrid Imitation-Reinforcement Learning Architecture
  6. 2025Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight
  7. 2022A Benchmark Comparison of Imitation Learning-based Control Policies for Autonomous Racing
  8. 2022Discrete Control in Real-World Driving Environments using Deep Reinforcement Learning