# RAY-TOLD: Ray-Based Latent Dynamics for Dense Dynamic Obstacle Avoidance with TDMPC

arXiv: https://arxiv.org/abs/2604.27450 · HTML: https://mlsanity.com/paper/2604.27450/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 18 of 18)

- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning](https://mlsanity.com/paper/1710.03937/) (2017)
- [Diffusion Model Predictive Control](https://mlsanity.com/paper/2410.05364/) (2024)
- [Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation](https://mlsanity.com/paper/2209.10780/) (2022)
- [Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight](https://mlsanity.com/paper/2306.09852/) (2023)
- [IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control](https://mlsanity.com/paper/2306.00867/) (2023)
- [TD-M(PC)2: Improving Temporal Difference MPC Through Policy Constraint](https://mlsanity.com/paper/2502.03550/) (2025)
- [Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation](https://mlsanity.com/paper/2504.19322/) (2025)
- [iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning](https://mlsanity.com/paper/2406.02696/) (2024)
- [DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation](https://mlsanity.com/paper/2410.10646/) (2024)
- [SOMBRL: Scalable and Optimistic Model-Based RL](https://mlsanity.com/paper/2511.20066/) (2025)
- [Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments](https://mlsanity.com/paper/2109.07120/) (2021)
- [QT-TDM: Planning With Transformer Dynamics Model and Autoregressive Q-Learning](https://mlsanity.com/paper/2407.18841/) (2024)
- [MPC-Inspired Reinforcement Learning for Verifiable Model-Free Control](https://mlsanity.com/paper/2312.05332/) (2023)
- [Collision-Free Robot Navigation in Crowded Environments using Learning based Convex Model Predictive Control](https://mlsanity.com/paper/2403.01450/) (2024)
- [Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC](https://mlsanity.com/paper/2412.19669/) (2024)

## Abstract (arXiv)

Dense, dynamic crowds pose a persistent challenge for autonomous mobile robots. Purely reactive planning methods, such as Model Predictive Path Integral (MPPI) control, often fail to escape local minima in complex scenarios due to their limited prediction horizon. To bridge this gap, we propose Ray-based Task-Oriented Latent Dynamics (RAY-TOLD), a hybrid control architecture that integrates obstacle information into latent dynamics and utilizes the robustness of physics-based MPPI with the long-horizon foresight of reinforcement learning. RAY-TOLD leverages a LiDAR-centric latent dynamics model to encode high-dimensional sensor data into a compact state representation, enabling the learning of a terminal value function and a policy prior. We introduce a policy mixture sampling strategy that augments the MPPI candidate population with trajectories derived from the learned policy, effectively guiding the planner towards the goal while maintaining kinematic feasibility. Extensive tests in a stochastic environment with high-density dynamic obstacles demonstrate that our method outperforms the MPPI baseline, reducing the collision rate. The results confirm that blending short-horizon physics-based rollouts with learned long-horizon intent significantly enhances navigation reliability and safety.