# Two-step dynamic obstacle avoidance

arXiv: https://arxiv.org/abs/2311.16841 · HTML: https://mlsanity.com/paper/2311.16841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 112 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Self-Driving Cars: A Survey](https://mlsanity.com/paper/1901.04407/) (2019)
- [Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.07845/) (2016)
- [Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1805.01956/) (2018)
- [Memory-based control with recurrent neural networks](https://mlsanity.com/paper/1512.04455/) (2015)
- [Conditional Affordance Learning for Driving in Urban Environments](https://mlsanity.com/paper/1806.06498/) (2018)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [Collision Avoidance in Pedestrian-Rich Environments with Deep Reinforcement Learning](https://mlsanity.com/paper/1910.11689/) (2019)
- [A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications](https://mlsanity.com/paper/2111.01968/) (2021)
- [Deep Predictive Models for Collision Risk Assessment in Autonomous Driving](https://mlsanity.com/paper/1711.10453/) (2017)
- [A Vision Based Deep Reinforcement Learning Algorithm for UAV Obstacle Avoidance](https://mlsanity.com/paper/2103.06403/) (2021)
- [Taming an autonomous surface vehicle for path following and collision avoidance using deep reinforcement learning](https://mlsanity.com/paper/1912.08578/) (2019)
- [Risk-based implementation of COLREGs for autonomous surface vehicles using deep reinforcement learning](https://mlsanity.com/paper/2112.00115/) (2021)
- [Multi-UAV Conflict Resolution with Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/2111.14598/) (2021)
- [Enhanced method for reinforcement learning based dynamic obstacle avoidance by assessment of collision risk](https://mlsanity.com/paper/2212.04123/) (2022)
- [Spatial-temporal recurrent reinforcement learning for autonomous ships](https://mlsanity.com/paper/2211.01004/) (2022)
- [2-Level Reinforcement Learning for Ships on Inland Waterways: Path Planning and Following](https://mlsanity.com/paper/2307.16769/) (2023)

## Abstract (arXiv)

Dynamic obstacle avoidance (DOA) is a fundamental challenge for any autonomous vehicle, independent of whether it operates in sea, air, or land. This paper proposes a two-step architecture for handling DOA tasks by combining supervised and reinforcement learning (RL). In the first step, we introduce a data-driven approach to estimate the collision risk (CR) of an obstacle using a recurrent neural network, which is trained in a supervised fashion and offers robustness to non-linear obstacle movements. In the second step, we include these CR estimates into the observation space of an RL agent to increase its situational awareness. We illustrate the power of our two-step approach by training different RL agents in a challenging environment that requires to navigate amid multiple obstacles. The non-linear movements of obstacles are exemplarily modeled based on stochastic processes and periodic patterns, although our architecture is suitable for any obstacle dynamics. The experiments reveal that integrating our CR metrics into the observation space doubles the performance in terms of reward, which is equivalent to halving the number of collisions in the considered environment. We also perform a generalization experiment to validate the proposal in an RL environment based on maritime traffic and real-world vessel trajectory data. Furthermore, we show that the architecture's performance improvement is independent of the applied RL algorithm.