# Self-Adaptive Driving in Nonstationary Environments through Conjectural Online Lookahead Adaptation

arXiv: https://arxiv.org/abs/2210.03209 · HTML: https://mlsanity.com/paper/2210.03209/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Equivalence Between Policy Gradients and Soft Q-Learning](https://mlsanity.com/paper/1704.06440/) (2017)
- [Online Meta-Learning](https://mlsanity.com/paper/1902.08438/) (2019)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments](https://mlsanity.com/paper/2005.10619/) (2020)
- [Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1911.04175/) (2019)
- [Continuous Meta-Learning without Tasks](https://mlsanity.com/paper/1912.08866/) (2019)
- [A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers](https://mlsanity.com/paper/1804.07871/) (2018)
- [Online Planning Algorithms for POMDPs](https://mlsanity.com/paper/1401.3436/) (2014)
- [Deep Reinforcement Learning amidst Lifelong Non-Stationarity](https://mlsanity.com/paper/2006.10701/) (2020)
- [Generalized Proximal Policy Optimization with Sample Reuse](https://mlsanity.com/paper/2111.00072/) (2021)
- [End-to-End Deep Reinforcement Learning for Lane Keeping Assist](https://mlsanity.com/paper/1612.04340/) (2016)
- [The Confluence of Networks, Games and Learning](https://mlsanity.com/paper/2105.08158/) (2021)
- [On Convergence Rate of Adaptive Multiscale Value Function Approximation For Reinforcement Learning](https://mlsanity.com/paper/1908.08578/) (2019)
- [Blackwell Online Learning for Markov Decision Processes](https://mlsanity.com/paper/2012.14043/) (2020)

## Built on by (most cited first, 12 of 12)

- [Symbiotic Game and Foundation Models for Cyber Deception Operations in Strategic Cyber Warfare](https://mlsanity.com/paper/2403.10570/) (2024)
- [Multi-level Traffic-Responsive Tilt Camera Surveillance through Predictive Correlated Online Learning](https://mlsanity.com/paper/2408.02208/) (2024)
- [In-Context Reinforcement Learning via Communicative World Models](https://mlsanity.com/paper/2508.06659/) (2025)
- [From Texts to Shields: Convergence of Large Language Models and Cybersecurity](https://mlsanity.com/paper/2505.00841/) (2025)
- [Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation](https://mlsanity.com/paper/2506.22365/) (2025)
- [In-Context Autonomous Network Incident Response: An End-to-End Large Language Model Agent Approach](https://mlsanity.com/paper/2602.13156/) (2026)
- [Digital Twin-based Driver Risk-Aware Intelligent Mobility Analytics for Urban Transportation Management](https://mlsanity.com/paper/2407.15025/) (2024)
- [Neurosymbolic Meta-Reinforcement Lookahead Learning Achieves Safe Self-Driving in Non-Stationary Environments](https://mlsanity.com/paper/2309.02328/) (2023)
- [Stackelberg Coupling of Online Representation Learning and Reinforcement Learning](https://mlsanity.com/paper/2508.07452/) (2025)
- [Agentic Incident Response through Digital Twin-Enhanced Multiscale Planning](https://mlsanity.com/paper/2608.02422/) (2026)
- [Cognitive Level-k Meta-Learning for Safe and Pedestrian-Aware Autonomous Driving](https://mlsanity.com/paper/2212.08800/) (2022)
- [Hierarchical Agentic Incident Response with Digital-Twin-Validated Attack Inference](https://mlsanity.com/paper/2608.15016/) (2026)

## Abstract (arXiv)

Powered by deep representation learning, reinforcement learning (RL) provides an end-to-end learning framework capable of solving self-driving (SD) tasks without manual designs. However, time-varying nonstationary environments cause proficient but specialized RL policies to fail at execution time. For example, an RL-based SD policy trained under sunny days does not generalize well to rainy weather. Even though meta learning enables the RL agent to adapt to new tasks/environments, its offline operation fails to equip the agent with online adaptation ability when facing nonstationary environments. This work proposes an online meta reinforcement learning algorithm based on the conjectural online lookahead adaptation (COLA). COLA determines the online adaptation at every step by maximizing the agent's conjecture of the future performance in a lookahead horizon. Experimental results demonstrate that under dynamically changing weather and lighting conditions, the COLA-based self-adaptive driving outperforms the baseline policies in terms of online adaptability. A demo video, source code, and appendixes are available at tt https://github.com/Panshark/COLA