# Weathering Ongoing Uncertainty: Learning and Planning in a Time-Varying Partially Observable Environment

arXiv: https://arxiv.org/abs/2312.03263 · HTML: https://mlsanity.com/paper/2312.03263/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 45 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [DESPOT: Online POMDP Planning with Regularization](https://mlsanity.com/paper/1609.03250/) (2016)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [Online Planning Algorithms for POMDPs](https://mlsanity.com/paper/1401.3436/) (2014)
- [Learning and Planning for Time-Varying MDPs Using Maximum Likelihood Estimation](https://mlsanity.com/paper/1911.12976/) (2019)
- [Memory Bounded Open-Loop Planning in Large POMDPs using Thompson Sampling](https://mlsanity.com/paper/1905.04020/) (2019)

## Built on by (most cited first, 4 of 4)

- [A Moral Imperative: The Need for Continual Superalignment of Large Language Models](https://mlsanity.com/paper/2403.14683/) (2024)
- [ComTraQ-MPC: Meta-Trained DQN-MPC Integration for Trajectory Tracking with Limited Active Localization Updates](https://mlsanity.com/paper/2403.01564/) (2024)
- [TAB-Fields: A Maximum Entropy Framework for Mission-Aware Adversarial Planning](https://mlsanity.com/paper/2412.02570/) (2024)
- [Enhancing Robot Navigation Policies with Task-Specific Uncertainty Managements](https://mlsanity.com/paper/2505.13837/) (2025)

## Abstract (arXiv)

Optimal decision-making presents a significant challenge for autonomous systems operating in uncertain, stochastic and time-varying environments. Environmental variability over time can significantly impact the system's optimal decision making strategy for mission completion. To model such environments, our work combines the previous notion of Time-Varying Markov Decision Processes (TVMDP) with partial observability and introduces Time-Varying Partially Observable Markov Decision Processes (TV-POMDP). We propose a two-pronged approach to accurately estimate and plan within the TV-POMDP: 1) Memory Prioritized State Estimation (MPSE), which leverages weighted memory to provide more accurate time-varying transition estimates; and 2) an MPSE-integrated planning strategy that optimizes long-term rewards while accounting for temporal constraint. We validate the proposed framework and algorithms using simulations and hardware, with robots exploring a partially observable, time-varying environments. Our results demonstrate superior performance over standard methods, highlighting the framework's effectiveness in stochastic, uncertain, time-varying domains.