# When to ASK: Uncertainty-Gated Language Assistance for Reinforcement Learning

arXiv: https://arxiv.org/abs/2604.02226 · HTML: https://mlsanity.com/paper/2604.02226/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 12 of 12)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning](https://mlsanity.com/paper/2003.12537/) (2020)
- [The Hidden Uncertainty in a Neural Networks Activations](https://mlsanity.com/paper/2012.03082/) (2020)
- [How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective](https://mlsanity.com/paper/2508.01300/) (2025)
- [Safe Learning of PDDL Domains with Conditional Effects -- Extended Version](https://mlsanity.com/paper/2403.15251/) (2024)
- [Retrieval-augmented reasoning with lean language models](https://mlsanity.com/paper/2508.11386/) (2025)

## Built on by (most cited first, 2 of 2)

- [When in Doubt, Plan It Out: Committed Small Language Model Deliberation for Reactive Reinforcement Learning](https://mlsanity.com/paper/2606.16995/) (2026)
- [ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability](https://mlsanity.com/paper/2607.02686/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) agents often struggle with out-of-distribution (OOD) scenarios, leading to high uncertainty and random behavior. While language models (LMs) contain valuable world knowledge, larger ones incur high computational costs, hindering real-time use, and exhibit limitations in autonomous planning. We introduce Adaptive Safety through Knowledge (ASK), which combines smaller LMs with trained RL policies to enhance OOD generalization without retraining. ASK employs Monte Carlo Dropout to assess uncertainty and queries the LM for action suggestions only when uncertainty exceeds a set threshold. This selective use preserves the efficiency of existing policies while leveraging the language model's reasoning in uncertain situations. In experiments on the FrozenLake environment, ASK shows no improvement in-domain, but demonstrates robust navigation in transfer tasks, achieving a reward of 0.95. Our findings indicate that effective neuro-symbolic integration requires careful orchestration rather than simple combination, highlighting the need for sufficient model scale and effective hybridization mechanisms for successful OOD generalization.