# Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Models

arXiv: https://arxiv.org/abs/2310.18127 · HTML: https://mlsanity.com/paper/2310.18127/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [ProAgent: Building Proactive Cooperative Agents with Large Language Models](https://mlsanity.com/paper/2308.11339/) (2023)
- [Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning](https://mlsanity.com/paper/2310.10103/) (2023)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Joint Prompt Optimization of Stacked LLMs using Variational Inference](https://mlsanity.com/paper/2306.12509/) (2023)
- [Introspective Tips: Large Language Model for In-Context Decision Making](https://mlsanity.com/paper/2305.11598/) (2023)
- [Adaptive Decision Making via Entropy Minimization](https://mlsanity.com/paper/1803.06638/) (2018)

## Built on by (most cited first, 9 of 9)

- [ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.09501/) (2025)
- [Efficient Reinforcement Learning with Large Language Model Priors](https://mlsanity.com/paper/2410.07927/) (2024)
- [Privacy-preserved LLM Cascade via CoT-enhanced Policy Learning](https://mlsanity.com/paper/2410.08014/) (2024)
- [DSADF: Thinking Fast and Slow for Decision Making](https://mlsanity.com/paper/2505.08189/) (2025)
- [Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting](https://mlsanity.com/paper/2410.19920/) (2024)
- [HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving](https://mlsanity.com/paper/2505.15793/) (2025)
- [Agentic Episodic Control](https://mlsanity.com/paper/2506.01442/) (2025)
- [On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners](https://mlsanity.com/paper/2606.23668/) (2026)
- [Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Audit](https://mlsanity.com/paper/2609.27087/) (2026)

## Abstract (arXiv)

Large language models (LLMs) demonstrate their promise in tackling complicated practical challenges by combining action-based policies with chain of thought (CoT) reasoning. Having high-quality prompts on hand, however, is vital to the framework's effectiveness. Currently, these prompts are handcrafted utilising extensive human labor, resulting in CoT policies that frequently fail to generalise. Human intervention is also required to develop grounding functions that ensure low-level controllers appropriately process CoT reasoning. In this paper, we propose a comprehensive training framework for complex task-solving, incorporating human prior knowledge into the learning of action policies. To that purpose, we offer a new leader-follower bilevel framework that is capable of learning to ask relevant questions (prompts) and subsequently undertaking reasoning to guide the learning of actions. The prompt policy is employed to make introspective revisions based on historical findings, leading the CoT process to consider the anticipated goals and generate outputs that lead to decisive, high-performing actions. The action policy subsequently learns to comprehend and integrate the CoT outputs to take actions. Our empirical data reveal that our framework outperforms leading methods in 5 decision-making tasks such as Overcooked and FourRoom.