mlsanityData to 14 Oct 2026

Efficient Sequential Decision Making with Large Language Models

arXiv 2406.12125 ↗First posted 17 Jun 2024cs.LG cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper focuses on extending the success of large language models (LLMs) to sequential decision making. Existing efforts either (i) re-train or finetune LLMs for decision making, or (ii) design prompts for pretrained LLMs. The former approach suffers from the computational burden of gradient updates, and the latter approach does not show promising results. In this paper, we propose a new approach that leverages online model selection algorithms to efficiently incorporate LLMs agents into sequential decision making. Statistically, our approach significantly outperforms both traditional decision making algorithms and vanilla LLM agents. Computationally, our approach avoids the need for expensive gradient updates of LLMs, and throughout the decision making process, it requires only a small number of LLM calls. We conduct extensive experiments to verify the effectiveness of our proposed approach. As an example, on a large-scale Amazon dataset, our approach achieves more than a 6x performance gain over baselines while calling LLMs in only 1.5% of the time steps.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

32 of 50 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  11. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  12. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 201
20 more
  1. 2020Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oraclescited by 197
  2. 2022Online Decision Transformercited by 196
  3. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  4. 2020Bypassing the Monster: A Faster and Simpler Optimal Algorithm for Contextual Bandits under Realizabilitycited by 115
  5. 2016Corralling a Band of Bandit Algorithmscited by 100
  6. 2018Practical Contextual Bandits with Regression Oraclescited by 93
  7. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  8. 2024Can large language models explore in-context?cited by 76
  9. 2020Model Selection in Contextual Stochastic Bandit Problemscited by 71
  10. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  11. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  12. 2022Contextual Bandits with Large Action Spaces: Made Practicalcited by 33
  13. 2021The Pareto Frontier of model selection for general Contextual Banditscited by 24
  14. 2022Contextual Bandits with Smooth Regret: Efficient Learning in Continuous Action Spacescited by 18
  15. 2020On Regret with Multiple Best Armscited by 16
  16. 2021Pareto Optimal Model Selection in Linear Banditscited by 14
  17. 2023Data-Driven Online Model Selection With Regret Guaranteescited by 11
  18. 2024Efficient Contextual Bandits with Uninformed Feedback Graphscited by 6
  19. 2021Near Instance Optimal Model Selection for Pure Exploration Linear Banditscited by 4
  20. 2023Infinite Action Contextual Bandits with Reusable Data Exhaustcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 43
  2. 2026ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RLcited by 14
  3. 2026HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agentscited by 9
  4. 2025Strategic Scaling of Test-Time Compute: A Bandit Learning Approachcited by 8
  5. 2024TraveLLM: Could you plan my new public transit route in face of a network disruption?cited by 5
  6. 2025Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLMcited by 2
  7. 2025Disentangling Exploration of Large Language Models by Optimal Exploitationcited by 1
  8. 2025Semantic Intelligence: Integrating GPT-4 with A Planning in Low-Cost Roboticscited by 1
  9. 2025Achieving Trustworthy Real-Time Decision Support Systems with Low-Latency Interpretable AI Modelscited by 1
  10. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  11. 2026Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gymcited by 1
  12. 2026Robust Uncertainty Quantification for Factual Generation of Large Language Modelscited by 0
4 more
  1. 2026When Do We Need LLMs? A Diagnostic for Language-Driven Banditscited by 0
  2. 2026Calibration-Gated LLM Pseudo-Observations for Online Contextual Banditscited by 0
  3. 2026Latent Action Reparameterization for Efficient Agent Inferencecited by 0
  4. 2026Phase-Aware Guidance Injection for Recurrent MAPPO in Assembly-Line Disruption Recoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DecisionLLM: Large Language Models for Long Sequence Decision Exploration
  2. 2025Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach
  3. 2025Enhancing Decision-Making of Large Language Models via Actor-Critic
  4. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Making
  5. 2024Building Decision Making Models Through Language Model Regime
  6. 2024UNO Arena for Evaluating Sequential Decision-Making Capability of Large Language Models
  7. 2025Feedback-Induced Performance Decline in LLM-Based Decision-Making
  8. 2024Efficient Reinforcement Learning with Large Language Model Priors