mlsanityData to 14 Oct 2026

Can large language models explore in-context?

arXiv 2403.15371 ↗First posted 22 Mar 2024cs.LG cs.AI cs.CLcited by 76

Not on the map. Why?

Cited by 21 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making. We focus on native performance of existing LLMs, without training interventions. We deploy LLMs as agents in simple multi-armed bandit environments, specifying the environment description and interaction history entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5, GPT-4, and Llama2, using a variety of prompt designs, and find that the models do not robustly engage in exploration without substantial interventions: i) Across all of our experiments, only one configuration resulted in satisfactory exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally summarized interaction history, presented as sufficient statistics; ii) All other configurations did not result in robust exploratory behavior, including those with chain-of-thought reasoning but unsummarized history. Although these findings can be interpreted positively, they suggest that external summarization -- which may not be possible in more complex settings -- is important for obtaining desirable behavior from LLM agents. We conclude that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required to empower LLM-based decision making agents in complex settings.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Builds on 66 of its 83 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

66 of 83 references.

References · 66

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
38 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2019Introduction to Multi-Armed Banditscited by 481
  7. 2017A Tutorial on Thompson Samplingcited by 433
  8. 2022Using cognitive psychology to understand GPT-3cited by 398
  9. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  10. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  12. 2023In-Context Learning Creates Task Vectorscited by 312
  13. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  14. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  15. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  16. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  17. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  18. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  19. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  20. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  21. 2023The Learnability of In-Context Learningcited by 127
  22. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 125
  23. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  24. 2023In-Context Convergence of Transformerscited by 115
  25. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  26. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  27. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  28. 2023In-Context Learning through the Bayesian Prismcited by 83
  29. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  30. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
  31. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
  32. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  33. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 67
  34. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  35. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  36. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  37. 2024CogBench: a large language model walks into a psychology labcited by 63
  38. 2023Meta-in-context learning in large language modelscited by 59

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 35; 2026: 24.

76 papers.

Sort:

By area:

  1. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  4. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  5. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 56
  6. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  7. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 35
  8. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  9. 2025AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprisecited by 33
  10. 2025Training a Generally Curious Agentcited by 31
  11. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  12. 2024Controlling Large Language Model Agents with Entropic Activation Steeringcited by 29
38 more
  1. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 29
  2. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 29
  3. 2024Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgmentscited by 26
  4. 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Makingcited by 22
  5. 2025Meta-RL Induces Exploration in Language Agentscited by 21
  6. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
  7. 2025Toward Efficient Exploration by Large Language Model Agentscited by 17
  8. 2024Efficient Sequential Decision Making with Large Language Modelscited by 16
  9. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  10. 2024Collective Innovation in Groups of Large Language Modelscited by 11
  11. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  12. 2024Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identificationscited by 10
  13. 2025EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agentscited by 10
  14. 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?cited by 10
  15. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 8
  16. 2025Should You Use Your Large Language Model to Explore or Exploit?cited by 8
  17. 2024LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Banditscited by 7
  18. 2024Reflection-Bench: Evaluating Epistemic Agency in Large Language Modelscited by 7
  19. 2025Large Language Models Think Too Fast To Explore Effectivelycited by 7
  20. 2025Meta-Prompt Optimization for LLM-Based Sequential Decision Makingcited by 7
  21. 2025Large Language Model-Enhanced Multi-Armed Banditscited by 7
  22. 2025MiGrATe: Mixed-Policy GRPO for Adaptation at Test-Timecited by 7
  23. 2025CaRT: Teaching LLM Agents to Know When They Know Enoughcited by 7
  24. 2026Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RLcited by 7
  25. 2025Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Modelscited by 6
  26. 2025Improving LLM-based Global Optimization with Search Space Partitioningcited by 6
  27. 2026Unlocking LLM Creativity in Science through Analogical Reasoningcited by 6
  28. 2025Do LLMs Strategically Reveal, Conceal, and Infer Information? A Theoretical and Empirical Analysis in The Chameleon Gamecited by 5
  29. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 5
  30. 2025In-Context Reinforcement Learning via Communicative World Modelscited by 5
  31. 2026Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidancecited by 5
  32. 2025Understanding Before Reasoning: Enhancing Chain-of-Thought with Iterative Summarization Pre-Promptingcited by 4
  33. 2025Comparing Exploration-Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-armed Bandit Experimentscited by 4
  34. 2025When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Trainingcited by 3
  35. 2025Safe In-Context Reinforcement Learningcited by 3
  36. 2026Exploration and Exploitation Errors Are Measurable for Language Model Agentscited by 3
  37. 2026Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvementcited by 3
  38. 2026RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systemscited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective
  2. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration
  3. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Making
  4. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?
  5. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods
  6. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  7. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems
  8. 2025Expanding the Action Space of LLMs to Reason Beyond Language