mlsanityData to 7 Oct 2026

Efficient Exploration for LLMs

arXiv 2402.00396 ↗First posted 1 Feb 2024cs.LG cs.AI cs.CLcited by 44

Not on the map. Why?

Cited by 22 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present evidence of substantial benefit from efficient exploration in gathering human feedback to improve large language models. In our experiments, an agent sequentially generates queries while fitting a reward model to the feedback received. Our best-performing agent generates queries using double Thompson sampling, with uncertainty represented by an epistemic neural network. Our results demonstrate that efficient exploration enables high levels of performance with far fewer queries. Further, both uncertainty estimation and the choice of exploration scheme play critical roles.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

23 of 39 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2018Exploration by Random Network Distillationcited by 966
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  7. 2016Deep Exploration via Bootstrapped DQNcited by 518
  8. 2017A Tutorial on Thompson Samplingcited by 420
  9. 2017Count-Based Exploration with Neural Density Modelscited by 255
  10. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  11. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  12. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
11 more
  1. 2017Deep Exploration via Randomized Value Functionscited by 155
  2. 2020Neural Thompson Samplingcited by 127
  3. 2021Epistemic Neural Networkscited by 120
  4. 2015Contextual Dueling Banditscited by 101
  5. 2017Ensemble Samplingcited by 62
  6. 2016Double Thompson Sampling for Dueling Banditscited by 42
  7. 2020Hypermodels for Explorationcited by 37
  8. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  9. 2021Deciding What to Learn: A Rate-Distortion Approachcited by 23
  10. 2021The Neural Testbed: Evaluating Joint Predictionscited by 22
  11. 2022Evaluating High-Order Predictive Distributions in Deep Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 17; 2026: 13.

44 papers.

Sort:

By area:

  1. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  2. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  3. 2024Cost-Effective Online Multi-LLM Selection with Versatile Reward Modelscited by 36
  4. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  5. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  6. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  7. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  8. 2024Sample-Efficient Alignment for LLMscited by 16
  9. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  10. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 16
  11. 2025Toward Efficient Exploration by Large Language Model Agentscited by 15
  12. 2025Efficient Process Reward Model Training via Active Learningcited by 14
32 more
  1. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  2. 2024On the Importance of Uncertainty in Decision-Making with Large Language Modelscited by 9
  3. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Modelscited by 9
  4. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  5. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
  6. 2024LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Banditscited by 7
  7. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 7
  8. 2025Large Language Model-Enhanced Multi-Armed Banditscited by 7
  9. 2025LLM-Powered Preference Elicitation in Combinatorial Assignmentcited by 7
  10. 2025From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignmentcited by 6
  11. 2025Feel-Good Thompson Sampling for Contextual Bandits: a Markov Chain Monte Carlo Showdowncited by 4
  12. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  13. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3
  14. 2025Interpretable Reward Modeling with Active Concept Bottleneckscited by 3
  15. 2025FedPOB: Sample-Efficient Federated Prompt Optimization via Banditscited by 3
  16. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  17. 2024Active Few-Shot Fine-Tuningcited by 2
  18. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  19. 2024Temporal-Difference Variational Continual Learningcited by 1
  20. 2025Prompt Optimization with Logged Bandit Datacited by 1
  21. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  22. 2026ActiveUltraFeedback: Efficient Preference Data Generation using Active Learningcited by 1
  23. 2026Online Learning and Equilibrium Computation with Ranking Feedbackcited by 1
  24. 2026A Unifying Lens on Reward Uncertainty in RLHFcited by 1
  25. 2024Exploration Unboundcited by 0
  26. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignmentcited by 0
  27. 2026Efficient Exploration at Scalecited by 0
  28. 2026Data-dependent Exploration for Online Reinforcement Learning from Human Feedbackcited by 0
  29. 2026Optimality of Sub-network Laplace Approximations: New Results and Methodscited by 0
  30. 2026Epistemic Uncertainty for Test-Time Discoverycited by 0
  31. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  32. 2026Specifying Reward Functions for RL Without Environment Samplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration
  2. 2025The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling
  3. 2025Large Language Models Think Too Fast To Explore Effectively
  4. 2025Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning
  5. 2024Can large language models explore in-context?
  6. 2025Should You Use Your Large Language Model to Explore or Exploit?
  7. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
  8. 2025First Return, Entropy-Eliciting Explore