mlsanityData to 14 Oct 2026

Learning to Optimize via Information-Directed Sampling

arXiv 1403.5556 ↗First posted 21 Mar 2014cs.LGcited by 168

Not on the map. Why?

Abstract (arXiv)

We propose information-directed sampling -- a new approach to online optimization problems in which a decision-maker must balance between exploration and exploitation while learning from partial feedback. Each action is sampled in a manner that minimizes the ratio between squared expected single-period regret and a measure of information gain: the mutual information between the optimal action and the next observation. We establish an expected regret bound for information-directed sampling that applies across a very general class of models and scales with the entropy of the optimal action distribution. We illustrate through simple analytic examples how information-directed sampling accounts for kinds of information that alternative approaches do not adequately address and that this can lead to dramatic performance gains. For the widely studied Bernoulli, Gaussian, and linear bandit problems, we demonstrate state-of-the-art simulation performance.

Key facts. Cited by 168 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Mar 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 22; 2022: 31; 2023: 26; 2024: 24; 2025: 26; 2026: 38.

168 papers.

Sort:

By area:

  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  2. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  3. 2021Reinforcement Learning, Bit by Bitcited by 59
  4. 2022Joint Entropy Search for Maximally-Informed Bayesian Optimizationcited by 55
  5. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  6. 2024Efficient Exploration for LLMscited by 45
  7. 2022On the Complexity of Adversarial Decision Makingcited by 33
  8. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  9. 2023Tight Guarantees for Interactive Decision Making with the Decision-Estimation Coefficientcited by 30
  10. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  11. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 29
  12. 2024A Unified Confidence Sequence for Generalized Linear Models, with Applications to Banditscited by 29
38 more
  1. 2021Information Directed Reward Learning for Reinforcement Learningcited by 24
  2. 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
  3. 2021Deciding What to Learn: A Rate-Distortion Approachcited by 23
  4. 2022From Dirichlet to Rubin: Optimistic Exploration in RL without Bonusescited by 22
  5. 2022Non-Stationary Bandit Learning via Predictive Samplingcited by 21
  6. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  7. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  8. 2021Information Directed Sampling for Sparse Linear Banditscited by 20
  9. 2022Lifting the Information Ratio: An Information-Theoretic Analysis of Thompson Sampling for Contextual Banditscited by 19
  10. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  11. 2023Bayesian Design Principles for Frequentist Sequential Learningcited by 18
  12. 2025Toward Efficient Exploration by Large Language Model Agentscited by 17
  13. 2021Metalearning Linear Bandits by Prior Updatecited by 16
  14. 2021An Experimental Design Perspective on Model-Based Reinforcement Learningcited by 16
  15. 2022Contextual Information-Directed Samplingcited by 16
  16. 2021The Value of Information When Deciding What to Learncited by 15
  17. 2022Information-Directed Selection for Top-Two Algorithmscited by 15
  18. 2021Deep Exploration for Recommendation Systemscited by 14
  19. 2022Minimax Regret for Partial Monitoring: Infinite Outcomes and Rustichini's Regretcited by 14
  20. 2022Modeling Human Exploration Through Resource-Rational Reinforcement Learningcited by 13
  21. 2022Artificial Replay: A Meta-Algorithm for Harnessing Historical Data in Banditscited by 13
  22. 2022Bayesian Fixed-Budget Best-Arm Identificationcited by 13
  23. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
  24. 2025Improved Regret Bounds for Gaussian Process Upper Confidence Bound in Bayesian Optimizationcited by 13
  25. 2021An empirical evaluation of active inference in multi-armed banditscited by 12
  26. 2021Bandit Phase Retrievalcited by 12
  27. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  28. 2022Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guaranteescited by 12
  29. 2023Linear Partial Monitoring for Sequential Decision-Making: Algorithms, Regret Bounds and Applicationscited by 12
  30. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 12
  31. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  32. 2021Bias-Robust Bayesian Optimization via Dueling Banditscited by 11
  33. 2021Rate-optimal Bayesian Simple Regret in Best Arm Identificationcited by 11
  34. 2023An Information-Theoretic Analysis of Nonstationary Bandit Learningcited by 11
  35. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
  36. 2023A Second-Order Method for Stochastic Bandit Convex Optimisationcited by 10
  37. 2023A Definition of Non-Stationary Banditscited by 9
  38. 2025Gaussian Process Upper Confidence Bound Achieves Nearly-Optimal Regret in Noise-Free Gaussian Process Banditscited by 9

50 most cited of 168 All 168 (JSON)