mlsanityData to 14 Oct 2026

Information-Theoretic Confidence Bounds for Reinforcement Learning

arXiv 1911.09724 ↗First posted 21 Nov 2019stat.ML cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

We integrate information-theoretic concepts into the design and analysis of optimistic algorithms and Thompson sampling. By making a connection between information-theoretic quantities and confidence bounds, we obtain results that relate the per-period performance of the agent with its information gain about the environment, thus explicitly characterizing the exploration-exploitation tradeoff. The resulting cumulative regret bound depends on the agent's uncertainty over the environment and quantifies the value of prior information. We show applicability of this approach to several environments, including linear bandits, tabular MDPs, and factored MDPs. These examples demonstrate the potential of a general information-theoretic approach for the design and analysis of reinforcement learning algorithms.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 10; 2023: 13; 2024: 10; 2025: 3; 2026: 4.

51 papers.

Sort:

By area:

  1. 2021Meta-Thompson Samplingcited by 63
  2. 2021Reinforcement Learning, Bit by Bitcited by 59
  3. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  4. 2021Hierarchical Bayesian Banditscited by 35
  5. 2021No Regrets for Learning the Prior in Banditscited by 30
  6. 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
  7. 2021Information Directed Sampling for Sparse Linear Banditscited by 20
  8. 2022Deep Hierarchy in Banditscited by 19
  9. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  10. 2025Toward Efficient Exploration by Large Language Model Agentscited by 17
  11. 2021Thompson Sampling with a Mixture Priorcited by 16
  12. 2021Metalearning Linear Bandits by Prior Updatecited by 16
38 more
  1. 2021PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Explorationcited by 14
  2. 2022Mixed-Effect Thompson Samplingcited by 14
  3. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  4. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  5. 2022Towards Scalable and Robust Structured Bandits: A Meta-Learning Frameworkcited by 11
  6. 2022Meta-Learning for Simple Regret Minimizationcited by 10
  7. 2022Graph Neural Network Banditscited by 10
  8. 2025On Rollouts in Model-Based Reinforcement Learningcited by 10
  9. 2022Multi-Task Off-Policy Learning from Bandit Feedbackcited by 9
  10. 2023Social Contract AI: Aligning AI Assistants with Implicit Group Normscited by 9
  11. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Modelscited by 8
  12. 2023Thompson Sampling with Diffusion Generative Priorcited by 7
  13. 2023Improved Bayesian Regret Bounds for Thompson Sampling in Reinforcement Learningcited by 7
  14. 2022On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learningcited by 6
  15. 2024Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgentcited by 6
  16. 2021A Bit Better? Quantifying Information for Bandit Learningcited by 5
  17. 2022Gaussian Imagination in Bandit Learningcited by 5
  18. 2024Diffusion Models Meet Contextual Banditscited by 5
  19. 2025Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RLcited by 5
  20. 2021Random Effect Banditscited by 4
  21. 2023STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learningcited by 4
  22. 2023Bayesian Reinforcement Learning with Limited Cognitive Loadcited by 4
  23. 2024Satisficing Exploration for Deep Reinforcement Learningcited by 4
  24. 2023Only Pay for What Is Uncertain: Variance-Adaptive Thompson Samplingcited by 3
  25. 2024On Bits and Bandits: Quantifying the Regret-Information Trade-offcited by 3
  26. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  27. 2024Prior-dependent analysis of posterior sampling reinforcement learning with function approximationcited by 2
  28. 2024Provably Efficient Information-Directed Sampling Algorithms for Multi-Agent Reinforcement Learningcited by 2
  29. 2024Online Posterior Sampling with a Diffusion Priorcited by 2
  30. 2026Cognitive Models and AI Algorithms Provide Templates for Designing Language Agentscited by 2
  31. 2021Online Sub-Sampling for Reinforcement Learning with General Function Approximationcited by 1
  32. 2023Overcoming Prior Misspecification in Online Learning to Rankcited by 1
  33. 2024Learning with Posterior Sampling for Revenue Management under Time-varying Demandcited by 1
  34. 2026Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learningcited by 1
  35. 2023Exploration in Model-based Reinforcement Learning with Randomized Rewardcited by 0
  36. 2023Finite-Time Logarithmic Bayes Regret Upper Boundscited by 0
  37. 2023Interactive Graph Convolutional Filteringcited by 0
  38. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0

50 most cited of 51 All 51 (JSON)