mlsanityData to 14 Oct 2026

Reinforcement Learning, Bit by Bit

arXiv 2103.04047 ↗First posted 6 Mar 2021cs.LG cs.AIcited by 59

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning agents have demonstrated remarkable achievements in simulated environments. Data efficiency poses an impediment to carrying this success over to real environments. The design of data-efficient agents calls for a deeper understanding of information acquisition and representation. We discuss concepts and regret analysis that together offer principled guidance. This line of thinking sheds light on questions of what information to seek, how to seek that information, and what information to retain. To illustrate concepts, we design simple agents that build on them and present computational results that highlight data efficiency.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 45 of its 103 references that are in the map. First posted on arXiv 6 Mar 2021.

Builds on

45 of 103 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2016Deep Exploration via Bootstrapped DQNcited by 527
  5. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  6. 2018Is Q-learning Provably Efficient?cited by 487
  7. 2017A Tutorial on Thompson Samplingcited by 433
  8. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  9. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  11. 2013Learning to Optimize Via Posterior Samplingcited by 256
  12. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
33 more
  1. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  2. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  3. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  4. 2017Deep Exploration via Randomized Value Functionscited by 156
  5. 2019Behaviour Suite for Reinforcement Learningcited by 132
  6. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  7. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  8. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 94
  9. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  10. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  11. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 58
  12. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  13. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  14. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  15. 2019An Information-Theoretic Approach to Minimax Regret in Partial Monitoringcited by 42
  16. 2020Information Directed Sampling for Linear Partial Monitoringcited by 40
  17. 2020Mirror Descent and the Information Ratiocited by 40
  18. 2018Satisficing in Time-Sensitive Bandit Learningcited by 39
  19. 2020Hypermodels for Explorationcited by 37
  20. 2018An Information-Theoretic Analysis for Thompson Sampling with Many Actionscited by 35
  21. 2020Asymptotically Optimal Information-Directed Samplingcited by 33
  22. 2019On the Performance of Thompson Sampling on Logistic Banditscited by 32
  23. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  24. 2020Neural networks with late-phase weightscited by 29
  25. 2017Information Directed Sampling for Stochastic Bandits with Graph Feedbackcited by 24
  26. 2019Connections Between Mirror Descent, Thompson Sampling and the Information Ratiocited by 23
  27. 2021Deciding What to Learn: A Rate-Distortion Approachcited by 23
  28. 2015Multi-scale exploration of convex functions and bandit convex optimizationcited by 20
  29. 2019First-Order Bayesian Regret Analysis of Thompson Samplingcited by 17
  30. 2020Temporal Difference Uncertainties as a Signal for Explorationcited by 15
  31. 2015Bandit Convex Optimization: sqrt{T} Regret in One Dimensioncited by 10
  32. 2021A Bit Better? Quantifying Information for Bandit Learningcited by 5
  33. 2016Bayesian Adaptive Data Analysis Guarantees from Subgaussianitycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 16; 2023: 16; 2024: 10; 2025: 6; 2026: 3.

59 papers.

Sort:

By area:

  1. 2021Epistemic Neural Networkscited by 120
  2. 2023A Definition of Continual Reinforcement Learningcited by 101
  3. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 37
  4. 2022Settling the Reward Hypothesiscited by 33
  5. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  6. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 29
  7. 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
  8. 2021The Neural Testbed: Evaluating Joint Predictionscited by 22
  9. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  10. 2022Non-Stationary Bandit Learning via Predictive Samplingcited by 21
  11. 2021Information Directed Sampling for Sparse Linear Banditscited by 20
  12. 2022Lifting the Information Ratio: An Information-Theoretic Analysis of Thompson Sampling for Contextual Banditscited by 19
38 more
  1. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  2. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  3. 2025Toward Efficient Exploration by Large Language Model Agentscited by 17
  4. 2022Contextual Information-Directed Samplingcited by 16
  5. 2021The Value of Information When Deciding What to Learncited by 15
  6. 2024Three Dogmas of Reinforcement Learningcited by 14
  7. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  8. 2023Linear Partial Monitoring for Sequential Decision-Making: Algorithms, Regret Bounds and Applicationscited by 12
  9. 2025Plasticity as the Mirror of Empowermentcited by 12
  10. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
  11. 2022Reinforcement Learning in Non-Markovian Environmentscited by 10
  12. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  13. 2023Efficient Exploration via Epistemic-Risk-Seeking Policy Optimizationcited by 10
  14. 2025On Rollouts in Model-Based Reinforcement Learningcited by 10
  15. 2025BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RLcited by 10
  16. 2023A Definition of Non-Stationary Banditscited by 9
  17. 2024Agent-state based policies in POMDPs: Beyond belief-state MDPscited by 9
  18. 2021Learning and Information in Stochastic Networks and Queuescited by 8
  19. 2023Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspectivecited by 8
  20. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Modelscited by 8
  21. 2023On the Convergence of Bounded Agentscited by 7
  22. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  23. 2022An Information-Theoretic Framework for Supervised Learningcited by 6
  24. 2022On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learningcited by 6
  25. 2023Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learningcited by 6
  26. 2024Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgentcited by 6
  27. 2024Periodic agent-state based Q-learning for POMDPscited by 6
  28. 2022Gaussian Imagination in Bandit Learningcited by 5
  29. 2022On learning history based policies for controlling Markov decision processescited by 5
  30. 2023An Invitation to Deep Reinforcement Learningcited by 5
  31. 2022Planning to the Information Horizon of BAMDPs via Epistemic State Abstractioncited by 4
  32. 2022Posterior Sampling for Continuing Environmentscited by 4
  33. 2023STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learningcited by 4
  34. 2023Bayesian Reinforcement Learning with Limited Cognitive Loadcited by 4
  35. 2024Satisficing Exploration for Deep Reinforcement Learningcited by 4
  36. 2024Value of Information and Reward Specification in Active Inference and POMDPscited by 4
  37. 2023Conditions on Preference Relations that Guarantee the Existence of Optimal Policiescited by 3
  38. 2023Efficient Online Decision Tree Learning with Active Feature Acquisitioncited by 2

50 most cited of 59 All 59 (JSON)