mlsanityData to 14 Oct 2026

RL theory and sample complexity

1,380 papers7 this weekLow confidence in this grouping: the papers may belong to more than one topic, or the group may changeLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About

Keywords: policy, reinforcement learning, reinforcement, algorithm, algorithms

Key facts. RL theory and sample complexity: 1,380 papers on arXiv since 2024 (data through 14 Oct 2026). 17 of its 1,380 mapped papers (1%) cite “The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise” (2024, arXiv 2401.07844). 43 new papers 30 Sep–14 Oct.

What’s new

30 Sep – 14 Oct 2026

43 new papers, 30 Sep – 14 Oct; 7 this week.

  1. This week · 7
  2. Average-Reward Policy Identification from a Single Continuing Trajectory also fits: Deep RL algorithms and representations
  3. Non-asymptotic Convergence of Average-reward Q-learning with Options
  4. On the Convergence of Semi-Gradient TD: A Mean Field Perspective
  5. What Information Buys in Active Inference also fits: Deep RL algorithms and representations
  6. Fast Rates for Offline Q-Learning with KL Regularization
  7. Stein Variational Distributional Reinforcement Learning: Score-Guided Particles for Multivariate Return Distributions also fits: RL with diffusion and flow models
  8. Density-Matrix Reinforcement Learning also fits: Deep RL algorithms and representations
  9. Earlier · 36
  10. Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning also fits: RL for games and combinatorial optimization
  11. Mixing Regularization for Average-Reward Q-Learning and Q-Planning
  12. Hitting-Time Bounds for Asynchronous Q-Learning
All 43
  1. Partially Informed Stochastic Optimization via Zeroth-Order Residual Tracking
  2. Variance-Reduced Policy Gradient is Nearly Minimax-Optimal
  3. Structure-Aware Numerical Stabilization for Half-Precision Proximal Policy Optimization also fits: RL for energy, networks and operations
  4. Toward Optimal Regret in Adversarial MDPs with Stochastic Hard Constraints
  5. Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control
  6. Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach also fits: RL for games and combinatorial optimization
  7. A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
  8. Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradient
  9. Convex-Concave Reinforcement Learning
  10. Reinforcement Learning with Segment Reward Feedback under Linear Function Approximation also fits: Deep RL algorithms and representations
  11. Adapting to Changes in Agent Behavior via Finite-Depth Policy Sensitivity also fits: Deep RL algorithms and representations
  12. Optimal Control with Learned Critics under Unmodeled State Dependencies also fits: RL for robot navigation and UAV control
  13. Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Model
  14. Variance-Aware Fine-Grained Gap-Dependent Bounds for Online Reinforcement Learning
  15. Exact Fast Batch Simulation for Tabular Reinforcement Learning also fits: Deep RL algorithms and representations
  16. Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression also fits: Deep RL algorithms and representations
  17. A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning also fits: Deep RL algorithms and representations
  18. Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion also fits: Deep RL algorithms and representations
  19. Single or Multiple Policies for Phase-Structured Reinforcement Learning?
  20. Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds
  21. Bellman Error Minimization Via Linear Programming Normalization also fits: RL for games and combinatorial optimization
  22. Instance-Dependent Regret for CMDPs with Step-Wise Constraints also fits: Safe and constrained RL
  23. Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning also fits: RL with diffusion and flow models
  24. FERPO: Forward Entropy-Regularized Policy Optimization
  25. Q-Learning for Reachability in MEC-Free MDPs
  26. Towards Optimal Policy Improvement
  27. Rethinking Probability-Based Reinforcement Learning From Posterior Concentration also fits: Deep RL algorithms and representations
  28. Rate-Optimal Algorithm for Adversarial Linear CMDPs
  29. Learning Goal-Reaching Quasimetric Geometry From Finite-Time Reachability also fits: Deep RL algorithms and representations
  30. Learning Infinite-Horizon Average-Reward CMDPs via State Augmentation
  31. Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching also fits: Deep RL algorithms and representations
  32. Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback also fits: Safe and constrained RL
  33. DAMPER: Return-Prioritized Gradient Control for Smooth Policies also fits: Deep RL algorithms and representations

Foundations

Most cited by papers here

  1. The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise2024cited by 17 here
  2. Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning2024cited by 13 here
  3. Elementary Analysis of Policy Gradient Methods2024cited by 12 here
  4. Truly No-Regret Learning in Constrained MDPs2024cited by 11 here
  5. More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning2024cited by 10 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 9 here
  2. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 15 here
  3. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 18 here
  4. 2024Simplifying Deep Temporal Difference Learningcited by 14 here
  5. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 11 here
19 more it builds on, not in this chain
  1. 2024The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noisecited by 17 here
  2. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 9 here
  3. 2024Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolationcited by 8 here
  4. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 10 here
  5. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 8 here
  6. 2024Truly No-Regret Learning in Constrained MDPscited by 11 here
  7. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 13 here
  8. 2024Learning Adversarial MDPs with Stochastic Hard Constraintscited by 8 here
  9. 2024Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPscited by 8 here
  10. 2024Elementary Analysis of Policy Gradient Methodscited by 12 here
  11. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 10 here
  12. 2024Reinforcement learningcited by 32 here
  13. 2024Sample-Efficient Constrained Reinforcement Learning with General Parameterizationcited by 9 here
  14. 2024Reinforcement Learning for Jump-Diffusions, with Financial Applicationscited by 12 here
  15. 2024Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximationcited by 10 here
  16. 2024Gymnasium: A Standard Interface for Reinforcement Learning Environmentscited by 53 here
  17. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 18 here
  18. 2024Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noisecited by 9 here
  19. 2025Linear Q-Learning Does Not Diverge in L2: Convergence Rates to a Bounded Setcited by 10 here

The 24 most cited of 913.

Browse

30 of 1,380, ranked 6–35

  1. 6Linear Q-Learning Does Not Diverge in L2: Convergence Rates to a Bounded Set202510
  2. 7Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms202410
  3. 8Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation202410
  4. 9Sample-Efficient Constrained Reinforcement Learning with General Parameterization20249
  5. 10Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise20249
  6. 11Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolation20248
  7. 12Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation20248
  8. 13Learning Adversarial MDPs with Stochastic Hard Constraints20248
  9. 14Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPs20248
  10. 15Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization20248
All 30
  1. 16A Sharper Global Convergence Analysis for Average Reward Reinforcement Learning via an Actor-Critic Approach20248
  2. 17Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning20257
  3. 18Averaging n-step Returns Reduces Variance in Reinforcement Learning20247
  4. 19A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents20247
  5. 20Nonparametric Bellman Mappings for Reinforcement Learning: Application to Robust Adaptive Filtering20247
  6. 21Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption20247
  7. 22Optimistic Q-learning for average reward and episodic reinforcement learning20247
  8. 23Stochastic Approximation with Unbounded Markovian Noise: A General-Purpose Theorem20247
  9. 24Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning20256
  10. 25Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration20256
  11. 26Wasserstein Policy Optimization20256
  12. 27Global Convergence for Average Reward Constrained MDPs with Primal-Dual Actor Critic Algorithm20256
  13. 28Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems20256
  14. 29Simple Policy Optimization20246
  15. 30Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm20246
  16. 31Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts20246
  17. 32SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning20246
  18. 33Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity20246
  19. 34Distributionally Robust Constrained Reinforcement Learning under Strong Duality20246
  20. 35Model-Free Robust Reinforcement Learning with Sample Complexity Analysis20246