# Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent

arXiv: https://arxiv.org/abs/2007.07422 · HTML: https://mlsanity.com/paper/2007.07422/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2020.

## Built on by (most cited first, 26 of 26)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [Personalized Prompt Learning for Explainable Recommendation](https://mlsanity.com/paper/2202.07371/) (2022)
- [Counterfactual Explainable Recommendation](https://mlsanity.com/paper/2108.10539/) (2021)
- [A Survey on Accuracy-oriented Neural Recommendation: From Collaborative Filtering to Information-rich Recommendation](https://mlsanity.com/paper/2104.13030/) (2021)
- [Personalized Transformer for Explainable Recommendation](https://mlsanity.com/paper/2105.11601/) (2021)
- [RevCore: Review-augmented Conversational Recommendation](https://mlsanity.com/paper/2106.00957/) (2021)
- [LLMs for Explainable AI: A Comprehensive Survey](https://mlsanity.com/paper/2504.00125/) (2025)
- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Large Language Models as Evaluators for Recommendation Explanations](https://mlsanity.com/paper/2406.03248/) (2024)
- [TREA: Tree-Structure Reasoning Schema for Conversational Recommendation](https://mlsanity.com/paper/2307.10543/) (2023)
- [Measuring "Why" in Recommender Systems: a Comprehensive Survey on the Evaluation of Explainable Recommendation](https://mlsanity.com/paper/2202.06466/) (2022)
- [Zero-shot Persuasive Chatbots with LLM-Generated Strategies and Information Retrieval](https://mlsanity.com/paper/2407.03585/) (2024)
- [Self-explaining deep models with logic rule reasoning](https://mlsanity.com/paper/2210.07024/) (2022)
- [Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizes](https://mlsanity.com/paper/2211.13638/) (2022)
- [Beyond Persuasion: Towards Conversational Recommender System with Credible Explanations](https://mlsanity.com/paper/2409.14399/) (2024)
- [Fast Multi-Step Critiquing for VAE-based Recommender Systems](https://mlsanity.com/paper/2105.00774/) (2021)
- [Towards Explainable Conversational Recommender Systems](https://mlsanity.com/paper/2305.18363/) (2023)
- [Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI](https://mlsanity.com/paper/2308.06274/) (2023)
- [EGCR: Explanation Generation for Conversational Recommendation](https://mlsanity.com/paper/2208.08035/) (2022)
- [Category Aware Explainable Conversational Recommendation](https://mlsanity.com/paper/2103.08733/) (2021)
- [Quality-aware News Recommendation](https://mlsanity.com/paper/2202.13605/) (2022)
- [Improving Items and Contexts Understanding with Descriptive Graph for Conversational Recommendation](https://mlsanity.com/paper/2304.09093/) (2023)
- [Salespeople vs SalesBot: Exploring the Role of Educational Value in Conversational Recommender Systems](https://mlsanity.com/paper/2310.17749/) (2023)
- [Cross-domain User Preference Learning for Cold-start Recommendation](https://mlsanity.com/paper/2112.03667/) (2021)
- [Heavy-Ball Q-Learning with Residual Weighting Correction](https://mlsanity.com/paper/2606.27112/) (2026)

## Abstract (arXiv)

Existing convergence analyses of Q-learning mostly focus on the vanilla stochastic gradient descent (SGD) type of updates. Despite the Adaptive Moment Estimation (Adam) has been commonly used for practical Q-learning algorithms, there has not been any convergence guarantee provided for Q-learning with such type of updates. In this paper, we first characterize the convergence rate for Q-AMSGrad, which is the Q-learning algorithm with AMSGrad update (a commonly adopted alternative of Adam for theoretical analysis). To further improve the performance, we propose to incorporate the momentum restart scheme to Q-AMSGrad, resulting in the so-called Q-AMSGradR algorithm. The convergence rate of Q-AMSGradR is also established. Our experiments on a linear quadratic regulator problem show that the two proposed Q-learning algorithms outperform the vanilla Q-learning with SGD updates. The two algorithms also exhibit significantly better performance than the DQN learning method over a batch of Atari 2600 games.