mlsanityData to 14 Oct 2026

Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusing

arXiv 2509.08329 ↗First posted 10 Sep 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) algorithms often require long training to become useful, especially in complex environments with sparse rewards. While techniques like reward shaping and curriculum learning exist to accelerate training, these are often extremely specific and require the developer's professionalism and dedicated expertise in the problem's domain. Tackling this challenge, in this study, we explore the effectiveness of pre-trained Large Language Models (LLMs) as tutors in a student-teacher architecture with RL algorithms, hypothesizing that LLM-generated guidance allows for faster convergence. In particular, we explore the effectiveness of reusing the LLM's advice on the RL's convergence dynamics. Through an extensive empirical examination, which included 54 configurations, varying the RL algorithm (DQN, PPO, A2C), LLM tutor (Llama, Vicuna, DeepSeek), and environment (Blackjack, Snake, Connect Four), our results demonstrate that LLM tutoring significantly accelerates RL convergence while maintaining comparable optimal performance. Furthermore, the advice reuse mechanism shows a further improvement in training duration but also results in less stable convergence dynamics. Our findings suggest that LLM tutoring generally improves convergence, and its effectiveness is sensitive to the specific task, RL algorithm, and LLM model combination.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 56 of its 106 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

56 of 106 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
38 more
  1. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  2. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  3. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  4. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 337
  5. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  6. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  7. 2020A Survey on the Explainability of Supervised Machine Learningcited by 227
  8. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  9. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  10. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 116
  11. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  12. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  13. 2020FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Financecited by 87
  14. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  15. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  16. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  17. 2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
  18. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  19. 2018Deep Reinforcement Learning For Sequence to Sequence Modelscited by 39
  20. 2023Large Language Models for Scientific Synthesis, Inference and Explanationcited by 39
  21. 2021Analyzing Commonsense Emergence in Few-shot Knowledge Modelscited by 34
  22. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  23. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 29
  24. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  25. 2024iLLM-TSC: Integration reinforcement learning and large language model for traffic signal control policy improvementcited by 23
  26. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  27. 2024Review-driven Personalized Preference Reasoning with Large Language Models for Recommendationcited by 22
  28. 2024ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Modelscited by 19
  29. 2022Reward Modeling for Mitigating Toxicity in Transformer-based Language Modelscited by 18
  30. 2024DiarizationLM: Speaker Diarization Post-Processing with Large Language Modelscited by 16
  31. 2024Distributed Deep Reinforcement Learning Based Gradient Quantization for Federated Learning Enabled Vehicle Edge Computingcited by 15
  32. 2024How Can LLM Guide RL? A Value-Based Approachcited by 13
  33. 2023Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Modelscited by 11
  34. 2024Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challengescited by 10
  35. 2023LLM Augmented Hierarchical Agentscited by 9
  36. 2024RL-Pruner: Structured Pruning Using Reinforcement Learning for CNN Compression and Accelerationcited by 9
  37. 2025Align to Structure: Aligning Large Language Models with Structural Informationcited by 6
  38. 2018Deep Reinforcement Learning for Dynamic Urban Transportation Problemscited by 5

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  2. 2026Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning
  3. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  4. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  5. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  7. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
  8. 2025The Art of Scaling Reinforcement Learning Compute for LLMs