mlsanityData to 7 Oct 2026

Semi-Offline Reinforcement Learning for Optimized Text Generation

arXiv 2306.09712 ↗First posted 16 Jun 2023cs.LG cs.AI cs.CLcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In reinforcement learning (RL), there are two major settings for interacting with the environment: online and offline. Online methods explore the environment at significant time cost, and offline methods efficiently obtain reward signals by sacrificing exploration capability. We propose semi-offline RL, a novel paradigm that smoothly transits from offline to online settings, balances exploration capability and training cost, and provides a theoretical foundation for comparing different RL settings. Based on the semi-offline formulation, we present the RL setting that is optimal in terms of optimization cost, asymptotic error, and overfitting error bound. Extensive experiments show that our semi-offline approach is efficient and yields comparable or often better performance compared with state-of-the-art methods.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 55 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

33 of 55 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  10. 2015Teaching Machines to Read and Comprehendcited by 1,039
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
21 more
  1. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  2. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  3. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  4. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  5. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  6. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  7. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  8. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  9. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  10. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  11. 2020Text Generation by Learning from Demonstrationscited by 64
  12. 2019Scheduled Sampling for Transformerscited by 46
  13. 2017A Deep Reinforcement Learning Chatbotcited by 36
  14. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  15. 2022Generative Language Models for Paragraph-Level Question Generationcited by 28
  16. 2019Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarizationcited by 24
  17. 2017On overfitting and asymptotic bias in batch reinforcement learning with partial observabilitycited by 16
  18. 2017Cold-Start Reinforcement Learning with Softmax Policy Gradientcited by 15
  19. 2022Reinforcement Routing on Proximity Graph for Efficient Recommendationcited by 5
  20. 2022Generating Multiple-Length Summaries via Reinforcement Learning for Unsupervised Sentence Summarizationcited by 5
  21. 2018Connecting the Dots Between MLE and RL for Sequence Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 0; 2026: 2.

12 papers.

  1. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 92
  2. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  3. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 63
  4. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  5. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  6. 2024ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understandingcited by 23
  7. 2024Prototypical Reward Network for Data-Efficient RLHFcited by 22
  8. 2024Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Modelscited by 20
  9. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  10. 2024Real-time system optimal traffic routing under uncertainties -- Can physics models boost reinforcement learning?cited by 6
  11. 2026Efficient and Stable Reinforcement Learning for Diffusion Language Modelscited by 2
  12. 2026Aligning the Query Space: Greedy Information Projection for Language Model Data Selectioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Leveraging Offline Data in Online Reinforcement Learning
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  3. 2021A Minimalist Approach to Offline Reinforcement Learning
  4. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  5. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  6. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  7. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  8. 2025A Clean Slate for Offline Reinforcement Learning