mlsanityData to 14 Oct 2026

Policy Learning with a Natural Language Action Space: A Causal Approach

arXiv 2502.17538 ↗First posted 24 Feb 2025cs.CLcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

This paper introduces a novel causal framework for multi-stage decision-making in natural language action spaces where outcomes are only observed after a sequence of actions. While recent approaches like Proximal Policy Optimization (PPO) can handle such delayed-reward settings in high-dimensional action spaces, they typically require multiple models (policy, value, and reward) and substantial training data. Our approach employs Q-learning to estimate Dynamic Treatment Regimes (DTR) through a single model, enabling data-efficient policy learning via gradient ascent on language embeddings. A key technical contribution of our approach is a decoding strategy that translates optimized embeddings back into coherent natural language. We evaluate our approach on mental health intervention, hate speech countering, and sentiment transfer tasks, demonstrating significant improvements over competitive baselines across multiple metrics. Notably, our method achieves superior transfer strength while maintaining content preservation and fluency, as validated through human evaluation. Our work provides a practical foundation for learning optimal policies in complex language tasks where training data is limited.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 58 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

37 of 58 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2015Continuous control with deep reinforcement learningcited by 3,407
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
25 more
  1. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
  2. 2014Semi-Supervised Learning with Deep Generative Modelscited by 496
  3. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 410
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  5. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  6. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  7. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  8. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  9. 2021Desiderata for Representation Learning: A Causal Perspectivecited by 76
  10. 2018How to Make Causal Inferences Using Textscited by 59
  11. 2022Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Modelscited by 49
  12. 2018BanditSum: Extractive Summarization as a Contextual Banditcited by 46
  13. 2020Counterfactual Fairness with Disentangled Causal Effect Variational Autoencodercited by 45
  14. 2017Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learningcited by 44
  15. 2022Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Counteringcited by 40
  16. 2020Causal Effects of Linguistic Propertiescited by 39
  17. 2020Unsupervised Text Generation by Learning from Searchcited by 36
  18. 2022A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learningcited by 30
  19. 2021MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processescited by 19
  20. 2023Prompt-Based Editing for Text Style Transfercited by 16
  21. 2024Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatmentscited by 16
  22. 2023STEER: Unified Style Transfer with Expert Reinforcementcited by 14
  23. 2019Budgeted Policy Learning for Task-Oriented Dialogue Systemscited by 13
  24. 2019Interactive-Predictive Neural Machine Translation through Reinforcement and Imitationcited by 7
  25. 2024Causal Inference for Human-Language Model Collaborationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement
  2. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  3. 2025Causally-Enhanced Reinforcement Policy Optimization
  4. 2025Scaling Reinforcement Learning for Content Moderation with Large Language Models
  5. 2024Reinforcing Language Agents via Policy Optimization with Action Decomposition
  6. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
  7. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
  8. 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space