mlsanityData to 14 Oct 2026

ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

arXiv 2608.21969 ↗First posted 22 Aug 2026cs.CL cs.HC cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Humans naturally exhibit multiple forms of abstraction in reasoning and interaction, including temporal abstraction across decision timescales and strategic abstraction over communicative intents. Inspired by these complementary abstractions, we propose a two-level hierarchical reinforcement learning (HRL) framework for conversational agents that bridges the gap between existing token-level and utterance-level RL methods. Built upon a two-level Markov decision process (MDP), our framework conditions token-level response generation on utterance-level actions represented by explicit textual strategies. Based on theoretical analysis and efficiency considerations, we employ DQN to optimize the high-level Q-network and PPO to train the low-level actor-critic. To further alleviate reward sparsity and facilitate convergence, we introduce a dual-granularity reward mechanism that combines the utterance-level satisfaction score with token-level intrinsic self-consistency and a KL-divergence penalty. Experiments on both daily-life and emotional support conversations demonstrate that our method consistently outperforms a wide range of baselines in both strategy determination and response quality. Our implementation is available at https://github.com/AaronJi/ToSCA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 59 references that are in the map. First posted on arXiv 22 Aug 2026.

Builds on

30 of 59 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  8. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  9. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  10. 2016The Option-Critic Architecturecited by 448
  11. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  12. 2021Towards Emotional Support Dialog Systemscited by 299
18 more
  1. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  2. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  3. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  4. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  5. 2024Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Modelscited by 91
  6. 2024Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversationcited by 55
  7. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  8. 2024Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thoughtcited by 34
  9. 2022Efficient computation of rankings from pairwise comparisonscited by 31
  10. 2024Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Plannercited by 13
  11. 2023Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologuecited by 11
  12. 2022Attention Option-Criticcited by 8
  13. 2020An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generationcited by 7
  14. 2025Convert Language Model into a Value-based Strategic Plannercited by 5
  15. 2022A Systematic Evaluation of Response Selection for Open Domain Dialoguecited by 4
  16. 2025EmoFSM: A Finite State Machine for Emotional Support Conversationcited by 3
  17. 2025Modeling the One-to-Many Property in Open-Domain Dialogue with LLMscited by 3
  18. 2023An Empirical Study of Multitask Learning to Improve Open Domain Dialogue Systemscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Deep RL with Hierarchical Action Exploration for Dialogue Generation
  2. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  3. 2026Enhancing Social Intelligence in LLMs with Hierarchical Reasoning and Utterance-Level Goal Rewarding
  4. 2026Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance
  5. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  6. 2026IB-RL: Isolated Bilateral Reinforcement Learning for Strategic Dialogue Agents
  7. 2026Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents
  8. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback