Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
Optimizing large language models for industrial sales requires balancing long-term commercial objectives (e.g., conversion rate) with immediate linguistic constraints such as fluency and compliance. Conventional reinforcement learning often merges these heterogeneous goals into a single reward, causing high-magnitude session-level rewards to overwhelm subtler turn-level signals, which leads to unstable training or reward hacking. To address this issue, we propose Dual-Horizon Credit Assignment (DuCA), a framework that disentangles optimization across time scales. Its core, Horizon-Independent Advantage Normalization (HIAN), separately normalizes advantages from turn-level and session-level rewards before fusion, ensuring balanced gradient contributions from both immediate and long-term objectives to the policy update. Extensive experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline, achieving a 6.82% relative improvement in conversion rate, reducing inter-sentence repetition by 82.28%, and lowering identity detection rate by 27.35%, indicating a substantial improvement for an industrial sales scenario that effectively balances the dual demands of strategic performance and naturalistic language generation.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 15 references that are in the map. First posted on arXiv 2 Mar 2026.
Builds on
10 of 15 references.
Closest ancestors · 3
Same direction, newest first
- 2025Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents17 Sep
- 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment17 May
- 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
Other references · 7
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
- 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6
- 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modelingcited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
- 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
- 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
- 2025SalesRLAgent: A Reinforcement Learning Approach for Real-Time Sales Conversion Prediction and Optimization
- 2026Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
- 2026Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets
- 2026Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment
- 2026Hindsight Credit Assignment for Long-Horizon LLM Agents