mlsanityData to 7 Oct 2026

Decoupling Strategy and Execution in Task-Focused Dialogue via Goal-Oriented Preference Optimization

arXiv 2602.15854 ↗First posted 24 Jan 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Large language models show potential in task-oriented dialogue systems, yet existing training methods often rely on token-level likelihood or preference optimization, which poorly align with long-horizon task success. To address this, we propose Goal-Oriented Preference Optimization (GOPO), a hierarchical reinforcement learning framework that decouples strategy planning from response generation via an Expert Agent and a Customer Service Agent. The Expert Agent optimizes multi-turn goal preferences at the dialogue-trajectory level, while the Customer Service Agent generates responses strictly aligned with the selected strategy. We evaluate GOPO on public benchmarks and e-commerce customer service datasets, and introduce Task-focused Sequential Engagement (TSE), a sequence-level metric derived from real e-commerce interaction data. On the Mgshop dataset, GOPO improves TSE by 7.7% and 10.3% over PPO and Memento, with consistent gains in sequence-level reward and generation quality. Furthermore, a 14B model trained with GOPO achieves 2.7% and 1.5% higher TSE than Qwen-235B and GPT-5.2, respectively. Ablation studies confirm the Expert Agent's critical role in long-horizon optimization. GOPO demonstrates consistent improvements across other datasets as well. This work establishes a new paradigm for task-oriented dialogue systems in commercial scenarios, with code and datasets to be made public.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 32 references that are in the map. First posted on arXiv 24 Jan 2026.

Builds on

22 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Aligning Spoken Dialogue Models from User Interactions26 Jun

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  9. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  10. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  11. 2018Neural Approaches to Conversational AIcited by 164
  12. 2020A Survey on Conversational Recommender Systemscited by 142
9 more
  1. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  2. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  3. 2024sDPO: Don't Use Your Data All at Oncecited by 46
  4. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agentscited by 45
  5. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 32
  6. 2025Single-stream Policy Optimizationcited by 27
  7. 2022Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergencecited by 21
  8. 2025Beyond ReAct: A Planner-Centric Framework for Complex Tool-Augmented LLM Reasoningcited by 16
  9. 2025Reinforcement Learning Constrained Beam Search for Parameter Optimization of Paper Drying Under Flexible Constraintscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems
  2. 2026Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
  3. 2026Enhancing Social Intelligence in LLMs with Hierarchical Reasoning and Utterance-Level Goal Rewarding
  4. 2026ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning
  5. 2025MindFlow+: A Self-Evolving Agent for E-Commerce Customer Service
  6. 2026Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue
  7. 2026Dialogue Model Optimization via Agent Game and Adaptive Tree-based GRPO
  8. 2025Teaching LLM to be Persuasive: Reward-Enhanced Policy Optimization for Alignment from Heterogeneous Rewards