mlsanityData to 7 Oct 2026

Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback

arXiv 2403.11330 ↗First posted 17 Mar 2024cs.CL cs.AI cs.HCcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

We describe an approach for aligning an LLM-based dialogue agent based on global (i.e., dialogue-level) rewards, while also taking into account naturally-occurring multimodal signals. At a high level, our approach (dubbed GELI) learns a local, turn-level reward model by decomposing the human-provided Global Explicit (GE) session-level reward, using Local Implicit (LI) multimodal reward signals to crossmodally shape the reward decomposition step. This decomposed reward model is then used as part of the standard RHLF pipeline improve an LLM-based dialog agent. We run quantitative and qualitative human studies to evaluate the performance of our GELI approach, and find that it shows consistent improvements across various conversational metrics compared to baseline methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 33 of its 47 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

33 of 47 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  12. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
21 more
  1. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  2. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  3. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  4. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  5. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
  6. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  7. 2022Evaluating Human-Language Model Interactioncited by 90
  8. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  9. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 58
  10. 2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
  11. 2019Cross-modal knowledge distillation for action recognitioncited by 44
  12. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  13. 2019What Can Learned Intrinsic Rewards Capture?cited by 36
  14. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  15. 2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
  16. 2018Learning Self-Imitating Diverse Policiescited by 29
  17. 2018Generative Adversarial Self-Imitation Learningcited by 25
  18. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  19. 2019MOSS: End-to-End Dialog System Framework with Modular Supervisioncited by 19
  20. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  21. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition
  2. 2026SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness
  3. 2024Sparse Rewards Can Self-Train Dialogue Agents
  4. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
  5. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback
  6. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluators
  7. 2025Training Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression
  8. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward