mlsanityData to 7 Oct 2026

CoGen: Learning from Feedback with Coupled Comprehension and Generation

arXiv 2408.15992 ↗First posted 28 Aug 2024cs.CL cs.AI cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Systems with both language comprehension and generation capabilities can benefit from the tight connection between the two. This work studies coupling comprehension and generation with focus on continually learning from interaction with users. We propose techniques to tightly integrate the two capabilities for both learning and inference. We situate our studies in two-player reference games, and deploy various models for thousands of interactions with human users, while learning from interaction feedback signals. We show dramatic improvements in performance over time, with comprehension-generation coupling leading to performance improvements up to 26% in absolute terms and up to 17% higher accuracies compared to a non-coupled system. Our analysis also shows coupling has substantial qualitative impact on the system's language, making it significantly more human-like.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 52 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

22 of 52 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  4. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  5. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  6. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  7. 2016Reasoning About Pragmatics with Neural Listeners and Speakerscited by 68
  8. 2017Colors in Context: A Pragmatic Neural Model for Grounded Language Understandingcited by 57
  9. 2017Unified Pragmatic Models for Generating and Following Instructionscited by 55
  10. 2017Context-aware Captions from Context-agnostic Supervisioncited by 43
  11. 2022Abstract Visual Reasoning with Tangram Shapescited by 43
  12. 2018Pragmatically Informative Image Captioning with Character-Level Inferencecited by 42
10 more
  1. 2019Continual adaptation for efficient machine communicationcited by 28
  2. 2021From partners to populations: A hierarchical Bayesian account of coordination and conventioncited by 27
  3. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  4. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  5. 2020Learning to refer informatively by amortizing pragmatic reasoningcited by 17
  6. 2020Empirica: a virtual lab for high-throughput macro-level experimentscited by 16
  7. 2023Computational Language Acquisition with Theory of Mindcited by 15
  8. 2018An Incremental Iterated Response Model of Pragmaticscited by 11
  9. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  10. 2021Analysis of Language Change in Collaborative Instruction Followingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2025Success and Cost Elicit Convention Formation for Efficient Communicationcited by 7
  2. 2024Retrospective Learning from Interactionscited by 6
  3. 2025Playpen: An Environment for Exploring Learning Through Conversational Interactioncited by 2
  4. 2025MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Languagecited by 1
  5. 2025Improving LLMs' Learning for Coreference Resolutioncited by 1
  6. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretationcited by 0
  7. 2026Learning to Refer from Estimated Listener Gazecited by 0