mlsanityData to 7 Oct 2026

Countering Language Drift via Visual Grounding

arXiv 1909.04499 ↗First posted 10 Sep 2019cs.CL cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

Emergent multi-agent communication protocols are very different from natural language and not easily interpretable by humans. We find that agents that were initially pretrained to produce natural language can also experience detrimental language drift: when a non-linguistic reward is used in a goal-based task, e.g. some scalar success metric, the communication protocol may easily and radically diverge from natural language. We recast translation as a multi-agent communication game and examine auxiliary training constraints for their effectiveness in mitigating language drift. We show that a combination of syntactic (language model likelihood) and semantic (visual grounding) constraints gives the best communication performance, allowing pre-trained agents to retain English syntax while learning to accurately convey the intended meaning.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 6; 2023: 15; 2024: 14; 2025: 6; 2026: 5.

55 papers.

Sort:

By area:

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  3. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  4. 2023Cones: Concept Neurons in Diffusion Models for Customized Generationcited by 150
  5. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  6. 2023ITI-GEN: Inclusive Text-to-Image Generationcited by 64
  7. 2024DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalizationcited by 60
  8. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  9. 2023Customizing 360-Degree Panoramas through Text-to-Image Diffusion Modelscited by 47
  10. 2023Language Model Alignment with Elastic Resetcited by 34
  11. 2023All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Modelscited by 33
  12. 2024Attention Calibration for Disentangled Text-to-Image Personalizationcited by 31
38 more
  1. 2023AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatarscited by 29
  2. 2023ViCo: Plug-and-play Visual Condition for Personalized Text-to-image Generationcited by 23
  3. 2021Emergent Discrete Communication in Semantic Spacescited by 20
  4. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  5. 2023NewMove: Customizing text-to-video models with novel motionscited by 18
  6. 2024Face2Diffusion for Fast and Editable Face Personalizationcited by 17
  7. 2024A Review of the Applications of Deep Learning-Based Emergent Communicationcited by 17
  8. 2024Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communicationcited by 17
  9. 2023Generate Anything Anywhere in Any Scenecited by 14
  10. 2025Fine-Tuning Visual Autoregressive Models for Subject-Driven Generationcited by 13
  11. 2023Towards More Human-like AI Communication: A Review of Emergent Communication Researchcited by 12
  12. 2024ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflectionscited by 12
  13. 2023Investigating Personalization Methods in Text to Music Generationcited by 11
  14. 2021Dynamic population-based meta-learning for multi-agent communication with natural languagecited by 10
  15. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  16. 2021Situated Dialogue Learning through Procedural Environment Generationcited by 9
  17. 2021Calibrate your listeners! Robust communication-based training for pragmatic speakerscited by 9
  18. 2024Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Modelscited by 9
  19. 2025Spontaneous Giving and Calculated Greed in Language Modelscited by 9
  20. 2021Multitasking Inhibits Semantic Driftcited by 8
  21. 2023My3DGen: A Scalable Personalized 3D Generative Modelcited by 8
  22. 2023The Big Data Myth: Using Diffusion Models for Dataset Generation to Train Deep Detection Modelscited by 7
  23. 2024U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentationcited by 7
  24. 2025Personalized Text-to-Image Generation with Auto-Regressive Modelscited by 6
  25. 2021The emergence of visual semantics through communication gamescited by 5
  26. 2022Intra-agent speech permits zero-shot task acquisitioncited by 5
  27. 2024MyTimeMachine: Personalized Facial Age Transformationcited by 5
  28. 2022Learning to translate by learning to communicatecited by 4
  29. 2024DreamBlend: Advancing Personalized Fine-tuning of Text-to-Image Diffusion Modelscited by 4
  30. 2022Know your audience: specializing grounded language models with listener subtractioncited by 3
  31. 2025Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interactioncited by 3
  32. 2024ComFusion: Personalized Subject Generation in Multiple Specific Scenes From Single Imagecited by 2
  33. 2024Semantics and Spatiality of Emergent Communicationcited by 2
  34. 2021An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Gamescited by 1
  35. 2021EVOQUER: Enhancing Temporal Grounding with Video-Pivoted BackQuery Generationcited by 1
  36. 2023CB2: Collaborative Natural Language Interaction Research Platformcited by 1
  37. 2024JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuningcited by 1
  38. 2026Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communitiescited by 1

50 most cited of 55 All 55 (JSON)