mlsanityData to 14 Oct 2026

Emergent Communication in Interactive Sketch Question Answering

arXiv 2310.15597 ↗First posted 24 Oct 2023cs.AI cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Vision-based emergent communication (EC) aims to learn to communicate through sketches and demystify the evolution of human communication. Ironically, previous works neglect multi-round interaction, which is indispensable in human communication. To fill this gap, we first introduce a novel Interactive Sketch Question Answering (ISQA) task, where two collaborative players are interacting through sketches to answer a question about an image in a multi-round manner. To accomplish this task, we design a new and efficient interactive EC system, which can achieve an effective balance among three evaluation factors, including the question answering accuracy, drawing complexity and human interpretability. Our experimental results including human evaluation demonstrate that multi-round interactive mechanism facilitates targeted and efficient communication between intelligent agents with decent human interpretability.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

39 of 48 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2015Fast R-CNNcited by 2,933
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2016Least Squares Generative Adversarial Networkscited by 1,001
27 more
  1. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  2. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  3. 2017A Neural Representation of Sketch Drawingscited by 313
  4. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  5. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  6. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 263
  7. 2022Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Mapscited by 255
  8. 2016Multi-Agent Cooperation and the Emergence of (Natural) Languagecited by 185
  9. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 170
  10. 2015Exploring Models and Data for Image Question Answeringcited by 139
  11. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 114
  12. 2022Learning to generate line drawings that convey geometry and semanticscited by 83
  13. 2019Learning to Paint With Model-based Deep Reinforcement Learningcited by 71
  14. 2019On the Pitfalls of Measuring Emergent Communicationcited by 71
  15. 2022CLIPascene: Scene Sketching with Different Types and Levels of Abstractioncited by 64
  16. 2018Emergent Communication through Negotiationcited by 62
  17. 2015Ask Your Neurons: A Neural-based Approach to Answering Questions about Imagescited by 51
  18. 2018Learning to Sketch with Shortcut Cycle Consistencycited by 47
  19. 2018DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architecturescited by 40
  20. 2021Learning to Draw: Emergent Communication through Sketchingcited by 23
  21. 2021Emergent Discrete Communication in Semantic Spacescited by 20
  22. 2020Learning to cooperate: Emergent communication in multi-agent navigationcited by 18
  23. 2020SketchEmbedNet: Learning Novel Concepts by Imitating Drawingscited by 18
  24. 2021Emergent Graphical Conventions in a Visual Communication Gamecited by 16
  25. 2019Pragmatic inference and visual abstraction enable contextual flexibility during visual communicationcited by 14
  26. 2021Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communicationcited by 12
  27. 2021Designing Games for Enabling Co-creation with Social Agentscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  2. 2024SimSiam Naming Game: A Unified Approach for Emergent Communication and Representation Learningcited by 1
  3. 2026Drawing with Strangers: Population Scaling Drives Zero-Shot Mutual Intelligibility in Emergent Sketchingcited by 0