mlsanityData to 7 Oct 2026

PatchGame: Learning to Signal Mid-level Patches in Referential Games

arXiv 2111.01785 ↗First posted 2 Nov 2021cs.CV cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We study a referential game (a type of signaling game) where two agents communicate with each other via a discrete bottleneck to achieve a common goal. In our referential game, the goal of the speaker is to compose a message or a symbolic representation of "important" image patches, while the task for the listener is to match the speaker's message to a different view of the same image. We show that it is indeed possible for the two agents to develop a communication protocol without explicit or implicit supervision. We further investigate the developed protocol and show the applications in speeding up recent Vision Transformers by using only important patches, and as pre-training for downstream recognition tasks (e.g., classification). Code available at https://github.com/kampta/PatchGame.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 51 of its 84 references that are in the map. First posted on arXiv 2 Nov 2021.

Builds on

51 of 84 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2017Neural Discrete Representation Learningcited by 4,650
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  12. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  13. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  14. 2016Learning Multiagent Communication with Backpropagationcited by 355
  15. 2021Generating Images with Sparse Representationscited by 262
  16. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 257
  17. 2020Fast Differentiable Sorting and Rankingcited by 196
  18. 2016Multi-Agent Cooperation and the Emergence of (Natural) Languagecited by 181
  19. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  20. 2020Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNscited by 151
  21. 2016Discrete Variational Autoencoderscited by 124
  22. 2017Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbolscited by 123
  23. 2020LayoutTransformer: Layout Generation and Completion with Self-attentioncited by 122
  24. 2018Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Inputcited by 102
  25. 2019Measuring Compositionality in Representation Learningcited by 84
  26. 2021A Study of Face Obfuscation in ImageNetcited by 83
  27. 2017Learning with Latent Languagecited by 75
  28. 2017Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learningcited by 70
  29. 2019On the Pitfalls of Measuring Emergent Communicationcited by 66
  30. 2018Referring Relationshipscited by 29
  31. 2017Emergent Translation in Multi-Agent Communicationcited by 22
  32. 2015A Roadmap towards Machine Intelligencecited by 15
  33. 2016Learning to Play Guess Who? and Inventing a Grounded Language as a Consequencecited by 15
  34. 2019Miss Tools and Mr Fruit: Emergent communication in agents learning about object affordancescited by 11
  35. 2020PatchVAE: Learning Local Latent Codes for Recognitioncited by 10
  36. 2017Analogs of Linguistic Structure in Deep Representationscited by 7
  37. 2021The emergence of visual semantics through communication gamescited by 5
  38. 2019Avoiding hashing and encouraging visual semantics in referential emergent language gamescited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2023Diffusion Models Beat GANs on Image Classificationcited by 57
  2. 2023Brain Decodes Deep Netscited by 14
  3. 2024Semantics and Spatiality of Emergent Communicationcited by 2