mlsanityData to 7 Oct 2026

XferBench: a Data-Driven Benchmark for Emergent Language

arXiv 2407.03456 ↗First posted 3 Jul 2024cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language -- the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergent language baselines.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

23 of 42 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  8. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  9. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 168
  10. 2018Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Inputcited by 102
  11. 2019Ease-of-Teaching and Language Structure from Emergent Communicationcited by 65
  12. 2019EGG: a toolkit for research on Emergence of lanGuage in Gamescited by 42
11 more
  1. 2021Emergent Communication of Generalizationscited by 41
  2. 2017ShapeWorld - A new test methodology for multimodal language understandingcited by 37
  3. 2021Learning to Draw: Emergent Communication through Sketchingcited by 23
  4. 2022Emergent Communication for Understanding Human Language Evolution: What's Missing?cited by 23
  5. 2022Linking Emergent and Natural Languages via Corpus Transfercited by 19
  6. 2020Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populationscited by 18
  7. 2020Emergent Communication Pretraining for Few-Shot Machine Translationcited by 13
  8. 2023EC^2: Emergent Communication for Embodied Controlcited by 10
  9. 2022Learning to translate by learning to communicatecited by 4
  10. 2019Biology and Compositionality: Empirical Considerations for Emergent-Communication Protocolscited by 3
  11. 2021Towards Learning to Speak and Hear Through Multi-Agent Communication over a Continuous Acoustic Channelcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024ELCC: the Emergent Language Corpus Collectioncited by 2