mlsanityData to 7 Oct 2026

Spontaneous Emerging Preference in Two-tower Language Model

arXiv 2210.07041 ↗First posted 13 Oct 2022cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The ever-growing size of the foundation language model has brought significant performance gains in various types of downstream tasks. With the existence of side-effects brought about by the large size of the foundation language model such as deployment cost, availability issues, and environmental cost, there is some interest in exploring other possible directions, such as a divide-and-conquer scheme. In this paper, we are asking a basic question: are language processes naturally dividable? We study this problem with a simple two-tower language model setting, where two language models with identical configurations are trained side-by-side cooperatively. With this setting, we discover the spontaneous emerging preference phenomenon, where some of the tokens are consistently better predicted by one tower while others by another tower. This phenomenon is qualitatively stable, regardless of model configuration and type, suggesting this as an intrinsic property of natural language. This study suggests that interesting properties of natural language are still waiting to be discovered, which may aid the future development of natural language processing techniques.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

30 of 40 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
18 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2017Mixed Precision Trainingcited by 1,135
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2019Are Sixteen Heads Really Better than One?cited by 803
  6. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  7. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  8. 2019Green AIcited by 620
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  11. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  12. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  13. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  14. 2021How to Train BERT with an Academic Budgetcited by 80
  15. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  16. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
  17. 2019Cooperative Learning of Disjoint Syntax and Semanticscited by 27
  18. 2018On the achievability of blind source separation for high-dimensional nonlinear source mixturescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions
  2. 2025Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
  3. 2026Lasting Effects of Abstract Pretraining Beyond Perplexity
  4. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
  5. 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debates
  6. 2024Improving Context-Aware Preference Modeling for Language Models
  7. 2024Searching for Structure: Investigating Emergent Communication with Large Language Models
  8. 2026Natural Language Processing: A Comprehensive Practical Guide from Tokenisation to RLHF