mlsanityData to 7 Oct 2026

Language Modelling as a Multi-Task Problem

arXiv 2101.11287 ↗First posted 27 Jan 2021cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose to study language modelling as a multi-task problem, bringing together three strands of research: multi-task learning, linguistics, and interpretability. Based on hypotheses derived from linguistic theory, we investigate whether language models adhere to learning principles of multi-task learning during training. To showcase the idea, we analyse the generalisation behaviour of language models as they learn the linguistic concept of Negative Polarity Items (NPIs). Our experiments demonstrate that a multi-task setting naturally emerges within the objective of the more general task of language modelling.We argue that this insight is valuable for multi-task learning, linguistics and interpretability research and can lead to exciting new findings in all three domains.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 20 of its 51 references that are in the map. First posted on arXiv 27 Jan 2021.

Builds on

20 of 51 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  3. 2017A Survey on Multi-Task Learningcited by 838
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  5. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  6. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  7. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  8. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  9. 2018Characterizing and Avoiding Negative Transfercited by 226
  10. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  11. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  12. 2015Multi-task Sequence to Sequence Learningcited by 94
8 more
  1. 2017One Model To Learn Them Allcited by 86
  2. 2017Identifying beneficial task relations for multi-task learning in deep neural networkscited by 59
  3. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  4. 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
  5. 2020Can neural networks acquire a structural bias from raw linguistic data?cited by 35
  6. 2017Multi-Task Learning for Mental Health using Social Media Textcited by 25
  7. 2019Structural Supervision Improves Learning of Non-Local Grammatical Dependenciescited by 24
  8. 2020diagNNose: A Library for Neural Activation Analysiscited by 7

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 3; 2024: 4.

10 papers.

  1. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  2. 2023The Learnability of In-Context Learningcited by 126
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  4. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  5. 2023Language acquisition: do children and language models follow similar learning stages?cited by 40
  6. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  7. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  8. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  9. 2024Interpretability of Language Models via Task Spacescited by 4
  10. 2021Transformers in the loop: Polarity in neural models of languagecited by 0