mlsanityData to 7 Oct 2026

Linguistic Knowledge and Transferability of Contextual Representations

arXiv 1903.08855 ↗First posted 21 Mar 2019cs.CLcited by 382

Not on the map. Why?

Abstract (arXiv)

Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language. To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and BERT) with a suite of seventeen diverse probing tasks. We find that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge (e.g., conjunct identification). To investigate the transferability of contextual word representations, we quantify differences in the transferability of individual layers within contextualizers, especially between recurrent neural networks (RNNs) and transformers. For instance, higher layers of RNNs are more task-specific, while transformer layers do not exhibit the same monotonic trend. In addition, to better understand what makes contextual word representations transferable, we compare language model pretraining with eleven supervised pretraining tasks. For any given task, pretraining on a closely related task yields better performance than language model pretraining (which is better on average) when the pretraining dataset is fixed. However, language model pretraining on more data gives the best results.

Key facts. Cited by 382 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 99; 2022: 81; 2023: 67; 2024: 52; 2025: 50; 2026: 33.

382 papers.

Sort:

By area:

  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  4. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  5. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  6. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  8. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  9. 2024Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editingcited by 134
  10. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  11. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  12. 2023The geometry of hidden representations of large transformer modelscited by 108
38 more
  1. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  2. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  3. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  4. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  5. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  6. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  7. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  8. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  9. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  10. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  11. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  12. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  13. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
  14. 2022Discovering Latent Concepts Learned in BERTcited by 56
  15. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  16. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  17. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  18. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  19. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  20. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  21. 2022Memory Efficient Continual Learning with Transformerscited by 43
  22. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  23. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  24. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  25. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  26. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  27. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  28. 2021What if This Modified That? Syntactic Interventions via Counterfactual Embeddingscited by 39
  29. 2022Prompting Language Models for Linguistic Structurecited by 39
  30. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  31. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  32. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  33. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformerscited by 36
  34. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  35. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  36. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  37. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  38. 2021The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoderscited by 31

50 most cited of 382 All 382 (JSON)