mlsanityData to 7 Oct 2026

What do you learn from context? Probing for sentence structure in contextualized word representations

arXiv 1905.06316 ↗First posted 15 May 2019cs.CLcited by 436

Not on the map. Why?

Abstract (arXiv)

Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

Key facts. Cited by 436 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 113; 2022: 89; 2023: 83; 2024: 71; 2025: 49; 2026: 31.

436 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  3. 2023Explainability for Large Language Models: A Surveycited by 331
  4. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  7. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  8. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  9. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  10. 2022A Review on Language Models as Knowledge Basescited by 156
  11. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  12. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
38 more
  1. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  2. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  3. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  4. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  5. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  6. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  7. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  8. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  9. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  10. 2021An Interpretability Illusion for BERTcited by 94
  11. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  12. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  13. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  14. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  15. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  16. 2021Chess as a Testbed for Language Model State Trackingcited by 69
  17. 2021Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classificationcited by 67
  18. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  19. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  20. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  21. 2022Probing for the Usage of Grammatical Numbercited by 57
  22. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  23. 2022Discovering Latent Concepts Learned in BERTcited by 56
  24. 2021Probing BERT in Hyperbolic Spacescited by 50
  25. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 44
  26. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  27. 2021Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integrationcited by 43
  28. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  29. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  30. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  31. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  32. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  33. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39
  34. 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
  35. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  36. 2021What do pre-trained code models know about code?cited by 36
  37. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  38. 2024A Vision Check-up for Language Modelscited by 36

50 most cited of 436 All 436 (JSON)