mlsanityData to 7 Oct 2026

BERT Rediscovers the Classical NLP Pipeline

arXiv 1905.05950 ↗First posted 15 May 2019cs.CLcited by 1,046

Not on the map. Why?

Cited by 40 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks. We focus on one such model, BERT, and aim to quantify where linguistic information is captured within the network. We find that the model represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference. Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically, revising lower-level decisions on the basis of disambiguating information from higher-level representations.

Key facts. Cited by 1,046 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 156; 2022: 156; 2023: 173; 2024: 119; 2025: 153; 2026: 289.

1,046 papers.

Sort:

By area:

  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  3. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  7. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  8. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  9. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  10. 2021Causal Abstractions of Neural Networkscited by 401
  11. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  12. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
38 more
  1. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  2. 2023Explainability for Large Language Models: A Surveycited by 331
  3. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  7. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  8. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  9. 2022Transformers Learn Shortcuts to Automatacited by 260
  10. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  11. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  12. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  13. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  14. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  15. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  16. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  17. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  18. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  19. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  20. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  21. 2022A Review on Language Models as Knowledge Basescited by 156
  22. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  23. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  24. 2021Unsupervised Speech Recognitioncited by 147
  25. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  26. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  27. 2022Analyzing Transformers in Embedding Spacecited by 131
  28. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  29. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  30. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  31. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  32. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  33. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  34. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  35. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  36. 2023The geometry of hidden representations of large transformer modelscited by 108
  37. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  38. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105

50 most cited of 1,046 All 1,046 (JSON)