mlsanityData to 7 Oct 2026

What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models

arXiv 2004.06499 ↗First posted 14 Apr 2020cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Peeking into the inner workings of BERT has shown that its layers resemble the classical NLP pipeline, with progressively more complex tasks being concentrated in later layers. To investigate to what extent these results also hold for a language other than English, we probe a Dutch BERT-based model and the multilingual BERT model for Dutch NLP tasks. In addition, through a deeper analysis of part-of-speech tagging, we show that also within a given task, information is spread over different parts of the network and the pipeline might not be as neat as it seems. Each layer has different specialisations, so that it may be more useful to combine information from different layers, instead of selecting a single one based on the best overall performance.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 9; 2023: 6; 2024: 6; 2025: 3; 2026: 7.

35 papers.

Sort:

By area:

  1. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  2. 2021Discourse Probing of Pretrained Language Modelscited by 38
  3. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  4. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  5. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  6. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
  7. 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16
  8. 2022Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathicited by 16
  9. 2023Unsupervised Layer-wise Score Aggregation for Textual OOD Detectioncited by 15
  10. 2024Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modificationcited by 14
  11. 2022A context-aware knowledge transferring strategy for CTC-based ASRcited by 12
  12. 2023Probing LLMs for Joint Encoding of Linguistic Categoriescited by 9
23 more
  1. 2024Natural Language Processing RELIES on Linguisticscited by 9
  2. 2023Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrievalcited by 8
  3. 2024C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learningcited by 8
  4. 2022Hyperbolic Relevance Matching for Neural Keyphrase Extractioncited by 7
  5. 2024Are there identifiable structural parts in the sentence embedding whole?cited by 6
  6. 2022Multi-Level Contrastive Learning for Cross-Lingual Alignmentcited by 4
  7. 2022Understanding Domain Learning in Language Models Through Subpopulation Analysiscited by 4
  8. 2022Explanation Regeneration via Information Bottleneckcited by 4
  9. 2024Probing Critical Learning Dynamics of PLMs for Hate Speech Detectioncited by 3
  10. 2026Uncovering the Latent Potential of Deep Intermediate Representationscited by 3
  11. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  12. 2025Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layerscited by 2
  13. 2022Feature Aggregation in Zero-Shot Cross-Lingual Transfer Using Multilingual BERTcited by 1
  14. 2023Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Modelscited by 1
  15. 2025QoSBERT: An Uncertainty-Aware Approach based on Pre-trained Language Models for Service Quality Predictioncited by 1
  16. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 1
  17. 2026LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden Statescited by 1
  18. 2021Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?cited by 0
  19. 2023REE-HDSC: Recognizing Extracted Entities for the Historical Database Suriname Curacaocited by 0
  20. 2026A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classificationcited by 0
  21. 2026Measuring Temporal Linguistic Emergence in Diffusion Language Modelscited by 0
  22. 2026Grammatical "grandmother neurons" are rare in LLMscited by 0
  23. 2026Fusing Visual and Textual Representations via Multi-layer Fusing Transformers for Vietnamese Visual Question Answeringcited by 0