Data to 7 Oct 2026

Dissecting Contextual Word Embeddings: Architecture and Representation

arXiv 1808.08949 ↗First posted 27 Aug 2018cs.CLcited by 146

Not on the map. Why?

Abstract (arXiv)

Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks. However, many questions remain as to how and why these models are so effective. In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g. LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned. We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks. Additionally, all architectures learn representations that vary with network depth, from exclusively morphological based at the word embedding layer through local syntax based in the lower contextual layers to longer range semantics such coreference at the upper layers. Together, these results suggest that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 34; 2022: 23; 2023: 28; 2024: 25; 2025: 22; 2026: 14.

146 papers.

Sort:

By area:

2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
2021Causal Abstractions of Neural Networkscited by 401
2023Explainability for Large Language Models: A Surveycited by 331
2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
2022A Review on Language Models as Knowledge Basescited by 156
2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
2021Conditional probing: measuring usable information beyond a baselinecited by 61
2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56

38 more

2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 56
2021Probing BERT in Hyperbolic Spacescited by 50
2023Resolving Knowledge Conflicts in Large Language Modelscited by 49
2022TransPolymer: a Transformer-based language model for polymer property predictionscited by 43
2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
2024Monitoring Latent World States in Language Models with Propositional Probescited by 39
2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 38
2022On the Explainability of Natural Language Processing Deep Modelscited by 37
2021What do pre-trained code models know about code?cited by 36
2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformerscited by 36
2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
2023Are Large Language Models Geospatially Knowledgeable?cited by 30
2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29
2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 29
2021Grounding Representation Similarity with Statistical Testingcited by 28
2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoningcited by 28
2022Structural generalization is hard for sequence-to-sequence modelscited by 24
2024Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformerscited by 24
2025Mechanistic Interpretability of Emotion Inference in Large Language Modelscited by 24
2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
2024Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentationcited by 21
2022Contextualized language models for semantic change detection: lessons learnedcited by 20
2023Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)cited by 20
2023Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layerscited by 20
2022Empowering the Fact-checkers! Automatic Identification of Claim Spans on Twittercited by 19
2021On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependenciescited by 18
2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 18
2021LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyondcited by 17
2024One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Promptscited by 16
2021Inspecting the concept knowledge graph encoded by modern language modelscited by 15
2021A Relation-Oriented Clustering Method for Open Relation Extractioncited by 15
2022Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Contextcited by 15
2023Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothnesscited by 15
2021Mediators in Determining what Processing BERT Performs Firstcited by 14
2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14

50 most cited of 146 All 146 (JSON)