mlsanityData to 7 Oct 2026

Assessing Phrasal Representation and Composition in Transformers

arXiv 2010.03763 ↗First posted 8 Oct 2020cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs, such as phrases. However, we have limited understanding of how these models handle representation of phrases, and whether this reflects sophisticated composition of phrase meaning like that done by humans. In this paper, we present systematic analysis of phrasal representations in state-of-the-art pre-trained transformers. We use tests leveraging human judgments of phrase similarity and meaning shift, and compare results before and after control of word overlap, to tease apart lexical effects versus composition effects. We find that phrase representation in these models relies heavily on word content, with little evidence of nuanced composition. We also identify variations in phrase representation quality across models, layers, and representation types, and make corresponding recommendations for usage of representations from these models.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 15; 2023: 5; 2024: 10; 2025: 4.

41 papers.

Sort:

By area:

  1. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  2. 2021Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Explorationcited by 44
  3. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  4. 2022Characterizing Intrinsic Compositionality in Transformers with Tree Projectionscited by 40
  5. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  6. 2021AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Modelscited by 27
  7. 2021Does Vision-and-Language Pretraining Improve Lexical Grounding?cited by 25
  8. 2022Neural reality of argument structure constructionscited by 25
  9. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 19
  10. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  11. 2023Uncovering Intermediate Variables in Transformers using Circuit Probingcited by 12
  12. 2022Modeling structure-building in the brain with CCG parsing and large language modelscited by 11
29 more
  1. 2024Scope Ambiguities in Large Language Modelscited by 11
  2. 2021Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymycited by 9
  3. 2022Interpretation of Black Box NLP Models: A Surveycited by 9
  4. 2022Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Modelscited by 9
  5. 2024Semantics of Multiword Expressions in Transformer-Based Models: A Surveycited by 9
  6. 2022Automatic Evaluation and Analysis of Idioms in Neural Machine Translationcited by 8
  7. 2022Can Language Models Be Specific? How?cited by 8
  8. 2022Transparency Helps Reveal When Language Models Learn Meaningcited by 8
  9. 2021On the Interplay Between Fine-tuning and Composition in Transformerscited by 6
  10. 2023Unbalanced Optimal Transport for Unbalanced Word Alignmentcited by 6
  11. 2023Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operationscited by 5
  12. 2022Hidden Schema Networkscited by 4
  13. 2024Learning High-Quality and General-Purpose Phrase Representationscited by 4
  14. 2024Interpretability of Language Models via Task Spacescited by 4
  15. 2024Interpreting token compositionality in LLMs: A robustness analysiscited by 4
  16. 2022Sample Efficient Approaches for Idiomaticity Detectioncited by 3
  17. 2022Using Paraphrases to Study Properties of Contextual Embeddingscited by 3
  18. 2022PiC: A Phrase-in-Context Dataset for Phrase Understanding and Semantic Searchcited by 3
  19. 2025Understanding Subword Compositionality of Large Language Modelscited by 3
  20. 2022Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5cited by 2
  21. 2023Towards Equipping Transformer with the Ability of Systematic Compositionalitycited by 2
  22. 2024Should We Fear Large Language Models? A Structural Analysis of the Human Reasoning System for Elucidating LLM Capabilities and Risks Through the Lens of Heidegger's Philosophycited by 2
  23. 2022kpfriends at SemEval-2022 Task 2: NEAMER -- Named Entity Augmented Multi-word Expression Recognizercited by 1
  24. 2024Span-Aggregatable, Contextualized Word Embeddings for Effective Phrase Miningcited by 1
  25. 2024Is Less More? Quality, Quantity and Context in Idiom Processing with Natural Language Modelscited by 1
  26. 2024Investigating Idiomaticity in Word Representationscited by 1
  27. 2022HiJoNLP at SemEval-2022 Task 2: Detecting Idiomaticity of Multiword Expressions using Multilingual Pretrained Language Modelscited by 0
  28. 2025Set-Theoretic Compositionality of Sentence Embeddingscited by 0
  29. 2025Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligencecited by 0