mlsanityData to 14 Oct 2026

Semantics of Multiword Expressions in Transformer-Based Models: A Survey

arXiv 2401.15393 ↗First posted 27 Jan 2024cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Multiword expressions (MWEs) are composed of multiple words and exhibit variable degrees of compositionality. As such, their meanings are notoriously difficult to model, and it is unclear to what extent this issue affects transformer architectures. Addressing this gap, we provide the first in-depth survey of MWE processing with transformer models. We overall find that they capture MWE semantics inconsistently, as shown by reliance on surface patterns and memorized information. MWE meaning is also strongly localized, predominantly in early layers of the architecture. Representations benefit from specific linguistic properties, such as lower semantic idiosyncrasy and ambiguity of target expressions. Our findings overall question the ability of transformer models to robustly capture fine-grained semantics. Furthermore, we highlight the need for more directly comparable evaluation setups.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 47 of its 95 references that are in the map. First posted on arXiv 27 Jan 2024.

Builds on

47 of 95 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
35 more
  1. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  4. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  5. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  6. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  7. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  8. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  9. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  10. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  11. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 56
  12. 2021Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Explorationcited by 44
  13. 2020Assessing Phrasal Representation and Composition in Transformerscited by 41
  14. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  15. 2022Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translationcited by 28
  16. 2019Still a Pain in the Neck: Evaluating Text Representations on Lexical Compositioncited by 27
  17. 2020EPIE Dataset: A Corpus For Possible Idiomatic Expressionscited by 24
  18. 2021Idiomatic Expression Identification using Semantic Compatibilitycited by 19
  19. 2018LIDIOMS: A Multilingual Linked Idioms Data Setcited by 15
  20. 2019What do you mean, BERT? Assessing BERT as a Distributional Semantics Modelcited by 14
  21. 2023From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?cited by 12
  22. 2019SemEval-2013 Task 4: Free Paraphrases of Noun Compoundscited by 11
  23. 2020Will it Unblend?cited by 11
  24. 2020NYTWIT: A Dataset of Novel Words in the New York Timescited by 10
  25. 2022Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Modelscited by 9
  26. 2022Systematicity in GPT-3's Interpretation of Novel English Noun Compoundscited by 9
  27. 2022Automatic Evaluation and Analysis of Idioms in Neural Machine Translationcited by 8
  28. 2020MICE: Mining Idioms with Contextual Embeddingscited by 7
  29. 2023A Psycholinguistic Analysis of BERT's Representations of Compoundscited by 7
  30. 2021On the Interplay Between Fine-tuning and Composition in Transformerscited by 6
  31. 2020A BERT-based Dual Embedding Model for Chinese Idiom Predictioncited by 4
  32. 2022Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressionscited by 4
  33. 2022Using Paraphrases to Study Properties of Contextual Embeddingscited by 3
  34. 2022PiC: A Phrase-in-Context Dataset for Phrase Understanding and Semantic Searchcited by 3
  35. 2023Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Spacecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2025Memorization or Reasoning? Exploring the Idiom Understanding of LLMscited by 7
  2. 2025NLP Datasets for Idiom and Figurative Language Taskscited by 2
  3. 2024Investigating Idiomaticity in Word Representationscited by 1
  4. 2026From Lemmas to Dependencies: What Signals Drive Light Verbs Classification?cited by 1
  5. 2025Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contextscited by 0
  6. 2025Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntaxcited by 0
  7. 2025Probing BERT for German Compound Semanticscited by 0
  8. 2025Anatomy of an Idiom: Tracing Non-Compositionality in Language Modelscited by 0
  9. 2026Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classificationcited by 0