mlsanityData to 7 Oct 2026

Sentence Ambiguity, Grammaticality and Complexity Probes

arXiv 2210.06928 ↗First posted 13 Oct 2022cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

It is unclear whether, how and where large pre-trained language models capture subtle linguistic traits like ambiguity, grammaticality and sentence complexity. We present results of automatic classification of these traits and compare their viability and patterns across representation types. We demonstrate that template-based datasets with surface-level artifacts should not be used for probing, careful comparisons with baselines should be done and that t-SNE plots should not be used to determine the presence of a feature among dense vectors representations. We also show how features might be highly localized in the layers for these models and get lost in the upper layers.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 40 of its 91 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

40 of 91 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  9. 2021GPT Understands, Toocited by 715
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  11. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  12. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
28 more
  1. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  2. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  3. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  4. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  5. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  6. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  7. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  8. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  9. 2021Revisiting the Uniform Information Density Hypothesiscited by 63
  10. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  11. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  12. 2021Probing BERT in Hyperbolic Spacescited by 50
  13. 2019Supervised and Unsupervised Neural Approaches to Text Readabilitycited by 45
  14. 2019Probing Biomedical Embeddings from Language Modelscited by 39
  15. 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
  16. 2020VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classificationcited by 35
  17. 2019Universal Text Representation from BERT: An Empirical Studycited by 28
  18. 2021How is BERT surprised? Layerwise detection of linguistic anomaliescited by 24
  19. 2019LINSPECTOR: Multilingual Probing Tasks for Word Representationscited by 22
  20. 2019Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddingscited by 15
  21. 2020Incorporating Visual Semantics into Sentence Representations within a Grounded Spacecited by 15
  22. 2020Probing Task-Oriented Dialogue Representation from Language Modelscited by 15
  23. 2020A Matter of Framing: The Impact of Linguistic Formalism on Probing Resultscited by 12
  24. 2020Modelling Lexical Ambiguity with Density Matricescited by 11
  25. 2022Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONEScited by 10
  26. 2013Recurrent Convolutional Neural Networks for Discourse Compositionalitycited by 7
  27. 2021Probing neural networks with t-SNE, class-specific projections and a guided tourcited by 6
  28. 2022EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarioscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and Englishcited by 0