mlsanityData to 7 Oct 2026

Acceptability Judgements via Examining the Topology of Attention Maps

arXiv 2205.09630 ↗First posted 19 May 2022cs.CL cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acceptability of a sentence has been underexplored. This paper approaches the paradigm of acceptability judgments with topological data analysis (TDA), showing that the geometric properties of the attention graph can be efficiently exploited for two standard practices in linguistics: binary judgments and linguistic minimal pairs. Topological features enhance the BERT-based acceptability classifier scores by 8%-24% on CoLA in three languages (English, Italian, and Swedish). By revealing the topological discrepancy between attention maps of minimal pairs, we achieve the human-level performance on the BLiMP benchmark, outperforming nine statistical and Transformer LM baselines. At the same time, TDA provides the foundation for analyzing the linguistic functions of attention heads and interpreting the correspondence between the graph features and grammatical phenomena.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 43 of its 89 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

43 of 89 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  10. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  12. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
31 more
  1. 2019The many Shapley values for model explanationcited by 341
  2. 2019Revealing the Dark Secrets of BERTcited by 300
  3. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  4. 2017An introduction to Topological Data Analysis: fundamental and practical aspects for data scientistscited by 171
  5. 2020Automatic Detection of Machine Generated Text: A Critical Surveycited by 160
  6. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  7. 2019Ripser: efficient computation of Vietoris-Rips persistence barcodescited by 115
  8. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  9. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  10. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  11. 2020Syntactic Structure from Deep Learningcited by 99
  12. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  13. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  14. 2021Representation Topology Divergence: A Method for Comparing Neural Network Representationscited by 66
  15. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 54
  16. 2021Artificial Text Detection via Examining the Topology of Attention Mapscited by 49
  17. 2019Head-Driven Phrase Structure Grammar Parsing on Penn Treebankcited by 43
  18. 2016How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairscited by 39
  19. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  20. 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
  21. 2020GPU-Accelerated Computation of Vietoris-Rips Persistence Barcodescited by 26
  22. 2020Linguistic Profiling of a Neural Language Modelcited by 23
  23. 2021Attention Can Reflect Syntactic Structure (If You Let It)cited by 22
  24. 2020On the Robustness of Language Encoders against Grammatical Errorscited by 18
  25. 2021Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpuscited by 18
  26. 2021DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharingcited by 16
  27. 2020Topology of Word Embeddings: Singularities Reflect Polysemycited by 15
  28. 2021The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERTcited by 15
  29. 2022Measure Estimation in the Barycentric Coding Modelcited by 9
  30. 2021Have Attention Heads in BERT Learned Constituency Grammar?cited by 5
  31. 2021Persistence Homology of TEDtalk: Do Sentence Embeddings Have a Topological Shape?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 4; 2025: 4; 2026: 2.

19 papers.

Sort:

  1. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
  2. 2023Learning Topology-Preserving Data Representationscited by 30
  3. 2023On the Detectability of ChatGPT Content: Benchmarking, Methodology, and Evaluation through the Lens of Academic Writingcited by 26
  4. 2025Hallucination Detection in LLMs with Topological Divergence on Attention Graphscited by 18
  5. 2022The Topological BERT: Transforming Attention into Topology for Natural Language Processingcited by 10
  6. 2023TOPFORMER: Topology-Aware Authorship Attribution of Deepfake Texts with Diverse Writing Stylescited by 10
  7. 2022Topological Data Analysis for Speech Processingcited by 9
  8. 2023Uncertainty Estimation of Transformers' Predictions via Topological Analysis of the Attention Matricescited by 8
  9. 2023Can BERT eat RuCoLA? Topological Data Analysis to Explaincited by 5
  10. 2024Quantum Transfer Learning for Acceptability Judgementscited by 2
  11. 2024Topological Data Analysis Applications in Natural Language Processing: A Surveycited by 2
  12. 2025QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgmentscited by 2
7 more
  1. 2024Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weightingcited by 1
  2. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  3. 2022Robust representations of oil wells' intervals via sparse attention mechanismcited by 0
  4. 2024MUSE: Integrating Multi-Knowledge for Knowledge Graph Completioncited by 0
  5. 2025Detecting Spelling and Grammatical Anomalies in Russian Poetry Textscited by 0
  6. 2025Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representationscited by 0
  7. 2026CodeTD: Topology of Attention Detects Hallucinations in Code LLMscited by 0