# Acceptability Judgements via Examining the Topology of Attention Maps

arXiv: https://arxiv.org/abs/2205.09630 · HTML: https://mlsanity.com/paper/2205.09630/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 89 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [An introduction to Topological Data Analysis: fundamental and practical aspects for data scientists](https://mlsanity.com/paper/1710.04019/) (2017)
- [Automatic Detection of Machine Generated Text: A Critical Survey](https://mlsanity.com/paper/2011.01314/) (2020)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Ripser: efficient computation of Vietoris-Rips persistence barcodes](https://mlsanity.com/paper/1908.02518/) (2019)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [When Do You Need Billions of Words of Pretraining Data?](https://mlsanity.com/paper/2011.04946/) (2020)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning](https://mlsanity.com/paper/1911.03090/) (2019)
- [Do Attention Heads in BERT Track Syntactic Dependencies?](https://mlsanity.com/paper/1911.12246/) (2019)
- [Representation Topology Divergence: A Method for Comparing Neural Network Representations](https://mlsanity.com/paper/2201.00058/) (2021)
- [Cross-Linguistic Syntactic Evaluation of Word Prediction Models](https://mlsanity.com/paper/2005.00187/) (2020)
- [Artificial Text Detection via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2109.04825/) (2021)
- [Head-Driven Phrase Structure Grammar Parsing on Penn Treebank](https://mlsanity.com/paper/1907.02684/) (2019)
- [How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs](https://mlsanity.com/paper/1612.04629/) (2016)

## Built on by (most cited first, 19 of 19)

- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [Learning Topology-Preserving Data Representations](https://mlsanity.com/paper/2302.00136/) (2023)
- [On the Detectability of ChatGPT Content: Benchmarking, Methodology, and Evaluation through the Lens of Academic Writing](https://mlsanity.com/paper/2306.05524/) (2023)
- [Hallucination Detection in LLMs with Topological Divergence on Attention Graphs](https://mlsanity.com/paper/2504.10063/) (2025)
- [The Topological BERT: Transforming Attention into Topology for Natural Language Processing](https://mlsanity.com/paper/2206.15195/) (2022)
- [TOPFORMER: Topology-Aware Authorship Attribution of Deepfake Texts with Diverse Writing Styles](https://mlsanity.com/paper/2309.12934/) (2023)
- [Topological Data Analysis for Speech Processing](https://mlsanity.com/paper/2211.17223/) (2022)
- [Uncertainty Estimation of Transformers' Predictions via Topological Analysis of the Attention Matrices](https://mlsanity.com/paper/2308.11295/) (2023)
- [Can BERT eat RuCoLA? Topological Data Analysis to Explain](https://mlsanity.com/paper/2304.01680/) (2023)
- [Quantum Transfer Learning for Acceptability Judgements](https://mlsanity.com/paper/2401.07777/) (2024)
- [Topological Data Analysis Applications in Natural Language Processing: A Survey](https://mlsanity.com/paper/2411.10298/) (2024)
- [QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments](https://mlsanity.com/paper/2508.16867/) (2025)
- [Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weighting](https://mlsanity.com/paper/2402.14184/) (2024)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Robust representations of oil wells' intervals via sparse attention mechanism](https://mlsanity.com/paper/2212.14246/) (2022)
- [MUSE: Integrating Multi-Knowledge for Knowledge Graph Completion](https://mlsanity.com/paper/2409.17536/) (2024)
- [Detecting Spelling and Grammatical Anomalies in Russian Poetry Texts](https://mlsanity.com/paper/2505.04507/) (2025)
- [Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations](https://mlsanity.com/paper/2506.21682/) (2025)
- [CodeTD: Topology of Attention Detects Hallucinations in Code LLMs](https://mlsanity.com/paper/2609.07779/) (2026)

## Abstract (arXiv)

The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acceptability of a sentence has been underexplored. This paper approaches the paradigm of acceptability judgments with topological data analysis (TDA), showing that the geometric properties of the attention graph can be efficiently exploited for two standard practices in linguistics: binary judgments and linguistic minimal pairs. Topological features enhance the BERT-based acceptability classifier scores by 8%-24% on CoLA in three languages (English, Italian, and Swedish). By revealing the topological discrepancy between attention maps of minimal pairs, we achieve the human-level performance on the BLiMP benchmark, outperforming nine statistical and Transformer LM baselines. At the same time, TDA provides the foundation for analyzing the linguistic functions of attention heads and interpreting the correspondence between the graph features and grammatical phenomena.