mlsanityData to 7 Oct 2026

Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement

arXiv 2212.04523 ↗First posted 8 Dec 2022cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

The long-distance agreement, evidence for syntactic structure, is increasingly used to assess the syntactic generalization of Neural Language Models. Much work has shown that transformers are capable of high accuracy in varied agreement tasks, but the mechanisms by which the models accomplish this behavior are still not well understood. To better understand transformers' internal working, this work contrasts how they handle two superficially similar but theoretically distinct agreement phenomena: subject-verb and object-past participle agreement in French. Using probing and counterfactual analysis methods, our experiments show that i) the agreement task suffers from several confounders which partially question the conclusions drawn so far and ii) transformers handle subject-verb and object-past participle agreements in a way that is consistent with their modeling in theoretical linguistics.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

28 of 44 references.

References · 28

  1. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  5. 2019Designing and Interpreting Probes with Control Taskscited by 608
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  7. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  8. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  9. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  10. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  11. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  12. 2018Colorless green recurrent networks dream hierarchicallycited by 210
16 more
  1. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  2. 2019Assessing BERT's Syntactic Abilitiescited by 191
  3. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  4. 2020Syntactic Structure from Deep Learningcited by 99
  5. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  6. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  7. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  8. 2022Probing for the Usage of Grammatical Numbercited by 57
  9. 2020Pareto Probing: Trading Off Accuracy for Complexitycited by 55
  10. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  11. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  12. 2017A Challenge Set Approach to Evaluating Machine Translationcited by 37
  13. 2020Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding wordscited by 28
  14. 2019Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignmentcited by 20
  15. 2022Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Taskcited by 13
  16. 2021Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreementcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5; 2025: 1.

7 papers.

  1. 2023SLOG: A Structural Generalization Benchmark for Semantic Parsingcited by 22
  2. 2024Probing the Category of Verbal Aspect in Transformer Language Modelscited by 5
  3. 2024Investigating grammatical abstraction in language models using few-shot learning of novel noun gendercited by 2
  4. 2024Learning Semantic Structure through First-Order-Logic Translationcited by 2
  5. 2025A Benchmark for End-to-End Zero-Shot Biomedical Relation Extraction with LLMs: Experiments with OpenAI Modelscited by 2
  6. 2024Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Modelscited by 1
  7. 2024Can Language Models Induce Grammatical Knowledge from Indirect Evidence?cited by 1