mlsanityData to 14 Oct 2026

Focused Contrastive Training for Test-based Constituency Analysis

arXiv 2109.15159 ↗First posted 30 Sep 2021cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose a scheme for self-training of grammaticality models for constituency analysis based on linguistic tests. A pre-trained language model is fine-tuned by contrastive estimation of grammatical sentences from a corpus, and ungrammatical sentences that were perturbed by a syntactic test, a transformation that is motivated by constituency theory. We show that consistent gains can be achieved if only certain positive instances are chosen for training, depending on whether they could be the result of a test transformation. This way, the positives, and negatives exhibit similar characteristics, which makes the objective more challenging for the language model, and also allows for additional markup that indicates the position of the test application within the sentence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 18 references that are in the map. First posted on arXiv 30 Sep 2021.

Builds on

12 of 18 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2018Neural Network Acceptability Judgmentscited by 958
  7. 2018Constituency Parsing with a Self-Attentive Encodercited by 199
  8. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105
  9. 2015Transition-Based Dependency Parsing with Stack Long Short-Term Memorycited by 45
  10. 2020Unsupervised Parsing via Constituency Testscited by 23
  11. 2019From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentionscited by 22
  12. 2020Heads-up! Unsupervised Constituency Parsing via Self-Attention Headscited by 12

Built on by

Not cited yet.