mlsanityData to 14 Oct 2026

Revisiting Acceptability Judgements

arXiv 2305.14091 ↗First posted 23 May 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work, we revisit linguistic acceptability in the context of large language models. We introduce CoLAC - Corpus of Linguistic Acceptability in Chinese, the first large-scale acceptability dataset for a non-Indo-European language. It is verified by native speakers and is the first acceptability dataset that comes with two sets of labels: a linguist label and a crowd label. Our experiments show that even the largest InstructGPT model performs only at chance level on CoLAC, while ChatGPT's performance (48.30 MCC) is also much below supervised models (59.03 MCC) and human (65.11 MCC). Through cross-lingual transfer experiments and fine-grained linguistic analysis, we provide detailed analysis of the model predictions and demonstrate for the first time that knowledge of linguistic acceptability can be transferred across typologically distinct languages, as well as be traced back to pre-training. Our dataset is publicly available at urlhttps://github.com/huhailinguist/CoLAC.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

35 of 50 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
23 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  12. 2018Neural Network Acceptability Judgmentscited by 958
  13. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  14. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  15. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  16. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  17. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 266
  18. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  19. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  20. 2019Uncertain Natural Language Inferencecited by 35
  21. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  22. 2022Language model acceptability judgements are not always robust to contextcited by 22
  23. 2021Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpuscited by 18

Built on by

Not cited yet.