Sort by Structure: Language Model Ranking as Dependency Probing
Not on the map. Why?
Abstract (arXiv)
Making an informed choice of pre-trained language model (LM) is critical for performance, yet environmentally costly, and as such widely underexplored. The field of Computer Vision has begun to tackle encoder ranking, with promising forays into Natural Language Processing, however they lack coverage of linguistic tasks such as structured prediction. We propose probing to rank LMs, specifically for parsing dependencies in a given language, by measuring the degree to which labeled trees are recoverable from an LM's contextualized embeddings. Across 46 typologically and architecturally diverse LM-language pairs, our probing approach predicts the best LM choice 79% of the time using orders of magnitude less compute than training a full parser. Within this study, we identify and analyze one recently proposed decoupled LM - RemBERT - and find it strikingly contains less inherent dependency information, but often yields the best parser after full fine-tuning. Without this outlier our approach identifies the best LM in 89% of cases.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 10 Jun 2022.
Builds on
30 of 53 references.
References · 30
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
- 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
- 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
- 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
- 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 188
18 more
- 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
- 2019Multilingual is not enough: BERT for Finnishcited by 119
- 2021KLUE: Korean Language Understanding Evaluationcited by 114
- 2020Rethinking embedding coupling in pre-trained language modelscited by 103
- 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
- 2020Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLPcited by 50
- 2020Predicting Performance for Natural Language Processing Taskscited by 49
- 2014A Weighted Correlation Index for Rankings with Tiescited by 41
- 2020A Tale of a Probe and a Parsercited by 40
- 2020GREEK-BERT: The Greeks visiting Sesame Streetcited by 38
- 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
- 2020Do Neural Language Models Show Preferences for Syntactic Formalisms?cited by 30
- 2021AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application Withcited by 28
- 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
- 2020Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Mediacited by 16
- 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16
- 2018An Investigation of the Interactions Between Pre-Trained Word Embeddings, Character Models and POS Tags in Dependency Parsingcited by 10
- 2022Probing for Labeled Dependency Treescited by 8
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.