Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models
Not on the map. Why?
Abstract (arXiv)
In this work, we propose a method that combines two popular research areas by injecting linguistic structures into pre-trained language models in the parameter-efficient fine-tuning (PEFT) setting. In our approach, parallel adapter modules encoding different linguistic structures are combined using a novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates are used to determine the importance of these modules at each layer of the model. To reduce the number of parameters, we first train the model for a fixed small number of steps before pruning the experts based on their importance scores. Our experiment results with three different pre-trained models show that our approach can outperform state-of-the-art PEFT methods with a comparable number of parameters. In addition, we provide additional analysis to examine the experts selected by each model at each layer to provide insights for future studies.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 60 of its 82 references that are in the map. First posted on arXiv 24 Oct 2023.
Builds on
60 of 82 references.
References · 60
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
38 more
- 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
- 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
- 2020AdapterHub: A Framework for Adapting Transformerscited by 441
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
- 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
- 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
- 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
- 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
- 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
- 2016End-to-End Relation Extraction using LSTMs on Sequences and Tree Structurescited by 161
- 2018Graph Convolution over Pruned Dependency Trees Improves Relation Extractioncited by 146
- 2017Encoding Sentences with Graph Convolutional Networks for Semantic Role Labelingcited by 137
- 2019Semantics-aware BERT for Language Understandingcited by 111
- 2020Relational Graph Attention Network for Aspect-based Sentiment Analysiscited by 104
- 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
- 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
- 2021A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learningcited by 82
- 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
- 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
- 2017Graph Convolutional Encoders for Syntax-aware Neural Machine Translationcited by 70
- 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
- 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
- 2021Syntax-BERT: Improving Pre-trained Transformers with Syntax Treescited by 51
- 2020Do Syntax Trees Help Pre-trained Transformers Extract Information?cited by 47
- 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
- 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
- 2020A Tale of a Probe and a Parsercited by 40
- 2019Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networkscited by 39
- 2021Robust Transfer Learning with Pretrained Language Models through Adapterscited by 35
- 2022DropNet: Reducing Neural Network Complexity via Iterative Pruningcited by 29
- 2021Structural Guidance for Transformer Language Modelscited by 25
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2024: 2.
2 papers.