Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT
Not on the map. Why?
Abstract (arXiv)
Smaller LLMs still face significant challenges even in medium-resourced languages, particularly when it comes to language-specific knowledge -- a problem not easily resolved with machine-translated data. In this case study on Icelandic, we aim to enhance the generation performance of an LLM by specialising it using unstructured text corpora. A key focus is on preventing interference with the models' capabilities of handling longer context during this adaptation. Through ablation studies using various parameter-efficient fine-tuning (PEFT) methods and setups, we find that increasing the number of trainable parameters leads to better and more robust language adaptation. LoRAs placed in the feed-forward layers and bottleneck adapters show promising results with sufficient parameters, while prefix tuning and (IA)3 are not suitable. Although improvements are consistent in 0-shot summarisation, some adapted models struggle with longer context lengths, an issue that can be mitigated by adapting only the final layers.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 17 Dec 2024.
Builds on
24 of 41 references.
References · 24
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
- 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
- 2023Benchmarking Large Language Models for News Summarizationcited by 445
- 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
- 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
12 more
- 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
- 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
- 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
- 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
- 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
- 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
- 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
- 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
- 2024Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Taskscited by 34
- 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
- 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
- 2024Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languagescited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.