KS-Lottery: Finding Certified Lottery Tickets for Multilingual Language Models
Not on the map. Why?
Abstract (arXiv)
The lottery ticket hypothesis posits the existence of ``winning tickets'' within a randomly initialized neural network. Do winning tickets exist for LLMs in fine-tuning scenarios? How can we find such winning tickets? In this paper, we propose KS-Lottery, a method to identify a small subset of LLM parameters highly effective in multilingual fine-tuning. Our key idea is to use Kolmogorov-Smirnov Test to analyze the distribution shift of parameters before and after fine-tuning. We further theoretically prove that KS-Lottery can find the certified winning tickets in the embedding layer, fine-tuning on the found parameters is guaranteed to perform as well as full fine-tuning. Comparing KS-Lottery with other parameter-efficient tuning algorithms on translation tasks, the experimental results show that KS-Lottery finds a much smaller set of parameters for fine-tuning while achieving the comparable performance as full fine-tuning LLM. Surprisingly, we find that fine-tuning 18 tokens' embedding of LLaMA suffices to reach the fine-tuning translation performance~footnotehttps://github.com/CONE-MT/KS-Lottery..
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 5 Feb 2024.
Builds on
36 of 42 references.
References · 36
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2023Mistral 7Bcited by 3,065
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
24 more
- 2023Large Language Modelscited by 774
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
- 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
- 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
- 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
- 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
- 2016Understanding Neural Networks through Representation Erasurecited by 223
- 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
- 2018Semidefinite relaxations for certifying robustness to adversarial examplescited by 178
- 2019Certified Robustness to Adversarial Word Substitutionscited by 164
- 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
- 2020Learning Certified Individually Fair Representationscited by 69
- 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
- 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
- 2023ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedbackcited by 34
- 2022Certified Robustness Against Natural Language Attacks by Causal Interventioncited by 23
- 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
- 2021Latent Space Smoothing for Individually Fair Representationscited by 18
- 2022Robust Lottery Tickets for Pre-trained Language Modelscited by 16
- 2021Domain Adaptation via Maximizing Surrogate Mutual Informationcited by 8
- 2021Certified Robustness via Randomized Smoothing over Multiplicative Parameters of Input Transformationscited by 6
- 2021Where do Models go Wrong? Parameter-Space Saliency Maps for Explainabilitycited by 6
- 2023A Challenging Benchmark for Low-Resource Learningcited by 5
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.