Multiple Choice Learning of Low-Rank Adapters for Language Modeling
Not on the map. Why?
Abstract (arXiv)
We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference time. Traditional language modeling is an intrinsically ill-posed problem: given a context, multiple futures may be equally plausible. Our approach leverages Multiple Choice Learning (MCL) and the winner-takes-all loss to efficiently handle ambiguity through Low-Rank Adaptation. We provide a theoretical interpretation of applying MCL to language modeling, assuming the data is generated from a mixture of distributions. We illustrate the proposed approach using mixtures of Markov chains. We then demonstrate with experiments on audio and visual captioning, as well as machine translation, that our method achieves high diversity and relevance in generated outputs. We release the code for applying LoRA-MCL to a wide range of language models.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 83 of its 124 references that are in the map. First posted on arXiv 14 Jul 2025.
Builds on
83 of 124 references.
References · 83
- 2017Attention Is All You Needcited by 60,361
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Visual Instruction Tuningcited by 7,869
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
38 more
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
- 2024Mixtral of Expertscited by 1,823
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2018Hierarchical Neural Story Generationcited by 1,193
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
- 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
- 2024PaliGemma: A versatile 3B VLM for transfercited by 696
- 2024Qwen2-Audio Technical Reportcited by 560
- 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
- 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2019Clotho: An Audio Captioning Datasetcited by 358
- 2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 280
- 2019Topic Modeling in Embedding Spacescited by 224
- 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
- 2024Mixture of LoRA Expertscited by 181
- 2019Image Captioning: Transforming Objects into Wordscited by 171
- 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
- 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
- 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 138
- 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
- 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
- 2022Truncation Sampling as Language Model Desmoothingcited by 123
- 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 118
- 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 113
- 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
- 2022Locally Typical Samplingcited by 111
- 2016Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypothesescited by 97
- 2019Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Predictioncited by 96
- 2020Trading Off Diversity and Quality in Natural Language Generationcited by 96
- 2023Soft Merging of Experts with Adaptive Routingcited by 90
- 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
- 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
- 2019A Discrete Hard EM Approach for Weakly Supervised Question Answeringcited by 78
- 2016Stochastic Multiple Choice Learning for Training Diverse Deep Ensemblescited by 77
- 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
50 most cited of 83 All 83 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.