mlsanityData to 14 Oct 2026

On Elastic Language Models

arXiv 2311.07204 ↗First posted 13 Nov 2023cs.IR cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large-scale pretrained language models have achieved compelling performance in a wide range of language understanding and information retrieval tasks. Knowledge distillation offers an opportunity to compress a large language model to a small one, in order to reach a reasonable latency-performance tradeoff. However, for scenarios where the number of requests (e.g., queries submitted to a search engine) is highly variant, the static tradeoff attained by the compressed language model might not always fit. Once a model is assigned with a static tradeoff, it could be inadequate in that the latency is too high when the number of requests is large or the performance is too low when the number of requests is small. To this end, we propose an elastic language model (ElasticLM) that elastically adjusts the tradeoff according to the request stream. The basic idea is to introduce a compute elasticity to the compressed language model, so that the tradeoff could vary on-the-fly along scalable and controllable compute. Specifically, we impose an elastic structure to enable ElasticLM with compute elasticity and design an elastic optimization to learn ElasticLM under compute elasticity. To serve ElasticLM, we apply an elastic schedule. Considering the specificity of information retrieval, we adapt ElasticLM to dense retrieval and reranking and present ElasticDenser and ElasticRanker respectively. Offline evaluation is conducted on a language understanding benchmark GLUE; and several information retrieval tasks including Natural Question, Trivia QA, and MS MARCO. The results show that ElasticLM along with ElasticDenser and ElasticRanker can perform correctly and competitively compared with an array of static baselines. Furthermore, online simulation with concurrency is also carried out. The results demonstrate that ElasticLM can provide elastic tradeoffs with respect to varying request stream.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 68 of its 92 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

68 of 92 references.

References · 68

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
38 more
  1. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2018Neural Network Acceptability Judgmentscited by 958
  6. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  10. 2019Passage Re-ranking with BERTcited by 645
  11. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  13. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  14. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  15. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  16. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  17. 2018Slimmable Neural Networkscited by 305
  18. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 286
  19. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  20. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
  21. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  22. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  23. 2019Document Expansion by Query Predictioncited by 231
  24. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  25. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
  26. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  27. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  28. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 165
  29. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  30. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 155
  31. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  32. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  33. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  34. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  35. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  36. 2016Learning to Match Using Local and Distributed Representations of Text for Web Searchcited by 68
  37. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  38. 2020RepBERT: Contextualized Text Embeddings for First-Stage Retrievalcited by 46

50 most cited of 68 All 68 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Elastic Architecture Search for Efficient Language Models
  2. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
  3. 2022QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation
  4. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
  5. 2024Compact Language Models via Pruning and Knowledge Distillation
  6. 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Survey
  8. 2024Enhancing Cloud-Based Large Language Model Processing with Elasticsearch and Transformer Models