mlsanityData to 7 Oct 2026

DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization

arXiv 2203.11239 ↗First posted 21 Mar 2022cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Large-scale pre-trained sequence-to-sequence models like BART and T5 achieve state-of-the-art performance on many generative NLP tasks. However, such models pose a great challenge in resource-constrained scenarios owing to their large memory requirements and high latency. To alleviate this issue, we propose to jointly distill and quantize the model, where knowledge is transferred from the full-precision teacher model to the quantized and distilled low-precision student model. Empirical analyses show that, despite the challenging nature of generative tasks, we were able to achieve a 16.5x model footprint compression ratio with little performance drop relative to the full-precision counterparts on multiple summarization and QA datasets. We further pushed the limit of compression ratio to 27.7x and presented the performance-efficiency trade-off for generative tasks using pre-trained models. To the best of our knowledge, this is the first work aiming to effectively distill and quantize sequence-to-sequence pre-trained models for language generation tasks.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

26 of 33 references.

References · 26

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
14 more
  1. 2016Sequence-Level Knowledge Distillationcited by 761
  2. 2018Universal Transformerscited by 613
  3. 2019ELI5: Long Form Question Answeringcited by 567
  4. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  5. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  6. 2016Ternary Weight Networkscited by 195
  7. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  8. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  9. 2019A Tensorized Transformer for Language Modelingcited by 85
  10. 2020Pre-trained Summarization Distillationcited by 70
  11. 2020Temporal Reasoning on Implicit Events from Distant Supervisioncited by 58
  12. 2015Neural Networks with Few Multiplicationscited by 46
  13. 2019Fully Quantized Transformer for Machine Translationcited by 36
  14. 2018Loss-aware Weight Quantization of Deep Networkscited by 31

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 15; 2024: 3; 2025: 2.

23 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  2. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  5. 2024SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanismscited by 31
  6. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  7. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  8. 2023Transformer models: an introduction and catalogcited by 21
  9. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  10. 2023Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answeringcited by 15
  11. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  12. 2022Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoderscited by 11
11 more
  1. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
  2. 2023xPQA: Cross-Lingual Product Question Answering across 12 Languagescited by 10
  3. 2023Binary and Ternary Natural Language Generationcited by 7
  4. 2023Greener yet Powerful: Taming Large Code Generation Models with Quantizationcited by 4
  5. 2023Task-agnostic Distillation of Encoder-Decoder Language Modelscited by 4
  6. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  7. 2023BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomialscited by 3
  8. 2025QUADS: QUAntized Distillation Framework for Efficient Speech Language Understandingcited by 2
  9. 2023To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiencycited by 1
  10. 2023Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Methodcited by 0
  11. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0