mlsanityData to 7 Oct 2026

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

arXiv 1910.13461 ↗First posted 29 Oct 2019cs.CL cs.LG stat.MLcited by 5,874

Not on the map. Why?

Cited by 147 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.

Key facts. Cited by 5,874 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 4; 2021: 850; 2022: 1342; 2023: 1608; 2024: 1163; 2025: 655; 2026: 252.

5,874 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2022Visual Prompt Tuningcited by 1,674
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  13. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  14. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  15. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  16. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  17. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  18. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  19. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  20. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  21. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  22. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  23. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  24. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  25. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  26. 2021GPT Understands, Toocited by 715
  27. 2024Chronos: Learning the Language of Time Seriescited by 712
  28. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  29. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  30. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  31. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  32. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  33. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  34. 2023A Comprehensive Overview of Large Language Modelscited by 639
  35. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  36. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  37. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  38. 2022Towards Reasoning in Large Language Models: A Surveycited by 593

50 most cited of 5,874 All 5,874 (JSON)