mlsanityData to 7 Oct 2026

Pre-trained Language Model Representations for Language Generation

arXiv 1903.09722 ↗First posted 22 Mar 2019cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Pre-trained language model representations have been successful in a wide range of language understanding tasks. In this paper, we examine different strategies to integrate pre-trained representations into sequence to sequence models and apply it to neural machine translation and abstractive summarization. We find that pre-trained representations are most effective when added to the encoder network which slows inference by only 14%. Our experiments in machine translation show gains of up to 5.3 BLEU in a simulated resource-poor setup. While returns diminish with more labeled data, we still observe improvements when millions of sentence-pairs are available. Finally, on abstractive summarization we achieve a new state of the art on the full text version of CNN/DailyMail.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 7; 2023: 10; 2024: 5.

34 papers.

Sort:

By area:

  1. 2023Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutionscited by 59
  2. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  3. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  4. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  5. 2021The Summary Loop: Learning to Write Abstractive Summaries Without Examplescited by 27
  6. 2021BERTTune: Fine-Tuning Neural Machine Translation with BERTScorecited by 18
  7. 2021AUGNLG: Few-shot Natural Language Generation using Self-trained Data Augmentationcited by 17
  8. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  9. 2021Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summariescited by 15
  10. 2021Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generationcited by 14
  11. 2021Diverse Pretrained Context Encodings Improve Document Translationcited by 13
  12. 2021Self-supervised and Supervised Joint Training for Resource-rich Machine Translationcited by 12
22 more
  1. 2023Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundariescited by 10
  2. 2022On the Effectiveness of Pretrained Models for API Learningcited by 9
  3. 2022Commonsense and Named Entity Aware Knowledge Grounded Dialogue Generationcited by 8
  4. 2021Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviewscited by 7
  5. 2022Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translationcited by 7
  6. 2023Neural Machine Translation for the Indigenous Languages of the Americas: An Introductioncited by 7
  7. 2021Improving speech recognition models with small samples for air traffic control systemscited by 6
  8. 2021Multimodal Conditionality for Natural Language Generationcited by 6
  9. 2023Learning To Rank Resources with GNNcited by 6
  10. 2024From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Modelscited by 6
  11. 2021ViTA: Visual-Linguistic Translation by Aligning Object Tagscited by 5
  12. 2022vec2text with Round-Trip Translationscited by 3
  13. 2023Analysis of Climate Campaigns on Social Media using Bayesian Model Averagingcited by 3
  14. 2023Absformer: Transformer-based Model for Unsupervised Multi-Document Abstractive Summarizationcited by 3
  15. 2022G3SR: Global Graph Guided Session-based Recommendationcited by 2
  16. 2023PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarizationcited by 2
  17. 2024Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimizationcited by 2
  18. 2024The Impact of Syntactic and Semantic Proximity on Machine Translation with Back-Translationcited by 2
  19. 2023Topics in Contextualised Attention Embeddingscited by 1
  20. 2022P3LM: Probabilistically Permuted Prophet Language Modeling for Generative Pre-Trainingcited by 0
  21. 2022Optimizing Deep Transformers for Chinese-Thai Low-Resource Translationcited by 0
  22. 2023Integrating Pre-trained Language Model into Neural Machine Translationcited by 0