mlsanityData to 7 Oct 2026

Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond

arXiv 2305.12289 ↗First posted 20 May 2023cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Is the output softmax layer, which is adopted by most language models (LMs), always the best way to compute the next word probability? Given so many attention layers in a modern transformer-based LM, are the pointer networks redundant nowadays? In this study, we discover that the answers to both questions are no. This is because the softmax bottleneck sometimes prevents the LMs from predicting the desired distribution and the pointer networks can be used to break the bottleneck efficiently. Based on the finding, we propose several softmax alternatives by simplifying the pointer networks and accelerating the word-by-word rerankers. In GPT-2, our proposals are significantly better and more efficient than mixture of softmax, a state-of-the-art softmax alternative. In summarization experiments, without significantly decreasing its training/testing speed, our best method based on T5-Small improves factCC score by 2 points in CNN/DM and XSUM dataset, and improves MAUVE scores by 30% in BookSum paragraph-level dataset.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 48 of its 67 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

48 of 67 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  12. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
36 more
  1. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  2. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  5. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  6. 2022A Contrastive Framework for Neural Text Generationcited by 224
  7. 2016Incorporating Copying Mechanism in Sequence-to-Sequence Learningcited by 217
  8. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
  9. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  10. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  11. 2022Training Language Models with Memory Augmentationcited by 112
  12. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  13. 2020Residual Energy-Based Models for Text Generationcited by 95
  14. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  15. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  16. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  17. 2022Mix and Match: Learning-free Controllable Text Generation using Energy Language Modelscited by 66
  18. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  19. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  20. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 50
  21. 2022FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarizationcited by 49
  22. 2019Plan, Write, and Revise: an Interactive System for Open-Domain Story Generationcited by 43
  23. 2022Controllable Text Generation with Neurally-Decomposed Oraclecited by 41
  24. 2020"You are grounded!": Latent Name Artifacts in Pre-trained Language Modelscited by 39
  25. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  26. 2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
  27. 2022Don't Say What You Don't Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Searchcited by 23
  28. 2021Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identitycited by 22
  29. 2022CoNT: Contrastive Neural Text Generationcited by 21
  30. 2022Improving the Faithfulness of Abstractive Summarization via Entity Coverage Controlcited by 21
  31. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  32. 2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
  33. 2022Towards Coherent and Consistent Use of Entities in Narrative Generationcited by 10
  34. 2022Generating Coherent Narratives by Learning Dynamic and Discrete Entity States with a Contrastive Frameworkcited by 6
  35. 2023HistAlign: Improving Context Dependency in Language Generation by Aligning with Historycited by 6
  36. 2023PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarizationcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 2.

7 papers.

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023PaniniQA: Enhancing Patient Education Through Interactive Question Answeringcited by 21
  3. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  4. 2023UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?cited by 14
  5. 2023Improving Summarization with Human Editscited by 7
  6. 2023To Copy, or not to Copy; That is a Critical Issue of the Output Softmax Layer in Neural Sequential Recommenderscited by 2
  7. 2024LargePiG: Your Large Language Model is Secretly a Pointer Generatorcited by 0