mlsanityData to 14 Oct 2026

Copy-Paste Language Models

arXiv 2610.19247 ↗First posted 13 Oct 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 55 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

49 of 55 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
37 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2022In-context Learning and Induction Headscited by 964
  9. 2015Pointer Networkscited by 895
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  13. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  14. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  15. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  16. 2016Incorporating Copying Mechanism in Sequence-to-Sequence Learningcited by 218
  17. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  18. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  19. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  20. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  21. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  22. 2022Training Language Models with Memory Augmentationcited by 113
  23. 2016Improving Neural Language Models with a Continuous Cachecited by 82
  24. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 63
  25. 2016Pointing the Unknown Wordscited by 43
  26. 2024Block-Attention for Efficient Prefillingcited by 37
  27. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 35
  28. 2023Closing the Curious Case of Neural Text Degenerationcited by 32
  29. 2023Copy Is All You Needcited by 27
  30. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 21
  31. 2018Sigsoftmax: Reanalysis of the Softmax Bottleneckcited by 20
  32. 2022Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practicecited by 15
  33. 2019Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearitiescited by 14
  34. 2018Direct Output Connection for a High-Rank Language Modelcited by 11
  35. 2026Lost in Backpropagation: The LM Head is a Gradient Bottleneckcited by 9
  36. 2023Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyondcited by 8
  37. 2026Retrieval Heads are Dynamiccited by 2

Built on by

Not cited yet.