mlsanityData to 14 Oct 2026

Corrector Sampling in Language Models

arXiv 2506.06215 ↗First posted 6 Jun 2025cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Autoregressive language models accumulate errors due to their fixed, irrevocable left-to-right token generation. To address this, we propose a new sampling method called Resample-Previous-Tokens (RPT). RPT mitigates error accumulation by iteratively revisiting and potentially replacing tokens in a window of previously generated text. This method can be integrated into existing autoregressive models, preserving their next-token-prediction quality and speed. Fine-tuning a pretrained 8B parameter model with RPT for only 100B resulted in ~10% relative improvements on reasoning and coding benchmarks compared to the standard sampling.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

22 of 33 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
10 more
  1. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  2. 2024Discrete Flow Matchingcited by 317
  3. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 305
  4. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  5. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 125
  6. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  7. 2024Generator Matching: Generative modeling with arbitrary Markov processescited by 86
  8. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 57
  9. 2024σ-GPTs: A New Approach to Autoregressive Modelscited by 20
  10. 2024Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Forward-Free Diffusion Language Models with BPTT-Free Looped Refinementcited by 2