mlsanityData to 7 Oct 2026

Scheduled Sampling for Transformers

arXiv 1906.07651 ↗First posted 18 Jun 2019cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Scheduled sampling is a technique for avoiding one of the known problems in sequence-to-sequence generation: exposure bias. It consists of feeding the model a mix of the teacher forced embeddings and the model predictions from the previous step in training time. The technique has been used for improving the model performance with recurrent neural networks (RNN). In the Transformer model, unlike the RNN, the generation of a new word attends to the full sentence generated so far, not only to the last word, and it is not straightforward to apply the scheduled sampling technique. We propose some structural changes to allow scheduled sampling to be applied to Transformer architecture, via a two-pass decoding strategy. Experiments on two language pairs achieve performance close to a teacher-forcing baseline and show that this technique is promising for further exploration.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 10; 2023: 13; 2024: 8; 2025: 3; 2026: 4.

46 papers.

Sort:

By area:

  1. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  2. 2023Learning to Generate Better Than Your LLMcited by 49
  3. 2024LARP: Tokenizing Videos with a Learned Autoregressive Generative Priorcited by 46
  4. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 45
  5. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  6. 2022Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objectscited by 24
  7. 2024Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Modelscited by 20
  8. 2024PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformerscited by 14
  9. 2022Neural Inertial Localizationcited by 13
  10. 2023Semi-Offline Reinforcement Learning for Optimized Text Generationcited by 12
  11. 2021WeChat Neural Machine Translation Systems for WMT21cited by 11
  12. 2021Scheduled Sampling Based on Decoding Steps for Neural Machine Translationcited by 11
34 more
  1. 2022Flipped Classroom: Effective Teaching for Time Series Forecastingcited by 11
  2. 2021Confidence-Aware Scheduled Sampling for Neural Machine Translationcited by 10
  3. 2022Auto-regressive Image Synthesis with Integrated Quantizationcited by 9
  4. 2023ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attentioncited by 9
  5. 2021Mixed Cross Entropy Loss for Neural Machine Translationcited by 8
  6. 2025Generative World Modelling for Humanoids: 1X World Model Challenge Technical Reportcited by 8
  7. 2024Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokenscited by 7
  8. 2023Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoderscited by 5
  9. 2023Translating away Translationese without Parallel Datacited by 5
  10. 2022Your Autoregressive Generative Model Can be Better If You Treat It as an Energy-Based Onecited by 4
  11. 2022Global memory transformer for processing long documentscited by 4
  12. 2023Beyond MLE: Convex Learning for Text Generationcited by 4
  13. 2022NEURAL MARIONETTE: A Transformer-based Multi-action Human Motion Synthesis Systemcited by 3
  14. 2024MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motionscited by 3
  15. 2026Deep Sketch-Based 3D Modeling: A Surveycited by 3
  16. 2021Adversarial Machine Learning in Text Analysis and Generationcited by 2
  17. 2021One Source, Two Targets: Challenges and Rewards of Dual Decodingcited by 2
  18. 2023An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generationcited by 2
  19. 2025SoilNet: A Multimodal Multitask Model for Hierarchical Classification of Soil Horizonscited by 2
  20. 2021Calliope -- A Polyphonic Music Transformercited by 1
  21. 2021Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translationcited by 1
  22. 2023To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiencycited by 1
  23. 2023Mitigating the Exposure Bias in Sentence-Level Grapheme-to-Phoneme (G2P) Transductioncited by 1
  24. 2023Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learningcited by 1
  25. 2024EmoFace: Emotion-Content Disentangled Speech-Driven 3D Talking Face Animationcited by 1
  26. 2022Towards Robust Online Dialogue Response Generationcited by 0
  27. 2022Detecting Network-based Internet Censorship via Latent Feature Representation Learningcited by 0
  28. 2023Dynamic Scheduled Sampling with Imitation Loss for Neural Text Generationcited by 0
  29. 2023Bilevel Scheduled Sampling for Dialogue Generationcited by 0
  30. 2023Multimodal Representations for Teacher-Guided Compositional Visual Reasoningcited by 0
  31. 2024Improving Autoregressive Training with Dynamic Oraclescited by 0
  32. 2026Pretraining Latent Information Feedback Transformers with Teacher Supervisioncited by 0
  33. 2026AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Modelscited by 0
  34. 2026A foundation for systematic analysis of transformers and RNNs for tractographycited by 0