mlsanityData to 7 Oct 2026

Meet in the Middle: A New Pre-training Paradigm

arXiv 2303.07295 ↗First posted 13 Mar 2023cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion, assuming that the next token only depends on the preceding ones. However, this assumption ignores the potential benefits of using the full sequence information during training, and the possibility of having context from both sides during inference. In this paper, we propose a new pre-training paradigm with techniques that jointly improve the training data efficiency and the capabilities of the LMs in the infilling task. The first is a training objective that aligns the predictions of a left-to-right LM with those of a right-to-left LM, trained on the same data but in reverse order. The second is a bidirectional inference procedure that enables both LMs to meet in the middle. We show the effectiveness of our pre-training paradigm with extensive experiments on both programming and natural language models, outperforming strong baselines.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 29 of its 30 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

29 of 30 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
17 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2017Mixed Precision Trainingcited by 1,135
  3. 2021Measuring Coding Challenge Competence With APPScited by 879
  4. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  7. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  8. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  9. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  10. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  11. 2020Enabling Language Models to Fill in the Blankscited by 121
  12. 2022Bidirectional Language Models Are Also Few-shot Learnerscited by 48
  13. 2018Regularizing Neural Machine Translation by Target-bidirectional Agreementcited by 40
  14. 2019Synchronous Bidirectional Neural Machine Translationcited by 27
  15. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18
  16. 2017Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioningcited by 8
  17. 2022Image Captioning via Compact Bidirectional Architecturecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 7; 2025: 6; 2026: 3.

22 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  5. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  6. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  7. 2024Arrows of Time for Large Language Modelscited by 18
  8. 2026DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvascited by 16
  9. 2025Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariescited by 14
  10. 2024The Belief State Transformercited by 12
  11. 2025Enabling Autoregressive Models to Fill In Masked Tokenscited by 7
  12. 2024Empowering Character-level Text Infilling by Eliminating Sub-Tokenscited by 5
10 more
  1. 2025MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilitiescited by 5
  2. 2023Self-Infilling Code Generationcited by 3
  3. 2026Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretrainingcited by 3
  4. 2024Reverse Modeling in Large Language Modelscited by 2
  5. 2025Bidirectional Action Sequence Learning for Long-term Action Anticipation with Large Language Modelscited by 2
  6. 2024Locally Differentially Private Embedding Models in Distributed Fraud Prevention Systemscited by 1
  7. 2025Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepanciescited by 1
  8. 2025From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation?cited by 1
  9. 2024Acquiring Bidirectionality via Large and Small Language Modelscited by 0
  10. 2026Line-Coupled Language Modelcited by 0