mlsanityData to 7 Oct 2026

Dual-objective Language Models: Training Efficiency Without Overfitting

arXiv 2512.14549 ↗First posted 16 Dec 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper combines autoregressive and masked-diffusion training objectives without any architectural modifications, resulting in flexible language models that outperform single-objective models. Autoregressive modeling has been a popular approach, partly because of its training efficiency; however, that comes at the cost of sensitivity to overfitting. On the other hand, masked-diffusion models are less efficient to train while being more resilient to overfitting. In this work, we demonstrate that dual-objective training achieves the best of both worlds. To derive the optimal balance between both objectives, we train and evaluate 50 language models under varying levels of data repetition. We show that it is optimal to combine both objectives under all evaluated settings and that the optimal balance is similar whether targeting autoregressive or masked-diffusion downstream performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 59 references that are in the map. First posted on arXiv 16 Dec 2025.

Builds on

44 of 59 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
32 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  6. 2025Large Language Diffusion Modelscited by 872
  7. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  9. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  12. 2023Scaling Data-Constrained Language Modelscited by 391
  13. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  14. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  15. 2022UL2: Unifying Language Learning Paradigmscited by 292
  16. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  17. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  18. 2024Scaling up Masked Diffusion Models on Textcited by 195
  19. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  20. 2021Thinking Like Transformerscited by 182
  21. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  22. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  23. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  24. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 113
  25. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  26. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  27. 2024GPT or BERT: why not both?cited by 26
  28. 2023An Analysis and Mitigation of the Reversal Cursecited by 22
  29. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecturecited by 16
  30. 2024BERTs are Generative In-Context Learnerscited by 11
  31. 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
  32. 2024Segment-Based Attention Masking for GPTscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison
  2. 2026Improved Large Language Diffusion Models
  3. 2025Diffusion Beats Autoregressive in Data-Constrained Settings
  4. 2023Likelihood-Based Diffusion Language Models
  5. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  6. 2026Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference
  7. 2026UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
  8. 2026Enhancing Diffusion Language Models with Autoregressive Post-Training Weights