mlsanityData to 7 Oct 2026

Sequence-level Large Language Model Training with Contrastive Preference Optimization

arXiv 2502.16433 ↗First posted 23 Feb 2025cs.CL cs.AI cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The next token prediction loss is the dominant self-supervised training objective for large language models and has achieved promising results in a variety of downstream tasks. However, upon closer investigation of this objective, we find that it lacks an understanding of sequence-level signals, leading to a mismatch between training and inference processes. To bridge this gap, we introduce a contrastive preference optimization (CPO) procedure that can inject sequence-level information into the language model at any training stage without expensive human labeled data. Our experiments show that the proposed objective surpasses the next token prediction in terms of win rate in the instruction-following and text generation tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 48 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

28 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2021Robust fine-tuning of zero-shot modelscited by 785
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
15 more
  1. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  2. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  3. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  4. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  5. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  6. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  7. 2019Generalization in Generation: A closer look at Exposure Biascited by 123
  8. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  9. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  10. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  11. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  12. 2020Residual Energy-Based Models for Text Generationcited by 95
  13. 2020Text Generation by Learning from Demonstrationscited by 64
  14. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  15. 2019Parallel Scheduled Samplingcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
  2. 2025Learning to Focus: Focal Attention for Selective and Scalable Transformerscited by 2
  3. 2026d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuning
  2. 2024Extensive Self-Contrast Enables Feedback-Free Language Model Alignment
  3. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  4. 2024Evolutionary Contrastive Distillation for Language Model Alignment
  5. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
  6. 2024Aligning Large Language Models via Fine-grained Supervision
  7. 2023Automatic Pair Construction for Contrastive Post-training
  8. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Data