mlsanityData to 7 Oct 2026

Automatic Pair Construction for Contrastive Post-training

arXiv 2310.02263 ↗First posted 3 Oct 2023cs.CL cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Alignment serves as an important step to steer large language models (LLMs) towards human preferences. In this paper, we propose an automatic way to construct contrastive data for LLM, using preference pairs from multiple models of varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We compare the contrastive techniques of SLiC and DPO to SFT baselines and find that DPO provides a step-function improvement even after continuing SFT saturates. We also explore a data curriculum learning scheme for contrastive post-training, which starts by learning from "easier" pairs and transitioning to "harder" ones, which further improves alignment. Finally, we scale up our experiments to train with more data and larger models like Orca. Remarkably, our automatic contrastive post-training further improves the performance of Orca, already a state-of-the-art instruction learning model tuned with GPT-4 outputs, to outperform ChatGPT.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

28 of 40 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
16 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  9. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  10. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  11. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  12. 2021Consequences of Misaligned AIcited by 79
  13. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  14. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
  15. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  16. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2025Training LLM-based Tutors to Improve Student Learning Outcomes in Dialoguescited by 40
  2. 2025ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedbackcited by 23
  3. 2024Aligning Large Language Models with Counterfactual DPOcited by 3
  4. 2025Mixtera: A Data Plane for Foundation Model Trainingcited by 2
  5. 2025Factors That Support Grounded Responses in LLM Conversations: A Rapid Reviewcited by 0
  6. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0
  7. 2026Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  2. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  3. 2024LIONs: An Empirically Optimized Approach to Align Language Models
  4. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  5. 2024Extensive Self-Contrast Enables Feedback-Free Language Model Alignment
  6. 2026Which Pairs to Compare for LLM Post-Training?
  7. 2025Sequence-level Large Language Model Training with Contrastive Preference Optimization
  8. 2024Evolutionary Contrastive Distillation for Language Model Alignment