mlsanityData to 7 Oct 2026

Binary Classifier Optimization for Large Language Model Alignment

arXiv 2404.04656 ↗First posted 6 Apr 2024cs.LG cs.AI cs.CLcited by 44

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

In real-world services such as ChatGPT, aligning models based on user feedback is crucial for improving model performance. However, due to the simplicity and convenience of providing feedback, users typically offer only basic binary signals, such as 'thumbs-up' or 'thumbs-down'. Most existing alignment research, on the other hand, relies on preference-based approaches that require both positive and negative responses as a pair. We propose Binary Classifier Optimization (BCO), a technique that effectively aligns LLMs using only binary feedback. BCO trains a binary classifier, where the logit serves as an implicit reward, effectively minimizing the Direct Preference Optimization (DPO) loss. We demonstrate that the binary cross-entropy loss employed in classifier training acts as an upper bound for the DPO loss. Additionally, a novel reward shift technique further minimizes the gap between the losses. We validate our methodology in two settings: first, on a paired preference dataset, where our method performs on par with DPO; and second, on a Likert-5 scale annotation dataset which stems from real users' queries. Our model consistently demonstrates effective and robust alignment across four base LLMs and three different datasets, showcasing the strength of our approach to learning from binary signals.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 37 references that are in the map. First posted on arXiv 6 Apr 2024.

Builds on

17 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2024Self-Rewarding Language Modelscited by 654
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  12. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
1 more
  1. 2024Stable LM 2 1.6B Technical Reportcited by 86

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 23; 2026: 14.

44 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  4. 2025Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Modelscited by 51
  5. 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 48
  6. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  7. 2025A Survey of Direct Preference Optimizationcited by 35
  8. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  9. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  10. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  11. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  12. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignmentcited by 7
32 more
  1. 2024Takin: A Cohort of Superior Quality Zero-shot Speech Generation Modelscited by 7
  2. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
  3. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  4. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual contextcited by 6
  5. 2025Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Modelscited by 6
  6. 2026Learning from Language Feedback via Variational Policy Distillationcited by 6
  7. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  8. 2025C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMscited by 5
  9. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  10. 2024Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Reviewcited by 3
  11. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignmentcited by 3
  12. 2025MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignmentcited by 3
  13. 2025Prompt and Parameter Co-Optimization for Large Language Modelscited by 2
  14. 2026Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retentioncited by 2
  15. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Datacited by 1
  16. 2025Best of mini-N in-loop Sampling: A Contextual Quality Reward Model for Reliable and Efficient Best-of-N Samplingcited by 1
  17. 2026Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributescited by 1
  18. 2026HTPO: Towards Exploration-Exploitation Balanced Policy Optimization via Hierarchical Token-level Objective Controlcited by 1
  19. 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Frameworkcited by 1
  20. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  21. 2025Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimatorcited by 0
  22. 2025OCC-MLLM-CoT-Alpha: Towards Multi-stage Occlusion Recognition Based on Large Language Models via 3D-Aware Supervision and Chain-of-Thoughts Guidancecited by 0
  23. 2025CALMA: A Process for Deriving Context-aligned Axes for Language Model Alignmentcited by 0
  24. 2025Humanline: Online Alignment as Perceptual Losscited by 0
  25. 2026Ostrakon-VL: Towards Domain-Expert MLLM for Food-Service and Retail Storescited by 0
  26. 2026BinaryPPO: Efficient Policy Optimization for Binary Classificationcited by 0
  27. 2026Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?cited by 0
  28. 2026Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignmentcited by 0
  29. 2026Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generationcited by 0
  30. 2026Convex Optimization for Alignment and Preference Learning on a Single GPUcited by 0
  31. 2026The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoningcited by 0
  32. 2026GuardPaint:SpeculativeSafetyDecodingforText-to-ImageGenerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning CodeLLMs with Direct Preference Optimization
  2. 2024TODO: Enhancing LLM Alignment with Ternary Preferences
  3. 2024SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks
  4. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
  5. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Training
  6. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
  7. 2026Small-Margin Preferences Still Matter-If You Train Them Right
  8. 2025ComPO: Preference Alignment via Comparison Oracles