mlsanityData to 14 Oct 2026

Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only

arXiv 2410.11055 ↗First posted 14 Oct 2024cs.CL cs.AIcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

In the absence of abundant reliable annotations for challenging tasks and contexts, how can we expand the frontier of LLM capabilities with potentially wrong answers? We focus on two research questions: (1) Can LLMs generate reliable preferences among wrong options? And if so, (2) Would alignment with such wrong-over-wrong preferences be helpful? We employ methods based on self-consistency, token probabilities, and LLM-as-a-judge to elicit wrong-over-wrong preferences, and fine-tune language models with preference optimization approaches using these synthesized preferences. Extensive experiments with seven LLMs and eight datasets demonstrate that (1) LLMs do have preliminary capability in distinguishing various shades of wrong, achieving up to 20.9% higher performance than random guess; (2) Alignment with wrong-over-wrong preferences helps LLMs to produce less wrong and sometimes even outright correct answers, while overall improving model calibration.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 78 of its 95 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

78 of 95 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration22 Jun
  2. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision14 Mar
  3. 2024Debating with More Persuasive LLMs Leads to More Truthful Answers9 Feb
  4. 2024A Roadmap to Pluralistic Alignment7 Feb
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  6. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models30 Aug
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  8. 2020Learning to summarize from human feedback2 Sep

Other references · 70

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2023Mistral 7Bcited by 3,164
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  13. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  14. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  15. 2024Self-Rewarding Language Modelscited by 681
  16. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  17. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  18. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  19. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  20. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  21. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  22. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  23. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  24. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  25. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  26. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  27. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  28. 2023Nash Learning from Human Feedbackcited by 224
  29. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  30. 2023Generative Judge for Evaluating Alignmentcited by 169
  31. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 159
  32. 2023Self-Alignment with Instruction Backtranslationcited by 151
  33. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  34. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  35. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 107
  36. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 98
  37. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  38. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 28
  2. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 23
  3. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  4. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 9
  5. 2025MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generationcited by 6
  6. 2026Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?cited by 1
  7. 2026Jailbreaking Large Language Models with Morality Attackscited by 1
  8. 2026From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Mergingcited by 1
  9. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  2. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  3. 2023Fake Alignment: Are LLMs Really Aligned Well?
  4. 2025Aligning What LLMs Do and Say: Towards Self-Consistent Explanations
  5. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
  6. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  7. 2024Dissecting Human and LLM Preferences
  8. 2026The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs