mlsanityData to 14 Oct 2026

Anchored Alignment for Self-Explanations Enhancement

arXiv 2410.13216 ↗First posted 17 Oct 2024cs.AI cs.CLcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

In this work, we introduce a methodology for alignment designed to enhance the ability of large language models (LLMs) to articulate their reasoning (self-explanation) even in the absence of annotated rationale explanations. Our alignment methodology comprises three key components: explanation quality assessment, self-instruction dataset generation, and model alignment. Additionally, we present a novel technique called Alignment with Anchor Preference Pairs, which improves the selection of preference pairs by categorizing model outputs into three groups: consistently correct, consistently incorrect, and variable. By applying tailored strategies to each category, we enhance the effectiveness of Direct Preference Optimization (DPO). Our experimental results demonstrate that this approach significantly improves explanation quality while maintaining accuracy compared to other fine-tuning strategies.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 36 of its 39 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

36 of 39 references.

References · 36

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  9. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
24 more
  1. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  2. 2024Self-Rewarding Language Modelscited by 681
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  4. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  6. 2024LoRA Learns Less and Forgets Lesscited by 328
  7. 2023Faithful Chain-of-Thought Reasoningcited by 288
  8. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  9. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  10. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  11. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  12. 2023Self-Alignment with Instruction Backtranslationcited by 151
  13. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  14. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  15. 2024Are self-explanations from Large Language Models faithful?cited by 101
  16. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  17. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  18. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 80
  19. 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanationscited by 73
  20. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 47
  21. 2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 36
  22. 2019Finding Generalizable Evidence by Learning to Convince Q&A Modelscited by 25
  23. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniquescited by 16
  24. 2023Faithfulness Measurable Masked Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning What LLMs Do and Say: Towards Self-Consistent Explanations
  2. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  3. 2024Is Free Self-Alignment Possible?
  4. 2024Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models
  5. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  6. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  7. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  8. 2024LIONs: An Empirically Optimized Approach to Align Language Models