mlsanityData to 14 Oct 2026

Alignment for Honesty

arXiv 2312.07000 ↗First posted 12 Dec 2023cs.CL cs.AIcited by 74

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent research has made significant strides in aligning large language models (LLMs) with helpfulness and harmlessness. In this paper, we argue for the importance of alignment for honesty, ensuring that LLMs proactively refuse to answer questions when they lack knowledge, while still not being overly conservative. However, a pivotal aspect of alignment for honesty involves discerning an LLM's knowledge boundaries, which demands comprehensive solutions in terms of metric development, benchmark creation, and training methodologies. We address these challenges by first establishing a precise problem definition and defining ``honesty'' inspired by the Analects of Confucius. This serves as a cornerstone for developing metrics that effectively measure an LLM's honesty by quantifying its progress post-alignment. Furthermore, we introduce a flexible training framework which is further instantiated by several efficient fine-tuning techniques that emphasize honesty without sacrificing performance on other tasks. Our extensive experiments reveal that these aligned models show a marked increase in honesty, as indicated by our proposed metrics. We open-source all relevant resources to facilitate future research at urlhttps://github.com/GAIR-NLP/alignment-for-honesty.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 69 of its 89 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

69 of 89 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Mitigating the Alignment Tax of RLHF12 Sep
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  3. 2023LIMA: Less Is More for Alignment18 May
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar
  6. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 63

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  13. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  14. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  15. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  16. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  17. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  18. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  19. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  20. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  21. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  22. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  23. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  24. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  25. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  26. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  27. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  28. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  29. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  30. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  31. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  32. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  33. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  34. 2023Challenges and Applications of Large Language Modelscited by 254
  35. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  36. 2023Do Large Language Models Know What They Don't Know?cited by 243
  37. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  38. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 28; 2026: 20.

74 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2024Language Models with Conformal Factuality Guaranteescited by 131
  3. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  4. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  5. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 87
  6. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 72
  7. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 66
  8. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  9. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  10. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMscited by 36
  11. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?cited by 34
  12. 2024How Susceptible are LLMs to Influence in Prompts?cited by 33
38 more
  1. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  2. 2024BeHonest: Benchmarking Honesty in Large Language Modelscited by 24
  3. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  4. 2024Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?cited by 23
  5. 2024A Survey on the Honesty of Large Language Modelscited by 23
  6. 2024I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBenchcited by 21
  7. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
  8. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  9. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 15
  10. 2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
  11. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  12. 2024Multi-group Uncertainty Quantification for Long-form Text Generationcited by 13
  13. 2024Fine-Tuning Large Language Models to Appropriately Abstain with Semantic Entropycited by 11
  14. 2025MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibrationcited by 11
  15. 2024Utilize the Flow before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuningcited by 10
  16. 2025CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstentioncited by 10
  17. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  18. 2025Alignment of large language models with constrained learningcited by 8
  19. 2025Federation of Agents: A Semantics-Aware Communication Fabric for Large-Scale Agentic AIcited by 7
  20. 2024VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoningcited by 6
  21. 2025Aligning Large Language Models for Faithful Integrity Against Opposing Argumentcited by 6
  22. 2025When Do LLMs Admit Their Mistakes? Understanding The Role Of Model Belief In Retractioncited by 6
  23. 2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 6
  24. 2025Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questionscited by 6
  25. 2025MASH: Modeling Abstention via Selective Help-Seekingcited by 6
  26. 2025Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMscited by 5
  27. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 5
  28. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Datacited by 4
  29. 2025The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factualitycited by 4
  30. 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
  31. 2024Dishonesty in Helpful and Harmless Alignmentcited by 3
  32. 2024Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editingcited by 3
  33. 2025SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstentioncited by 3
  34. 2025On the Robustness of Verbal Confidence of LLMs in Adversarial Attackscited by 3
  35. 2025Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Processcited by 3
  36. 2025Annotation-Efficient Universal Honesty Alignmentcited by 3
  37. 2026PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decodingcited by 3
  38. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scalingcited by 3

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024HonestLLM: Toward an Honest and Helpful Large Language Model
  2. 2026Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning
  3. 2026FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
  4. 2026When the Model Said 'No Comment', We Knew Helpfulness Was Dead, Honesty Was Alive, and Safety Was Terrified
  5. 2025Too Helpful, Too Harmless, Too Honest or Just Right?
  6. 2026Emergent Alignment
  7. 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?
  8. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA