mlsanityData to 14 Oct 2026

QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA

arXiv 2506.08123 ↗First posted 9 Jun 2025cs.CLcited by 28

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment of large language models (LLMs) with principles like helpfulness, honesty, and harmlessness typically relies on scalar rewards that obscure which objectives drive the training signal. We introduce QA-LIGN, which decomposes monolithic rewards into interpretable principle-specific evaluations through structured natural language programs. Models learn through a draft, critique, and revise pipeline, where symbolic evaluation against the rubrics provides transparent feedback for both initial and revised responses during GRPO training. Applied to uncensored Llama-3.1-8B-Instruct, QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate, achieving Pareto optimal safety-helpfulness performance and outperforming both DPO and GRPO with state-of-the-art reward models given equivalent training. These results demonstrate that making reward signals interpretable and modular improves alignment effectiveness, suggesting transparency enhances LLM safety.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 62 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

46 of 62 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Direct Preference Optimization with an Offset16 Feb
  2. 2023SALMON: Self-Alignment with Instructable Reward Models9 Oct
  3. 2023LIMA: Less Is More for Alignment18 May
  4. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 41

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
29 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  5. 2022Red Teaming Language Models with Language Modelscited by 1,026
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  7. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  11. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  13. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  14. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  15. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  16. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  17. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  18. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  19. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 156
  20. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  21. 2024Rule Based Rewards for Language Model Safetycited by 108
  22. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
  23. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 78
  24. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
  25. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  26. 2023Specific versus General Principles for Constitutional AIcited by 47
  27. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  28. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 37
  29. 2024When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Cipherscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 18.

28 papers.

Sort:

By area:

  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  4. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  5. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 12
  6. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 11
  7. 2026Self-Preference Bias in Rubric-Based Evaluation of Large Language Modelscited by 11
  8. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modelingcited by 6
  9. 2026Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judgecited by 6
  10. 2026AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judgecited by 5
  11. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  12. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 5
16 more
  1. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 5
  2. 2025Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraintscited by 4
  3. 2026SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utilitycited by 4
  4. 2026Asking What Matters: Reward-Driven Clarification for Software Engineering Taskscited by 4
  5. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  6. 2025RECAP: Transparent Inference-Time Emotion Alignment for Medical Dialogue Systemscited by 3
  7. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 3
  8. 2026Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scalecited by 2
  9. 2025Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewardscited by 1
  10. 2026When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-Oncited by 1
  11. 2026Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolutioncited by 1
  12. 2026Grounded Checklist Partial Credit for Agent Skill Trajectoriescited by 1
  13. 2025Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applicationscited by 0
  14. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0
  15. 2026Decomposer: Learning to Decompile Symbolic Music to Programscited by 0
  16. 2026From Constitutions to Control: Interpretable Rewards for Aligning Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  2. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  3. 2024ABC Align: Large Language Model Alignment for Safety & Accuracy
  4. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  5. 2024When to Trust LLMs: Aligning Confidence with Response Quality
  6. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
  7. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMs
  8. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!