mlsanityData to 14 Oct 2026

Via Negativa for AI Alignment: Why Negative Constraints Are Structurally Superior to Positive Preferences

arXiv 2603.16417 ↗First posted 17 Mar 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent empirical results have demonstrated that training large language models (LLMs) with negative-only feedback can match or exceed standard reinforcement learning from human feedback (RLHF). Negative Sample Reinforcement achieves parity with PPO on mathematical reasoning; Distributional Dispreference Optimization trains effectively using only dispreferred samples; and Constitutional AI outperforms pure RLHF on harmlessness benchmarks. Yet no unified theoretical account explains why negative signals are so effective. This paper proposes such an account: positive preferences and negative constraints are structurally asymmetric. Positive preferences ("which is better") encode continuously coupled, context-dependent human values that cannot be exhaustively specified -- leading models to learn surface correlates such as agreement with the user (sycophancy). Negative constraints ("what is wrong") encode discrete, finite, independently verifiable prohibitions that can converge to a stable boundary. This asymmetry -- rooted in Popper's falsification logic and the epistemology of negative knowledge -- explains both the sycophancy failure of preference-based RLHF and the surprising effectiveness of negative-signal methods. We argue that alignment research should shift its center of gravity from "learning what humans prefer" to "learning what humans reject," and offer testable predictions for this framework.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 18 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

10 of 18 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss7 Oct
  2. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  3. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization6 Mar

Other references · 7

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  3. 2023Large Language Model Unlearningcited by 302
  4. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  5. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  6. 2026How RLHF Amplifies Sycophancycited by 23
  7. 2026Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Onescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RLHF May Not Reflect Genuine Preferences
  2. 2026Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner
  3. 2026Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning
  4. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
  5. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  6. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  7. 2024Negative-Prompt-driven Alignment for Generative Language Model
  8. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications