mlsanityData to 14 Oct 2026

C-Instrument: Automating RL Data Generation and Hillclimbing with a Constitution-Grid Instrument

arXiv 2608.00180 ↗First posted 31 Jul 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Conflicting objectives are general in RL alignment, and training on them data-efficiently is hard. Training a safety guard with RL means optimizing two objectives that conflict: catch real harm, and do not refuse benign prompts. Our finding is that over-refusal improves 22.4% to 12.8%, while under-refusal on adversarial attacks silently worsens 0.27 to 0.33. We present C-Instrument, a constitution-grid data instrument that generates the RL training data, and C-LIM, a per-cell learnability score that decides each cell's move: prune, densify, amend, expand. C-LIM flags the dead-weight data region before any training budget is spent: 187 untargeted rows had bought zero gain, and our method lifts the same region's learning impact 0.733 to 0.80. Code and the constitution are open-sourced.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 17 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

16 of 17 references.

References · 16

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  7. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  8. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  9. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  10. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  11. 2025LIMR: Less is More for RL Scalingcited by 139
  12. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 100
4 more
  1. 2025Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Modelscited by 16
  2. 2024Chasing Random: Instruction Selection Strategies Fail to Generalizecited by 13
  3. 2025Calibrated Reasoning: An Explanatory Verifier for Dynamic and Efficient Problem-Solvingcited by 1
  4. 2026HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safetycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment
  2. 2025LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment
  3. 2026Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF
  4. 2026Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations
  5. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay
  6. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  7. 2025Assuring the Safety of Reinforcement Learning Components: AMLAS-RL
  8. 2025LLM Misalignment via Adversarial RLHF Platforms