# Consistency Training Helps Stop Sycophancy and Jailbreaks

arXiv: https://arxiv.org/abs/2510.27062 · HTML: https://mlsanity.com/paper/2510.27062/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 23 of 23)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models](https://mlsanity.com/paper/2406.18510/) (2024)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [Defending Against Unforeseen Failure Modes with Latent Adversarial Training](https://mlsanity.com/paper/2403.05030/) (2024)
- [Representation Bending for Large Language Model Safety](https://mlsanity.com/paper/2504.01550/) (2025)
- [Linear Probe Penalties Reduce LLM Sycophancy](https://mlsanity.com/paper/2412.00967/) (2024)
- [Scaling Trends in Language Model Robustness](https://mlsanity.com/paper/2407.18213/) (2024)

## Built on by (most cited first, 16 of 16)

- [Training Agents to Self-Report Misbehavior](https://mlsanity.com/paper/2602.22303/) (2026)
- [Positive Alignment: Artificial Intelligence for Human Flourishing](https://mlsanity.com/paper/2605.10310/) (2026)
- [Automated Researchers Can Mitigate Well-characterized Alignment Failures](https://mlsanity.com/paper/2608.28945/) (2026)
- [Consistency Training while Mitigating Obfuscation via Rate Matching](https://mlsanity.com/paper/2606.02211/) (2026)
- [On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation](https://mlsanity.com/paper/2605.21834/) (2026)
- [Consistency Training Can Entrench Misalignment](https://mlsanity.com/paper/2606.03810/) (2026)
- [Incoherent Values? Probing LLM Preferences Through Parametric Variation](https://mlsanity.com/paper/2606.21102/) (2026)
- [What are the Right Symmetries for Formal Theorem Proving?](https://mlsanity.com/paper/2605.22257/) (2026)
- [Consistency Training Along the Transformer Stack](https://mlsanity.com/paper/2606.05817/) (2026)
- [Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs](https://mlsanity.com/paper/2606.16011/) (2026)
- [Position: It's Time to Optimize LLMs for Self-Consistency](https://mlsanity.com/paper/2608.05188/) (2026)
- [Self-Mined Hardness for Safety Fine-Tuning](https://mlsanity.com/paper/2605.03226/) (2026)
- [Mitigating Adaptive Attacks against Reasoning Models with Activation Consistency Training](https://mlsanity.com/paper/2605.28467/) (2026)
- [Unlearning Deceptive Behaviors in LLMs with Contrastive Forget Sets](https://mlsanity.com/paper/2609.38909/) (2026)
- [RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents](https://mlsanity.com/paper/2610.06401/) (2026)
- [Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements](https://mlsanity.com/paper/2610.08540/) (2026)

## Abstract (arXiv)

An LLM's factuality and refusal training can be compromised by simple changes to a prompt. Models often adopt user beliefs (sycophancy) or satisfy inappropriate requests which are wrapped within special text (jailbreaking). We explore consistency training, a self-supervised paradigm that teaches a model to be invariant to certain irrelevant cues in the prompt. Instead of teaching the model what exact response to give on a particular prompt, we aim to teach the model to behave identically across prompt data augmentations (like adding leading questions or jailbreak text). We try enforcing this invariance in two ways: over the model's external outputs (Bias-augmented Consistency Training (BCT) from Chua et al. [2025]) and over its internal activations (Activation Consistency Training (ACT), a method we introduce). Both methods reduce Gemini 2.5 Flash's susceptibility to irrelevant cues. Because consistency training uses responses from the model itself as training data, it avoids issues that arise from stale training data, such as degrading model capabilities or enforcing outdated response guidelines. While BCT and ACT reduce sycophancy equally well, BCT does better at jailbreak reduction. We think that BCT can simplify training pipelines by removing reliance on static datasets. We argue that some alignment problems are better viewed not in terms of optimal responses, but rather as consistency issues.