# Probe-based Fine-tuning for Reducing Toxicity

arXiv: https://arxiv.org/abs/2510.21531 · HTML: https://mlsanity.com/paper/2510.21531/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 12 references that are in the map.
- First posted on arXiv 24 Oct 2025.

## Builds on (most cited first, 9 of 9)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Detecting Strategic Deception Using Linear Probes](https://mlsanity.com/paper/2502.03407/) (2025)
- [Obfuscated Activations Bypass LLM Latent-Space Defenses](https://mlsanity.com/paper/2412.09565/) (2024)
- [RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?](https://mlsanity.com/paper/2506.14261/) (2025)
- [A benchmark for toxic comment classification on Civil Comments dataset](https://mlsanity.com/paper/2301.11125/) (2023)
- [Preference Learning with Lie Detectors can Induce Honesty or Evasion](https://mlsanity.com/paper/2505.13787/) (2025)

## Built on by (most cited first, 2 of 2)

- [Inducing Artificial Uncertainty in Language Models](https://mlsanity.com/paper/2605.13595/) (2026)
- [Alignment via Training Against Probes Without Losing Monitorability](https://mlsanity.com/paper/2609.38645/) (2026)

## Abstract (arXiv)

Probes trained on model activations can detect undesirable behaviors like deception or biases that are difficult to identify from outputs alone. This makes them useful detectors to identify misbehavior. Furthermore, they are also valuable training signals, since they not only reward outputs, but also good internal processes for arriving at that output. However, training against interpretability tools raises a fundamental concern: when a monitor becomes a training target, it may cease to be reliable (Goodhart's Law). We propose two methods for training against probes based on Supervised Fine-tuning and Direct Preference Optimization. We conduct an initial exploration of these methods in a testbed for reducing toxicity and evaluate the amount by which probe accuracy drops when training against them. To retain the accuracy of probe-detectors after training, we attempt (1) to train against an ensemble of probes, (2) retain held-out probes that aren't used for training, and (3) retrain new probes after training. First, probe-based preference optimization unexpectedly preserves probe detectability better than classifier-based methods, suggesting the preference learning objective incentivizes maintaining rather than obfuscating relevant representations. Second, probe diversity provides minimal practical benefit - simply retraining probes after optimization recovers high detection accuracy. Our findings suggest probe-based training can be viable for certain alignment methods, though probe ensembles are largely unnecessary when retraining is feasible.