# Character Training for Risk-Averse Agents

arXiv: https://arxiv.org/abs/2609.38093 · HTML: https://mlsanity.com/paper/2609.38093/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Model Organisms for Emergent Misalignment](https://mlsanity.com/paper/2506.11613/) (2025)
- [Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment](https://mlsanity.com/paper/2601.10160/) (2026)
- [STEER: Assessing the Economic Rationality of Large Language Models](https://mlsanity.com/paper/2402.09552/) (2024)
- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [Model Spec Midtraining: Improving How Alignment Training Generalizes](https://mlsanity.com/paper/2605.02087/) (2026)
- [Emergent Risk Awareness in Rational Agents under Resource Constraints](https://mlsanity.com/paper/2505.23436/) (2025)
- [The LLM Has Left The Chat: Evidence of Bail Preferences in Large Language Models](https://mlsanity.com/paper/2509.04781/) (2025)
- [A dataset of questions on decision-theoretic reasoning in Newcomb-like problems](https://mlsanity.com/paper/2411.10588/) (2024)
- [When Role-playing, Do Models Believe What They Say?](https://mlsanity.com/paper/2606.11502/) (2026)
- [Persona Cartography: Charting Language Model Personality Traits in Weight Space](https://mlsanity.com/paper/2607.07916/) (2026)
- [Constitutional Midtraining: Content Presence Drives Alignment Gains](https://mlsanity.com/paper/2607.26654/) (2026)
- [Out-of-Distribution Generalization of Risk Aversion in Language Models](https://mlsanity.com/paper/2607.02755/) (2026)
- [Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models](https://mlsanity.com/paper/2607.26173/) (2026)
- [A dataset of rated conceptual arguments](https://mlsanity.com/paper/2607.27499/) (2026)
- [Stress-testing Alignment Midtraining](https://mlsanity.com/paper/2609.20412/) (2026)

## Abstract (arXiv)

Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making deals with humans over riskier strategies like rebelling. We train agents to be risk averse through character training, finding that persona traits provide a robust mechanism for instilling risk preferences. To do this, we construct a model constitution describing constant absolute risk aversion (CARA) over an agent's resources and instill it through on-policy distillation. Despite never seeing the benchmark's decision format during training, character-trained models are competitive with baselines trained directly on it, and generalise better than them out of distribution on two of our four models. We also modulate different aspects of the constitution, finding that token budget and model choice are the most influential aspect of character training to instill risk aversion. We conclude from these results that character training is a promising and scalable way to instil broad dispositions, which we can use to our advantage in mitigating risk from misaligned AI agents.