# Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI

arXiv: https://arxiv.org/abs/2511.01689 · HTML: https://mlsanity.com/paper/2511.01689/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 41 of its 62 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 41)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Iterative Reasoning Preference Optimization](https://mlsanity.com/paper/2404.19733/) (2024)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Personalization of Large Language Models: A Survey](https://mlsanity.com/paper/2411.00027/) (2024)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [A Survey of Personalized Large Language Models: Progress and Future Directions](https://mlsanity.com/paper/2502.11528/) (2025)
- [PersonalLLM: Tailoring LLMs to Individual Preferences](https://mlsanity.com/paper/2409.20296/) (2024)
- [Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics](https://mlsanity.com/paper/2406.14703/) (2024)
- [Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2504.12501/) (2025)
- [Specific versus General Principles for Constitutional AI](https://mlsanity.com/paper/2310.13798/) (2023)

## Built on by (most cited first, 24 of 24)

- [Shaping capabilities with token-level data filtering](https://mlsanity.com/paper/2601.21571/) (2026)
- [You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrases](https://mlsanity.com/paper/2603.09517/) (2026)
- [SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancy](https://mlsanity.com/paper/2604.02423/) (2026)
- [When Role-playing, Do Models Believe What They Say?](https://mlsanity.com/paper/2606.11502/) (2026)
- [Persona Cartography: Charting Language Model Personality Traits in Weight Space](https://mlsanity.com/paper/2607.07916/) (2026)
- [Probing Persona-Dependent Preferences in Language Models](https://mlsanity.com/paper/2605.13339/) (2026)
- [Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Games](https://mlsanity.com/paper/2606.17657/) (2026)
- [Incoherent Values? Probing LLM Preferences Through Parametric Variation](https://mlsanity.com/paper/2606.21102/) (2026)
- [(Mis)generalization of Helpful-only Fine-tuning](https://mlsanity.com/paper/2606.04413/) (2026)
- [Adversarial Robustness of Activation Steering in Large Language Models](https://mlsanity.com/paper/2606.07696/) (2026)
- [Measuring the Assistant's Harmlessness Preferences on the User Turn](https://mlsanity.com/paper/2609.23935/) (2026)
- ["Not Human, Funnier": How Machine Identity Shapes Humor Perception in Online AI Stand-up Comedy](https://mlsanity.com/paper/2602.12763/) (2026)
- [Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities](https://mlsanity.com/paper/2605.02897/) (2026)
- [How Value Induction Reshapes LLM Behaviour](https://mlsanity.com/paper/2605.07925/) (2026)
- [Understanding Goal Generalisation in Sequential Reinforcement Learning](https://mlsanity.com/paper/2605.23565/) (2026)
- [Coherence Maximization Improves Pluralistic Alignment](https://mlsanity.com/paper/2606.03110/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [Diff Mining: Logit Differences Reveal Finetuning Objectives](https://mlsanity.com/paper/2608.26462/) (2026)
- [Inducing Emergent Misalignment from Reward Hacks with Iterative DPO](https://mlsanity.com/paper/2609.06649/) (2026)
- [Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble](https://mlsanity.com/paper/2609.10883/) (2026)
- [Math for AI safety: an invitation for mathematicians](https://mlsanity.com/paper/2609.15289/) (2026)
- [Steering Language Model Goals with Value Transplant](https://mlsanity.com/paper/2609.34056/) (2026)
- [Character Training for Risk-Averse Agents](https://mlsanity.com/paper/2609.38093/) (2026)
- [Predicting Alignment Generalization with Value Representations](https://mlsanity.com/paper/2610.12410/) (2026)

## Abstract (arXiv)

The character of the "AI assistant" persona generated by modern chatbot large language models influences both surface-level behavior and apparent values, beliefs, and ethics. These all affect interaction quality, perceived intelligence, and alignment with both developer and user intentions. The shaping of this persona, known as character training, is a critical component of industry post-training, yet remains effectively unstudied in the academic literature. We introduce the first open implementation of character training, leveraging Constitutional AI and a new data pipeline using synthetic introspective data to shape the assistant persona in a more effective and controlled manner than alternatives such as constraining system prompts or activation steering. Specifically, we fine-tune three popular open-weights models using 11 example personas, such as humorous, deeply caring, or even malevolent. To track the effects of our approach, we introduce a method which analyzes revealed preferences, uncovering clear and holistic changes in character. We find these changes are more robust to adversarial prompting than the above two alternatives, while also leading to more coherent and realistic generations. Finally, we demonstrate this fine-tuning has little to no effect on general capabilities as measured by common benchmarks. We describe and open-source our full post-training method, the implementation of which can be found at https://github.com/maiush/OpenCharacterTraining.