VA-DPO: Valence-Arousal Direct Preference Optimization for Controllable Emotion Generation in Language Models
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
How precisely can we tell a language model how to feel? Most work on emotional generation answers with a discrete label - happy, angry, sad - which cannot express a target like "mildly downcast but calm." We instead specify the desired affect as a continuous point (v*, a*) in the Valence-Arousal plane and train the model to hit it. Our method, VA-DPO, is a small modification to Direct Preference Optimization: a frozen VA regressor scores each sampled generation by its Euclidean distance to the target, we keep only candidate pairs whose distance gap clears a margin tau, and we optimize a LoRA adapter with the ordinary DPO loss against a frozen reference. The DPO objective itself is unchanged; what is new is how the preference data is built. On Llama-3.1-8B-Instruct this cuts mean VA distance to the target by 33% over system-prompting and 25% over few-shot prompting, lifting valence/arousal correlation to r_v=0.93 and r_a=0.75. The gains carry over to Qwen3-8B and Llama-3.2-3B, and they do not come at the usual price: MMLU is unchanged (Delta=+0.0) and HellaSwag and TruthfulQA are preserved. We release the code, configs, and the preference-construction pipeline.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 17 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
10 of 17 references.
References · 10
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2022EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysiscited by 76
- 2024Style Vectors for Steering Generative Large Language Modelcited by 56
- 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 32
- 2026Valence-Arousal Subspace in LLMs: Circular Emotion Geometry and Multi-Behavioral Controlcited by 16
- 2025Don't Get Too Excited -- Eliciting Emotions in LLMscited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026You Can't Prefer Emotions You Don't Sample: Intensity Undershoot in DPO-Tuned LLMs
- 2026Learning to Prefer Reliably: Error-Augmented Emotion Preference Optimization with Calibrated Fusion
- 2024EmPO: Emotion Grounding for Empathetic Response Generation through Preference Optimization
- 2025On the Role of Preference Variance in Preference Optimization
- 2025A Survey of Direct Preference Optimization
- 2024New Desiderata for Direct Preference Optimization
- 2024Direct Preference Optimization with an Offset
- 2025EmoPrefer: Can Large Language Models Understand Human Emotion Preferences?