# Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization

arXiv: https://arxiv.org/abs/2509.21871 · HTML: https://mlsanity.com/paper/2509.21871/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 34 of its 71 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)

## Builds on (most cited first, 30 of 34)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [Spurious Rewards: Rethinking Training Signals in RLVR](https://mlsanity.com/paper/2506.10947/) (2025)
- [Generative Reward Models](https://mlsanity.com/paper/2410.12832/) (2024)
- [Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution](https://mlsanity.com/paper/2501.11561/) (2025)
- [Photo Aesthetics Ranking Network with Attributes and Content Adaptation](https://mlsanity.com/paper/1606.01621/) (2016)
- [Q-Insight: Understanding Image Quality via Visual Reinforcement Learning](https://mlsanity.com/paper/2503.22679/) (2025)
- [VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank](https://mlsanity.com/paper/2505.14460/) (2025)
- [JudgeLRM: Large Reasoning Models as a Judge](https://mlsanity.com/paper/2504.00050/) (2025)
- [J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning](https://mlsanity.com/paper/2505.10320/) (2025)
- [VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining](https://mlsanity.com/paper/2303.14302/) (2023)
- [Personalized Image Aesthetics Assessment with Rich Attributes](https://mlsanity.com/paper/2203.16754/) (2022)
- [UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmark](https://mlsanity.com/paper/2404.09619/) (2024)
- [ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding](https://mlsanity.com/paper/2507.14533/) (2025)
- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)

## Built on by (most cited first, 7 of 7)

- [PhotoFramer: Multi-modal Image Composition Instruction](https://mlsanity.com/paper/2512.00993/) (2025)
- [AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation](https://mlsanity.com/paper/2603.28068/) (2026)
- [AesRM: Improving Video Aesthetics with Expert-Level Feedback](https://mlsanity.com/paper/2604.28078/) (2026)
- [ReasonEdit: Towards Interpretable Image Editing Evaluation via Reinforcement Learning](https://mlsanity.com/paper/2605.07477/) (2026)
- [Beyond Absolute Scores: Relative Edit-induced Difference for Generalizable Image Aesthetic Assessment](https://mlsanity.com/paper/2606.05778/) (2026)
- [SE-MoLoRA: Shared-Expert LoRA Adapters for Domain-Specific Photographic Assessment](https://mlsanity.com/paper/2608.17514/) (2026)
- [ScenePercept: Scenario-Conditioned Perceptual Assessment of Real-World User-Generated Content Images](https://mlsanity.com/paper/2610.17661/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) are well suited to image aesthetic assessment, as they can capture high-level aesthetic features leveraging their cross-modal understanding capacity. However, the scarcity of multimodal aesthetic reasoning data and the inherently subjective nature of aesthetic judgment make it difficult for MLLMs to generate accurate aesthetic judgments with interpretable rationales. To this end, we propose Aes-R1, a comprehensive aesthetic reasoning framework with reinforcement learning (RL). Concretely, Aes-R1 integrates a pipeline, AesCoT, to construct and filter high-quality chain-of-thought aesthetic reasoning data used for cold-start. After teaching the model to generate structured explanations prior to scoring, we then employ the Relative-Absolute Policy Optimization (RAPO), a novel RL algorithm that jointly optimizes absolute score regression and relative ranking order, improving both per-image accuracy and cross-image preference judgments. Aes-R1 enables MLLMs to generate grounded explanations alongside faithful scores, thereby enhancing aesthetic scoring and reasoning in a unified framework. Extensive experiments demonstrate that Aes-R1 improves the backbone's average PLCC/SRCC by 47.9%/34.8%, surpassing state-of-the-art baselines of similar size. More ablation studies validate Aes-R1's robust generalization under limited supervision and in out-of-distribution scenarios.