# Trajectory Improvement and Reward Learning from Comparative Language Feedback

arXiv: https://arxiv.org/abs/2410.06401 · HTML: https://mlsanity.com/paper/2410.06401/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2024.

## Built on by (most cited first, 17 of 17)

- [Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://mlsanity.com/paper/2603.02115/) (2026)
- [MILE: Model-based Intervention Learning](https://mlsanity.com/paper/2502.13519/) (2025)
- [ADAPT: Actively Discovering and Adapting to Preferences for any Task](https://mlsanity.com/paper/2504.04040/) (2025)
- [LuciBot: Automated Robot Policy Learning from Generated Videos](https://mlsanity.com/paper/2503.09871/) (2025)
- [Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language](https://mlsanity.com/paper/2511.14565/) (2025)
- [From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedback](https://mlsanity.com/paper/2502.07645/) (2025)
- [GoalLadder: Incremental Goal Discovery with Vision-Language Models](https://mlsanity.com/paper/2506.16396/) (2025)
- [RDA: Reward Design Agent for Reinforcement Learning](https://mlsanity.com/paper/2606.01672/) (2026)
- [Causally Robust Reward Learning from Reason-Augmented Preference Feedback](https://mlsanity.com/paper/2603.04861/) (2026)
- [Training-Free Action Correction for VLA Model Failures via Language Feedback](https://mlsanity.com/paper/2608.29967/) (2026)
- [Adaptive Querying for Reward Learning from Human Feedback](https://mlsanity.com/paper/2412.07990/) (2024)
- [Computing Voting Rules with Improvement Feedback](https://mlsanity.com/paper/2502.12542/) (2025)
- [QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agents](https://mlsanity.com/paper/2511.17855/) (2025)
- [Riemannian Dueling Optimization](https://mlsanity.com/paper/2603.00023/) (2026)
- [On the Strengths and Weaknesses of Data for Open-set Embodied Assistance](https://mlsanity.com/paper/2603.04819/) (2026)
- [Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG](https://mlsanity.com/paper/2603.09011/) (2026)
- [EgoPriMo: Egocentric Motion Generation for Interactive Humanoid Control](https://mlsanity.com/paper/2606.08495/) (2026)

## Abstract (arXiv)

Learning from human feedback has gained traction in fields like robotics and natural language processing in recent years. While prior works mostly rely on human feedback in the form of comparisons, language is a preferable modality that provides more informative insights into user preferences. In this work, we aim to incorporate comparative language feedback to iteratively improve robot trajectories and to learn reward functions that encode human preferences. To achieve this goal, we learn a shared latent space that integrates trajectory data and language feedback, and subsequently leverage the learned latent space to improve trajectories and learn human preferences. To the best of our knowledge, we are the first to incorporate comparative language feedback into reward learning. Our simulation experiments demonstrate the effectiveness of the learned latent space and the success of our learning algorithms. We also conduct human subject studies that show our reward learning algorithm achieves a 23.9% higher subjective score on average and is 11.3% more time-efficient compared to preference-based reward learning, underscoring the superior performance of our method. Our website is at https://liralab.usc.edu/comparative-language-feedback/