mlsanityData to 14 Oct 2026

What Matters to You? Towards Visual Representation Alignment for Robot Learning

arXiv 2310.07932 ↗First posted 11 Oct 2023cs.RO cs.AI cs.CVcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

When operating in service of people, robots need to optimize rewards aligned with end-user preferences. Since robots will rely on raw perceptual inputs like RGB images, their rewards will inevitably use visual representations. Recently there has been excitement in using representations from pre-trained visual models, but key to making these work in robotics is fine-tuning, which is typically done via proxy tasks like dynamics prediction or enforcing temporal cycle-consistency. However, all these proxy tasks bypass the human's input on what matters to them, exacerbating spurious correlations and ultimately leading to robot behaviors that are misaligned with user preferences. In this work, we propose that robots should leverage human feedback to align their visual representations with the end-user and disentangle what matters for the task. We propose Representation-Aligned Preference-based Learning (RAPL), a method for solving the visual representation alignment problem and visual reward learning problem through the lens of preference-based learning and optimal transport. Across experiments in X-MAGICAL and in robotic manipulation, we find that RAPL's reward consistently generates preferred robot behaviors with high sample efficiency, and shows strong zero-shot generalization when the visual representation is learned from a different embodiment than the robot's.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 58 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

40 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  2. 2022Time-Efficient Reward Learning via Visually Assisted Cluster Ranking30 Nov
  3. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  4. 2021Preference-based Learning of Reward Function Features3 Mar
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  9. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  10. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  11. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  12. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
23 more
  1. 2022Masked Visual Pre-training for Motor Controlcited by 248
  2. 2020SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentationcited by 170
  3. 2023Language-Driven Representation Learning for Roboticscited by 165
  4. 2020Primal Wasserstein Imitation Learningcited by 121
  5. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  6. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  7. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  8. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  9. 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
  10. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 75
  11. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  12. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 67
  13. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  14. 2019Wasserstein Adversarial Imitation Learningcited by 60
  15. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 46
  16. 2023Optimal Transport for Offline Imitation Learningcited by 42
  17. 2021Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Modelscited by 41
  18. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 35
  19. 2020Imitation Learning with Sinkhorn Distancescited by 30
  20. 2020Feature Expansive Reward Learning: Rethinking Human Inputcited by 25
  21. 2023Alignment with human representations supports robust few-shot learningcited by 25
  22. 2023SIRL: Similarity-based Implicit Representation Learningcited by 14
  23. 2020Motion2Vec: Semi-Supervised Representation Learning from Surgical Videoscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 41
  2. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  3. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  4. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  5. 2025FDPP: Fine-tune Diffusion Policy with Human Preferencecited by 15
  6. 2024One-Shot Imitation under Mismatched Executioncited by 12
  7. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  8. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  9. 2025AnySafe: Adapting Latent Safety Filters at Runtime via Safety Constraint Parameterization in the Latent Spacecited by 7
  10. 2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
  11. 2024MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Interventioncited by 1
  12. 2026Enhancing Goal Inference via Correction Timingcited by 1
2 more
  1. 2024Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrationscited by 0
  2. 2026Toward a Theory of Value in AI Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment
  2. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences
  3. 2026Human Preference Modeling Using Visual Motion Prediction Improves Robot Skill Learning from Egocentric Human Video ↗
  4. 2024SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine
  5. 2026Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations
  6. 2026RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation
  7. 2024GRAPE: Generalizing Robot Policy via Preference Alignment
  8. 2026RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models