# SIRL: Similarity-based Implicit Representation Learning

arXiv: https://arxiv.org/abs/2301.00810 · HTML: https://mlsanity.com/paper/2301.00810/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 60 references that are in the map.
- First posted on arXiv 2 Jan 2023.

## Builds on (most cited first, 27 of 27)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos](https://mlsanity.com/paper/2103.16817/) (2021)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Playing hard exploration games by watching YouTube](https://mlsanity.com/paper/1805.11592/) (2018)
- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- [Self-Supervised Pretraining Improves Self-Supervised Pretraining](https://mlsanity.com/paper/2103.12718/) (2021)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Meta-Inverse Reinforcement Learning with Probabilistic Context Variables](https://mlsanity.com/paper/1909.09314/) (2019)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [Feature Expansive Reward Learning: Rethinking Human Input](https://mlsanity.com/paper/2006.13208/) (2020)
- [Multi-task Maximum Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.08882/) (2018)
- [ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapes](https://mlsanity.com/paper/2011.04812/) (2020)
- [On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning](https://mlsanity.com/paper/2206.03271/) (2022)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [Contrastive Predictive Coding Based Feature for Automatic Speaker Verification](https://mlsanity.com/paper/1904.01575/) (2019)
- [Learning Perceptual Concepts by Bootstrapping from Human Queries](https://mlsanity.com/paper/2111.05251/) (2021)
- [Teaching Robots to Span the Space of Functional Expressive Motion](https://mlsanity.com/paper/2203.02091/) (2022)
- [Meta Preference Learning for Fast User Adaptation in Human-Supervisory Multi-Robot Deployments](https://mlsanity.com/paper/2103.08008/) (2021)

## Built on by (most cited first, 14 of 14)

- [What Matters to You? Towards Visual Representation Alignment for Robot Learning](https://mlsanity.com/paper/2310.07932/) (2023)
- [Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following](https://mlsanity.com/paper/2502.05454/) (2025)
- [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023)
- [Learning with Language-Guided State Abstractions](https://mlsanity.com/paper/2402.18759/) (2024)
- [Human-oriented Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2310.03023/) (2023)
- [Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards](https://mlsanity.com/paper/2402.17975/) (2024)
- [Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing](https://mlsanity.com/paper/2410.10062/) (2024)
- [Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language](https://mlsanity.com/paper/2511.14565/) (2025)
- [Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitation](https://mlsanity.com/paper/2501.01367/) (2025)
- [Sample-efficient Adversarial Imitation Learning](https://mlsanity.com/paper/2303.07846/) (2023)
- [Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?](https://mlsanity.com/paper/2306.13004/) (2023)
- [Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations](https://mlsanity.com/paper/2408.05610/) (2024)
- [Robots that Suggest Safe Alternatives](https://mlsanity.com/paper/2409.09883/) (2024)
- [Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG](https://mlsanity.com/paper/2603.09011/) (2026)

## Abstract (arXiv)

When robots learn reward functions using high capacity models that take raw state directly as input, they need to both learn a representation for what matters in the task -- the task ``features" -- as well as how to combine these features into a single objective. If they try to do both at once from input designed to teach the full reward function, it is easy to end up with a representation that contains spurious correlations in the data, which fails to generalize to new settings. Instead, our ultimate goal is to enable robots to identify and isolate the causal features that people actually care about and use when they represent states and behavior. Our idea is that we can tune into this representation by asking users what behaviors they consider similar: behaviors will be similar if the features that matter are similar, even if low-level behavior is different; conversely, behaviors will be different if even one of the features that matter differs. This, in turn, is what enables the robot to disambiguate between what needs to go into the representation versus what is spurious, as well as what aspects of behavior can be compressed together versus not. The notion of learning representations based on similarity has a nice parallel in contrastive learning, a self-supervised representation learning technique that maps visually similar data points to similar embeddings, where similarity is defined by a designer through data augmentation heuristics. By contrast, in order to learn the representations that people use, so we can learn their preferences and objectives, we use their definition of similarity. In simulation as well as in a user study, we show that learning through such similarity queries leads to representations that, while far from perfect, are indeed more generalizable than self-supervised and task-input alternatives.