mlsanityData to 7 Oct 2026

Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation

arXiv 2307.06333 ↗First posted 12 Jul 2023cs.LG cs.AI cs.HCcited by 16

Not on the map. Why?

Abstract (arXiv)

Policies often fail due to distribution shift -- changes in the state and reward that occur when a policy is deployed in new environments. Data augmentation can increase robustness by making the model invariant to task-irrelevant changes in the agent's observation. However, designers don't know which concepts are irrelevant a priori, especially when different end users have different preferences about how the task is performed. We propose an interactive framework to leverage feedback directly from the user to identify personalized task-irrelevant concepts. Our key idea is to generate counterfactual demonstrations that allow users to quickly identify possible task-relevant and irrelevant concepts. The knowledge of task-irrelevant concepts is then used to perform data augmentation and thus obtain a policy adapted to personalized user objectives. We present experiments validating our framework on discrete and continuous control tasks with real human users. Our method (1) enables users to better understand agent failure, (2) reduces the number of demonstrations required for fine-tuning, and (3) aligns the agent to individual user task preferences.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

21 of 34 references.

References · 21

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  7. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  9. 2019Counterfactual Visual Explanationscited by 312
  10. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  11. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  12. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
9 more
  1. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  2. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  3. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 69
  4. 2019Leveraging Human Guidance for Deep Reinforcement Learning Taskscited by 42
  5. 2017Agent-Agnostic Human-in-the-Loop Reinforcement Learningcited by 23
  6. 2018Learning under Misspecified Objective Spacescited by 14
  7. 2019Counterfactual States for Atari Agents via Generative Deep Learningcited by 14
  8. 2021Learning Perceptual Concepts by Bootstrapping from Human Queriescited by 12
  9. 2019Safe Augmentation: Learning Task-Specific Transformations from Datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 3; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2024A Roadmap to Pluralistic Alignmentcited by 175
  3. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  4. 2024IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learningcited by 24
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  6. 2023Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedbackcited by 12
  7. 2024Preference-Conditioned Language-Guided Abstractioncited by 11
  8. 2024Learning with Language-Guided State Abstractionscited by 11
  9. 2024Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Inputcited by 8
  10. 2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
  11. 2024Robot Behavior Personalization from Sparse User Feedbackcited by 2
  12. 2026Early Warning Signals for OpenVLA Failure under Visual Distribution Shiftcited by 2
4 more
  1. 2025Cyber Shadows: Neutralizing Security Threats with AI and Targeted Policy Measurescited by 1
  2. 2025Task load dependent decision referrals for joint binary classification in human-automation teamscited by 0
  3. 2026Robots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanationscited by 0
  4. 2026Data-Efficient Adaptation of LLMs via Attention Head Reweightingcited by 0