mlsanityData to 7 Oct 2026

Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation

arXiv 2408.16228 ↗First posted 29 Aug 2024cs.RO cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Learned language-conditioned robot policies often struggle to effectively adapt to new real-world tasks even when pre-trained across a diverse set of instructions. We propose a novel approach for few-shot adaptation to unseen tasks that exploits the semantic understanding of task decomposition provided by vision-language models (VLMs). Our method, Policy Adaptation via Language Optimization (PALO), combines a handful of demonstrations of a task with proposed language decompositions sampled from a VLM to quickly enable rapid nonparametric adaptation, avoiding the need for a larger fine-tuning dataset. We evaluate PALO on extensive real-world experiments consisting of challenging unseen, long-horizon robot manipulation tasks. We find that PALO is able of consistently complete long-horizon, multi-tier tasks in the real world, outperforming state of the art pre-trained generalist policies, and methods that have access to the same demonstrations.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 59 of its 101 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

59 of 101 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  12. 2016Matching Networks for One Shot Learningcited by 2,246
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  4. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  8. 2019A Closer Look at Few-shot Classificationcited by 679
  9. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  13. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  14. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  15. 2022Visual Language Maps for Robot Navigationcited by 362
  16. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  17. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  18. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  19. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  20. 2024RT-H: Action Hierarchies Using Languagecited by 204
  21. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  22. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  23. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  24. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  25. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  26. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  27. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  28. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  29. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  30. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  31. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  32. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  33. 2023Learning to Model the World with Languagecited by 66
  34. 2019Human few-shot learning of compositional instructionscited by 47
  35. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  36. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  37. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  38. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
  2. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 42
  3. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  4. 2025Horizon Generalization in Reinforcement Learningcited by 15
  5. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  6. 2024STEER: Flexible Robotic Manipulation via Dense Language Groundingcited by 13
  7. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  8. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  9. 2025KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulationcited by 5
  10. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steeringcited by 4
  11. 2025Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulationcited by 3
  12. 2025ComposableNav: Instruction-Following Navigation in Dynamic Environments via Composable Diffusioncited by 3
6 more
  1. 2025FoldNet: Learning Generalizable Closed-Loop Policy for Garment Folding via Keypoint-Driven Asset and Demonstration Synthesiscited by 1
  2. 2026See Less, Specify More: Visual Evidence Budgets for Generalizable VLAscited by 1
  3. 2026Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latencycited by 1
  4. 2025ManiLong-Shot: Interaction-Aware One-Shot Imitation Learning for Long-Horizon Manipulationcited by 0
  5. 2026Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulationcited by 0
  6. 2026Multi-Granularity Language-Guided Imitation Learning via Instruction Decompositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Robotic Policy Adaptation via Weight-Space Meta-Learning
  2. 2023Language-Conditioned Semantic Search-Based Policy for Robotic Manipulation Tasks
  3. 2026VLS: Steering Pretrained Robot Policies via Vision-Language Models
  4. 2023Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data
  5. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
  6. 2024Learning with Language-Guided State Abstractions
  7. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  8. 2026PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models