mlsanityData to 7 Oct 2026

STEER: Flexible Robotic Manipulation via Dense Language Grounding

arXiv 2411.03409 ↗First posted 5 Nov 2024cs.RO cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

The complexity of the real world demands robotic systems that can intelligently adapt to unseen situations. We present STEER, a robot learning framework that bridges high-level, commonsense reasoning with precise, flexible low-level control. Our approach translates complex situational awareness into actionable low-level behavior through training language-grounded policies with dense annotation. By structuring policy training around fundamental, modular manipulation skills expressed in natural language, STEER exposes an expressive interface for humans or Vision-Language Models (VLMs) to intelligently orchestrate the robot's behavior by reasoning about the task and context. Our experiments demonstrate the skills learned via STEER can be combined to synthesize novel behaviors to adapt to new situations or perform completely new tasks without additional data collection or training.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

46 of 57 references.

References · 46

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
34 more
  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  3. 2016The Option-Critic Architecturecited by 433
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  5. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  6. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  7. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  8. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  9. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  10. 2024RT-H: Action Hierarchies Using Languagecited by 204
  11. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  12. 2024Behavior Generation with Latent Actionscited by 159
  13. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  14. 2023Structured World Models from Human Videoscited by 144
  15. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  16. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  17. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  18. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  19. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  20. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  21. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  22. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  23. 2023Data Quality in Imitation Learningcited by 72
  24. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 71
  25. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  26. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  27. 2018TACO: Learning Task Decomposition via Temporal Alignment for Controlcited by 61
  28. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  29. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  30. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  31. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitationcited by 18
  32. 2020Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulationcited by 17
  33. 2017Simultaneous Policy Learning and Latent State Inference for Imitating Driver Behaviorcited by 16
  34. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Modelscited by 28
  2. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 21
  3. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  4. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  5. 2025A Taxonomy for Evaluating Generalist Robot Manipulation Policiescited by 15
  6. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policiescited by 11
  7. 2025Learning Affordances at Inference-Time for Vision-Language-Action Modelscited by 6
  8. 2026FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policiescited by 5
  9. 2026InSight: Self-Guided Skill Acquisition via Steerable VLAscited by 2
  10. 2026See Less, Specify More: Visual Evidence Budgets for Generalizable VLAscited by 1
  11. 2026Visual Verification Enables Inference-time Steering and Autonomous Policy Improvementcited by 1
  12. 2026Multi-Granularity Language-Guided Imitation Learning via Instruction Decompositioncited by 0
1 more
  1. 2026YUBI-STAG: Contact and Semantic-Rich Alignment for VLAs via Automated Video-Language Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
  2. 2025LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation
  3. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation
  4. 2026R3: Training Robots to Reason in Natural Language via Reinforcement Learning
  5. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
  6. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulation
  7. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  8. 2026Language Movement Primitives: Grounding Language Models in Robot Motion