mlsanityData to 7 Oct 2026

Confusion-Aware In-Context-Learning for Vision-Language Models in Robotic Manipulation

arXiv 2603.15134 ↗First posted 16 Mar 2026cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) have significantly improved the generalization capabilities of robotic manipulation. However, VLM-based systems often suffer from a lack of robustness, leading to unpredictable errors, particularly in scenarios involving confusable objects. Our preliminary analysis reveals that these failures are mainly caused by shortcut learning problem inherently in VLMs, limiting their ability to accurately distinguish between confusable features. To this end, we propose Confusion-Aware In-Context Learning (CAICL), a method that enhances VLM performance in confusable scenarios for robotic manipulation. The approach begins with confusion localization and analysis, identifying potential sources of confusion. This information is then used as a prompt for the VLM to focus on features most likely to cause misidentification. Extensive experiments on the VIMA-Bench show that CAICL effectively addresses the shortcut learning issue, achieving a 85.5% success rate and showing good stability across tasks with different degrees of generalization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 50 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

41 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
28 more
  1. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  3. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2022A Generalist Agentcited by 802
  9. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  10. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  13. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  14. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  15. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  16. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  17. 2021Medical Visual Question Answering: A Surveycited by 93
  18. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  19. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  20. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  21. 2023Mitigating Biases for Instruction-following Language Models via Bias Neurons Eliminationcited by 23
  22. 2023Impact of Co-occurrence on Factual Knowledge of Large Language Modelscited by 19
  23. 2023Integration and Performance Analysis of Artificial Intelligence and Computer Vision Based on Deep Learning Algorithmscited by 19
  24. 2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modelingcited by 18
  25. 2024Synthetic Oversampling: Theory and A Practical Approach Using LLMs to Address Data Imbalancecited by 16
  26. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learningcited by 14
  27. 2023Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggerscited by 10
  28. 2024Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Modelscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026M2-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills
  2. 2025Improving Generalization of Language-Conditioned Robot Manipulation
  3. 2026AC^2-VLA: Action-Context-Aware Adaptive Computation in Vision-Language-Action Models for Efficient Robotic Manipulation
  4. 2025Perceiving, Reasoning, Adapting: A Dual-Layer Framework for VLM-Guided Precision Robotic Manipulation
  5. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  7. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models
  8. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulation