mlsanityData to 14 Oct 2026

The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors

arXiv 2610.16977 ↗First posted 12 Oct 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual in-context learning (ICL) enables vision-language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depends critically on the examples selected. Recent counterfactual selection methods improve ICL by constructing demonstrations that expose how individual attribute changes affect the answer. However, they require multi-stage pipelines involving attribute extraction, caption engineering, and composed image matching. We propose a simpler alternative grounded in a key observation: a model's own prediction errors are natural counterfactual demonstrations that directly encode its failure modes. Our method, SMILE (Self-Mined Visual In-Context Learning from Errors), builds a model-specific pool of hard negatives offline by recording the target VLM's own mistakes. At test time, the prediction serves as a direction signal that surfaces, in the semantic embedding space, the past mistakes most likely to recur for the current query. This prediction-conditioned identification provides a unified mechanism for both closed-set classification and open-ended visual question answering without task-specific design. Across four benchmarks and four VLMs, SMILE delivers an average task-score gain of +3.09 points over the SOTA baseline at 5x lower latency. Project page: https://tree-shu-zhao.github.io/SMILE/

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 56 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

50 of 56 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Retrieving Counterfactuals Improves Visual In-Context Learning17 Mar
  2. 2025Sherlock: Self-Correcting Reasoning in Vision-Language Models28 May
  3. 2024Vision-Language Models Can Self-Improve Reasoning via Reflection30 Oct
  4. 2024Visual In-Context Learning for Large Vision-Language Models18 Feb
  5. 2023MAFA: Managing False Negatives for Vision-Language Pre-training11 Dec
  6. 2023How to Configure Good In-Context Sequence for Visual Question Answering4 Dec
  7. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  7. 2025Qwen3-VL Technical Reportcited by 2,793
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
30 more
  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  2. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  3. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  4. 2020Contrastive Learning with Hard Negative Samplescited by 628
  5. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  6. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  7. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  8. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  9. 2023Towards In-context Scene Understandingcited by 47
  10. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  11. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 41
  12. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 34
  13. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  14. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 27
  15. 2025ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learningcited by 26
  16. 2024C-ICL: Contrastive In-context Learning for Information Extractioncited by 23
  17. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 22
  18. 2025MapFusion: A Novel BEV Feature Fusion Network for Multi-modal Map Constructioncited by 21
  19. 2025SafeMap: Robust HD Map Construction from Incomplete Observationscited by 14
  20. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?cited by 13
  21. 2024DRUM: Learning Demonstration Retriever for Large MUlti-modal Modelscited by 10
  22. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 7
  23. 2025Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learningcited by 7
  24. 2021Rescuing Deep Hashing from Dead Bits Problemcited by 4
  25. 2023Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Datacited by 4
  26. 2024Is In-Context Learning a Type of Error-Driven Learning? Evidence from the Inverse Frequency Effect in Structural Primingcited by 4
  27. 2025MoRA: Missing Modality Low-Rank Adaptation for Visual Recognitioncited by 4
  28. 2024In-context Learning in Presence of Spurious Correlationscited by 3
  29. 2024KALAHash: Knowledge-Anchored Low-Resource Adaptation for Deep Hashingcited by 2
  30. 2022Mathematical Justification of Hard Negative Mining via Isometric Approximation Theoremcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Failing to See or Failing to Know? Attributing Errors in Vision-Language Models
  2. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priors
  3. 2026Seeing and Solving Are Not Enough for Vision-Language Models
  4. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?
  5. 2026Discovering Failure Modes in Vision-Language Models using RL
  6. 2026MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models
  7. 2025Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations
  8. 2026Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models