mlsanityData to 14 Oct 2026

CF-VLM:CounterFactual Vision-Language Fine-tuning

arXiv 2506.17267 ↗First posted 10 Jun 2025cs.LG cs.AIcited by 59

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Recent advances in vision-language models (VLMs) have greatly improved cross-modal semantic understanding, yet significant limitations remain in fine-grained discrimination and deep causal reasoning tasks. Existing VLMs often rely on superficial statistical correlations, lacking the ability to capture the underlying causal logic between visual and textual content. To address this, we propose CounterFactual Vision-Language Fine-tuning (CF-VLM), a novel framework that enhances the causal reasoning capabilities of VLMs through the targeted use of counterfactual samples. CF-VLM introduces three complementary training objectives: maintaining foundational cross-modal alignment, reinforcing the uniqueness and stability of factual scene representations against coherent counterfactuals, and sharpening the model's sensitivity to minimal but critical causal edits. Extensive experiments demonstrate that CF-VLM consistently outperforms strong baselines and state-of-the-art methods on compositional reasoning and generalization benchmarks. Furthermore, it shows promise in mitigating visual hallucinations, indicating improved factual consistency. Our CF-VLM provides a robust foundation for deploying VLMs in high-stakes, real-world scenarios requiring reliable reasoning and interpretability.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 67 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

52 of 67 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning29 May
  2. 2024CELLO: Causal Evaluation of Large Vision-Language Models27 Jun
  3. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs12 Jun

Other references · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
37 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  9. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  10. 2024What matters when building vision-language models?cited by 326
  11. 2019Counterfactual Visual Explanationscited by 313
  12. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  13. 2023Teaching CLIP to Count to Tencited by 198
  14. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  15. 2021A Survey of Visual Transformerscited by 117
  16. 2021Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identificationcited by 114
  17. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  18. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  19. 2020Learning What Makes a Difference from Counterfactual Examples and Gradient Supervisioncited by 66
  20. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  21. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  22. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  23. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  24. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 41
  25. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representationscited by 37
  26. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 37
  27. 2024Causal Inference with Large Language Model: A Surveycited by 32
  28. 2022DISCO: Distilling Counterfactuals with Large Language Modelscited by 30
  29. 2022Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Modelscited by 28
  30. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  31. 2021Counterfactual Samples Synthesizing and Training for Robust Visual Question Answeringcited by 22
  32. 2022CPL: Counterfactual Prompt Learning for Vision and Language Modelscited by 18
  33. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificitycited by 14
  34. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Modelscited by 8
  35. 2024Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Trainingcited by 8
  36. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentationcited by 6
  37. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 37; 2026: 22.

59 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 98
  2. 2025MAT-Agent: Adaptive Multi-Agent Training Optimizationcited by 35
  3. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 34
  4. 2025OSC: Cognitive Orchestration through Dynamic Knowledge Alignment in Multi-Agent LLM Collaborationcited by 30
  5. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAscited by 30
  6. 2026FastVMT: Eliminating Redundancy in Video Motion Transfercited by 23
  7. 2025Failure-Driven Workflow Refinementcited by 20
  8. 2025Hybrid Token Compression for Vision-Language Modelscited by 20
  9. 2025Learning Dynamics of VLM Finetuningcited by 17
  10. 2025Top-Down Semantic Refinement for Image Captioningcited by 16
  11. 2025RaCoT: Plug-and-Play Contrastive Example Generation Mechanism for Enhanced LLM Reasoning Reliabilitycited by 14
  12. 2025CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoorcited by 12
38 more
  1. 20253DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scalecited by 12
  2. 2026Group Editing: Edit Multiple Images in One Gocited by 11
  3. 2025STORM: Search-Guided Generative World Models for Robotic Manipulationcited by 7
  4. 2025GHOST: Hallucination-Inducing Image Generation for Multimodal LLMscited by 4
  5. 2025CoAgent: Collaborative Planning and Consistency Agent for Coherent Video Generationcited by 4
  6. 2025MultiMotion: Multi Subject Video Motion Transfer via Video Diffusion Transformercited by 3
  7. 2026SeqWalker: Sequential-Horizon Vision-and-Language Navigation with Hierarchical Planningcited by 3
  8. 2025GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extractioncited by 2
  9. 2025Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understandingcited by 2
  10. 2025Large Language Models as Discounted Bayesian Filterscited by 2
  11. 2025Adaptive-VoCo: Complexity-Aware Visual Token Compression for Vision-Language Modelscited by 2
  12. 2026ResWM: Residual-Action World Model for Visual RLcited by 2
  13. 2026CFPO: Counterfactual Policy Optimization for Multimodal Reasoningcited by 2
  14. 2025Backward-Friendly Optimization: Training Large Language Models with Approximate Gradients under Memory Constraintscited by 1
  15. 2025Agent-GSPO: Communication-Efficient Multi-Agent Systems via Group Sequence Policy Optimizationcited by 1
  16. 2025Reflective Confidence: Correcting Reasoning Flaws via Online Self-Correctioncited by 1
  17. 2025MoE-DiffuSeq: Enhancing Long-Document Diffusion Models with Sparse Attention and Mixture of Expertscited by 1
  18. 2026ForgetMark: Stealthy Fingerprint Embedding via Targeted Unlearning in Language Modelscited by 1
  19. 2026Process-of-Thought Reasoning for Videoscited by 1
  20. 2026InstanceAnimator: Multi-Instance Sketch Video Colorizationcited by 1
  21. 2026Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Modelscited by 1
  22. 2025Overconfident and Blind to Details: Fixing Prompt Insensitivity with Abductive Preference Learningcited by 0
  23. 2025DynaPose4D: High-Quality 4D Dynamic Content Generation via Pose Alignment Losscited by 0
  24. 2025Guardian: Decoupling Exploration from Safety in Reinforcement Learningcited by 0
  25. 2025Causal Invariance and Counterfactual Learning Driven Cooperative Game for Multi-Label Classificationcited by 0
  26. 2025GTMA: Dynamic Representation Optimization for OOD Vision-Language Modelscited by 0
  27. 2025PTTA: A Pure Text-to-Animation Framework for High-Quality Creationcited by 0
  28. 2025SirenPose: Dynamic Scene Reconstruction via Geometric Supervisioncited by 0
  29. 2025SA-DiffuSeq: Addressing Computational and Scalability Challenges in Long-Document Generation with Sparse Attentioncited by 0
  30. 2025RevFFN: Memory-Efficient Full-Parameter Fine-Tuning of Mixture-of-Experts LLMs with Reversible Blockscited by 0
  31. 2025A Three-Level Alignment Framework for Large-Scale 3D Retrieval and Controlled 4D Generationcited by 0
  32. 2025Learning Dynamic Scene Reconstruction with Sinusoidal Geometric Priorscited by 0
  33. 2025Towards Efficient Post-Training via Fourier-Driven Adapter Architecturescited by 0
  34. 2025Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domainscited by 0
  35. 2025Forgetting as a Feature: Cognitive Alignment of Large Language Modelscited by 0
  36. 2026LogicGaze: Benchmarking Causal Consistency in Visual Narratives via Counterfactual Verificationcited by 0
  37. 2026Bridging the Semantic Chasm: Synergistic Conceptual Anchoring for Generalized Few-Shot and Zero-Shot OOD Perceptioncited by 0
  38. 2026A Scalable Curiosity-Driven Game-Theoretic Framework for Long-Tail Multi-Label Learning in Data Miningcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  2. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
  3. 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models
  4. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
  5. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models
  6. 2025What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning
  7. 2026Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency
  8. 2023What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models