mlsanityData to 7 Oct 2026

CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning

arXiv 2402.04236 ↗First posted 6 Feb 2024cs.CV cs.CLcited by 64

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have demonstrated their broad effectiveness thanks to extensive training in aligning visual instructions to responses. However, such training of conclusive alignment leads models to ignore essential visual reasoning, further resulting in failures in meticulous visual problems and unfaithful responses. Drawing inspiration from human cognition in solving visual problems (e.g., marking, zoom in), this paper introduces Chain of Manipulations, a mechanism that enables VLMs to solve problems step-by-step with evidence. After training, models can solve various visual problems by eliciting intrinsic manipulations (e.g., grounding, zoom in) with results (e.g., boxes, image) actively without involving external tools, while also allowing users to trace error causes. We study the roadmap to implement this mechanism, including (1) a flexible design of manipulations upon extensive analysis, (2) an efficient automated data generation pipeline, (3) a compatible VLM architecture capable of multi-turn multi-image, and (4) a model training process for versatile capabilities. With the design, we also manually annotate 6K high-quality samples for the challenging graphical mathematical problems. Our trained model, CogCoM, equipped with this mechanism with 17B parameters achieves state-of-the-art performance across 9 benchmarks from 4 categories, demonstrating the effectiveness while preserving the interpretability. Our code, model weights, and collected data are publicly available at https://github.com/THUDM/CogCoM.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 58 of its 71 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

58 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models5 Dec
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2023Look, Remember and Reason: Grounded reasoning in videos with language models30 Jun
  4. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  5. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 52

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2015VQA: Visual Question Answeringcited by 2,489
38 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  12. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  13. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  16. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  17. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  18. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  19. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  20. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  21. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  22. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  23. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  24. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  25. 2019Scene Text Visual Question Answeringcited by 327
  26. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  27. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  28. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  29. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  30. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  31. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  32. 2018TallyQA: Answering Complex Counting Questionscited by 152
  33. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  34. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  35. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  36. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  37. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  38. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Datacited by 33

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 31; 2026: 20.

64 papers.

Sort:

By area:

  1. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  2. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  3. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  4. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  5. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  6. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  7. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  8. 2025VGR: Visual Grounded Reasoningcited by 70
  9. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  10. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instructcited by 42
  11. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  12. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
38 more
  1. 2024MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classificationcited by 35
  2. 2025TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoningcited by 34
  3. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  4. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  5. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  6. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  7. 2025Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Modelscited by 24
  8. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
  9. 2024The Evolution of Multimodal Model Architecturescited by 21
  10. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  11. 2024From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesiscited by 17
  12. 2025Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarioscited by 16
  13. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 16
  14. 2025Thinking in 360deg: Humanoid Visual Search in the Wildcited by 14
  15. 2024LATTE: Learning to Think with Vision Specialistscited by 13
  16. 2025DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoningcited by 10
  17. 2024VisAidMath: Benchmarking Visual-Aided Mathematical Reasoningcited by 9
  18. 2025NAVIG: Natural Language-guided Analysis with Vision Language Models for Image Geo-localizationcited by 8
  19. 2025Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Groundingcited by 7
  20. 2025CoFFT: Chain of Foresight-Focus Thought for Visual Language Modelscited by 6
  21. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 6
  22. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policycited by 6
  23. 2024Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Modelscited by 5
  24. 2025CoT Referring: Improving Referring Expression Tasks with Grounded Reasoningcited by 5
  25. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scalescited by 5
  26. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoningcited by 5
  27. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  28. 2025Thinking with Images via Self-Calling Agentcited by 4
  29. 2026AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoningcited by 4
  30. 2026Leveraging Latent Visual Reasoning in Silencecited by 4
  31. 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Modelscited by 3
  32. 2025Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learningcited by 3
  33. 2026CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Frameworkcited by 3
  34. 2026Semantic-Enriched Latent Visual Reasoningcited by 3
  35. 2024Improving Visual Storytelling with Multimodal Large Language Modelscited by 2
  36. 2026Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoningcited by 2
  37. 2025Dynamic Knowledge Integration for Enhanced Vision-Language Reasoningcited by 1
  38. 2026VIRAASAT: Traversing Novel Paths for Indian Cultural Reasoningcited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  2. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
  3. 2025DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning
  4. 2025Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains
  5. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
  6. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
  7. 2023Chain of Thought Prompt Tuning in Vision Language Models
  8. 2026CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions