mlsanityData to 7 Oct 2026

Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models

arXiv 2403.17359 ↗First posted 26 Mar 2024cs.CLcited by 27

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

We present a Chain-of-Action (CoA) framework for multimodal and retrieval-augmented Question-Answering (QA). Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information. Our key contribution is a novel reasoning-retrieval mechanism that decomposes a complex question into a reasoning chain via systematic prompting and pre-designed actions. Methodologically, we propose three types of domain-adaptable `Plug-and-Play' actions for retrieving real-time information from heterogeneous sources. We also propose a multi-reference faith score (MRFS) to verify and resolve conflicts in the answers. Empirically, we exploit both public benchmarks and a Web3 case study to demonstrate the capability of CoA over other methods.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 40 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

27 of 40 references.

References · 27

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
15 more
  1. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  2. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  3. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  4. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  5. 2023Tool Learning with Foundation Modelscited by 302
  6. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  7. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 147
  8. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  9. 2020Knowledge-Aware Language Model Pretrainingcited by 52
  10. 2023Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environmentscited by 52
  11. 2023Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Modelscited by 31
  12. 2024Distilling Semantic Priors from SAM to Efficient Image Restoration Modelscited by 25
  13. 2024Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detectioncited by 22
  14. 2024Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contextscited by 19
  15. 2023When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 5.

27 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  5. 2025MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaversecited by 50
  6. 2024Improving Retrieval Augmented Language Model with Self-Reasoningcited by 28
  7. 2024RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Modelscited by 27
  8. 2024Enhance Image-to-Image Generation with LLaVA-generated Promptscited by 20
  9. 2024Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Actioncited by 19
  10. 2024Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?cited by 17
  11. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 11
  12. 2024Decoupled Alignment for Robust Plug-and-Play Adaptationcited by 9
15 more
  1. 2024Advanced AI Framework for Enhanced Detection and Assessment of Abdominal Trauma: Integrating 3D Segmentation with 2D CNN and RNN Modelscited by 9
  2. 2025Do Code LLMs Understand Design Patterns?cited by 9
  3. 2025SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheetscited by 6
  4. 2024Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Modelscited by 5
  5. 2026FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoningcited by 5
  6. 2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safetycited by 4
  7. 2025AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learningcited by 3
  8. 2025FinRobot: Generative Business Process AI Agents for Enterprise Resource Planning in Financecited by 2
  9. 2025FairReason: Balancing Reasoning and Social Bias in MLLMscited by 2
  10. 2025Masked Generative Policy for Robotic Controlcited by 2
  11. 2026U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generationcited by 2
  12. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  13. 2026Attention Sinks and Outliers in Attention Residualscited by 2
  14. 2025MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generationcited by 1
  15. 2025PDF Retrieval Augmented Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
  2. 2026Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation
  3. 2025Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective
  4. 2024MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering
  5. 2025Chain of Questions: Guiding Multimodal Curiosity in Language Models
  6. 2026LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes
  7. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thought
  8. 2024Structured Chain-of-Thought Prompting for Few-Shot Generation of Content-Grounded QA Conversations