mlsanityData to 7 Oct 2026

FSMR: A Feature Swapping Multi-modal Reasoning Approach with Joint Textual and Visual Clues

arXiv 2403.20026 ↗First posted 29 Mar 2024cs.CV cs.CLcited by 0

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Multi-modal reasoning plays a vital role in bridging the gap between textual and visual information, enabling a deeper understanding of the context. This paper presents the Feature Swapping Multi-modal Reasoning (FSMR) model, designed to enhance multi-modal reasoning through feature swapping. FSMR leverages a pre-trained visual-language model as an encoder, accommodating both text and image inputs for effective feature representation from both modalities. It introduces a unique feature swapping module, enabling the exchange of features between identified objects in images and corresponding vocabulary words in text, thereby enhancing the model's comprehension of the interplay between images and text. To further bolster its multi-modal alignment capabilities, FSMR incorporates a multi-modal cross-attention mechanism, facilitating the joint modeling of textual and visual information. During training, we employ image-text matching and cross-entropy losses to ensure semantic consistency between visual and language elements. Extensive experiments on the PMR dataset demonstrate FSMR's superiority over state-of-the-art baseline models across various performance metrics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 19 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

16 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues8 May

Other references · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2017Mask R-CNNcited by 7,259
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  10. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  11. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35
  12. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learningcited by 13
3 more
  1. 2020Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learningcited by 8
  2. 2021Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Cluescited by 8
  3. 2022Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanationscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
  2. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning
  3. 2025From Easy to Hard: The MIR Benchmark for Progressive Interleaved Multi-Image Reasoning
  4. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
  5. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  6. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation
  7. 2025Context-Aware Multi-Turn Visual-Textual Reasoning in LVLMs via Dynamic Memory and Adaptive Visual Guidance
  8. 2025Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation