mlsanityData to 7 Oct 2026

OmniFusion Technical Report

arXiv 2404.06212 ↗First posted 9 Apr 2024cs.CV cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an textitOmniFusion model based on a pretrained LLM and adapters for visual modality. We evaluated and compared several architecture design principles for better text and visual data coupling: MLP and transformer adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their fusing approach, image encoding method (whole image or tiles encoding) and two 7B LLMs (the proprietary one and open-source Mistral). Experiments on 8 visual-language benchmarks show the top score for the best OmniFusion setup in terms of different VQA tasks in comparison with open-source LLaVA-like solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We also propose a variety of situations, where OmniFusion provides highly-detailed answers in different domains: housekeeping, sightseeing, culture, medicine, handwritten and scanned equations recognition, etc. Mistral-based OmniFusion model is an open-source solution with weights, training and inference scripts available at https://github.com/AIRI-Institute/OmniFusion.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

34 of 43 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
22 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  8. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  9. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  10. 2021OCR-free Document Understanding Transformercited by 400
  11. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  12. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  13. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  14. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  15. 2023Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusioncited by 114
  16. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  17. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 96
  18. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  19. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  20. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  21. 2023Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objectscited by 17
  22. 2023InfMLLM: A Unified Framework for Visual-Language Taskscited by 11

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

  1. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  2. 2025MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understandingcited by 16
  3. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  4. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  5. 2025Visually Interpretable Subtask Reasoning for Visual Question Answeringcited by 4
  6. 2025Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulationcited by 4