mlsanityData to 14 Oct 2026

AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model

arXiv 2309.16058 ↗First posted 27 Sep 2023cs.LG cs.CL cs.CVcited by 93

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i.e. text, image, video, audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the powerful text-based reasoning abilities of the state-of-the-art LLMs including LLaMA-2 (70B), and converts modality-specific signals to the joint textual space through a pre-trained aligner module. To further strengthen the multimodal LLM's capabilities, we fine-tune the model with a multimodal instruction set manually collected to cover diverse topics and tasks beyond simple QAs. We conduct comprehensive empirical analysis comprising both human and automatic evaluations, and demonstrate state-of-the-art performance on various multimodal tasks.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). Builds on 59 of its 67 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

59 of 67 references.

References · 59

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  13. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  14. 2019Towards VQA Models That Can Readcited by 1,718
  15. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  16. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  17. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  18. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  19. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  20. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  21. 2020Designing Network Design Spacescited by 960
  22. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  23. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  24. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  25. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  26. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  27. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  28. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  29. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  30. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  31. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  32. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  33. 2016Visual Dialogcited by 377
  34. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  35. 2019Clotho: An Audio Captioning Datasetcited by 358
  36. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  37. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  38. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 47; 2025: 23; 2026: 7.

93 papers.

Sort:

By area:

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  2. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  3. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  4. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  5. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  6. 2024Large Language Models for Time Series: A Surveycited by 114
  7. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  8. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 93
  9. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  10. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  11. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 76
  12. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
38 more
  1. 2024PaPaGei: Open Foundation Models for Optical Physiological Signalscited by 69
  2. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  3. 2025Scaling Laws for Native Multimodal Modelscited by 50
  4. 2024When Large Language Model Agents Meet 6G Networks: Perception, Grounding, and Alignmentcited by 43
  5. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  6. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  7. 2025HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Modelscited by 33
  8. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  9. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  10. 2024MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generationcited by 29
  11. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  12. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 28
  13. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  14. 2024MIO: A Foundation Model on Multimodal Tokenscited by 24
  15. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imagingcited by 22
  16. 2024Cross-Modal Projection in Multimodal LLMs Doesn't Really Project Visual Attributes to Textual Spacecited by 22
  17. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  18. 2023Boosting Large Language Model for Speech Synthesis: An Empirical Studycited by 21
  19. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  20. 2024The Evolution of Multimodal Model Architecturescited by 21
  21. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 19
  22. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  23. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  24. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  25. 2024LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Modelcited by 17
  26. 2024Altogether: Image Captioning via Re-aligning Alt-textcited by 17
  27. 2024GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewearcited by 16
  28. 2024AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capabilitycited by 15
  29. 2024ExpLLM: Towards Chain of Thought for Facial Expression Recognitioncited by 15
  30. 2023De-Diffusion Makes Text a Strong Cross-Modal Interfacecited by 14
  31. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  32. 2023Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generatorcited by 11
  33. 2023Data-Efficient Multimodal Fusion on a Single GPUcited by 11
  34. 2024Step Differences in Instructional Videocited by 11
  35. 2025SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactionscited by 11
  36. 2024AccidentGPT: Large Multi-Modal Foundation Model for Traffic Accident Analysiscited by 10
  37. 2024MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasetscited by 10
  38. 2024Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Themcited by 10

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
  2. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
  3. 2026Toward Native Multimodal Modeling: A Roadmap
  4. 2023On the Performance of Multimodal Language Models
  5. 2024OmniBench: Towards The Future of Universal Omni-Language Models
  6. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
  7. 2023NExT-GPT: Any-to-Any Multimodal LLM
  8. 2025FysicsWorld: A Unified Full-Modality Benchmark for Any-to-Any Understanding, Generation, and Reasoning