mlsanityData to 14 Oct 2026

NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?

arXiv 2411.17794 ↗First posted 26 Nov 2024cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have made notable advances in visual understanding, yet their abilities to recognize objects modified by specific attributes remain an open question. To address this, we explore MLLMs' reasoning capabilities in object recognition, ranging from commonsense to beyond-commonsense scenarios. We introduce a novel benchmark, NEMO, which comprises 900 images of origiNal fruits and their corresponding attributE-MOdified ones; along with a set of 2,700 questions including open-, multiple-choice-, unsolvable types. We assess 26 recent open-sourced and commercial models using our benchmark. The findings highlight pronounced performance gaps in recognizing objects in NEMO and reveal distinct answer preferences across different models. Although stronger vision encoders improve performance, MLLMs still lag behind standalone vision encoders. Interestingly, scaling up the model size does not consistently yield better outcomes, as deeper analysis reveals that larger LLMs can weaken vision encoders during fine-tuning. These insights shed light on critical limitations in current MLLMs and suggest potential pathways toward developing more versatile and resilient multimodal models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 47 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

36 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  5. 2023Link-Context Learning for Multimodal LLMs15 Aug
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
16 more
  1. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  2. 2024Yi: Open Foundation Models by 01.AIcited by 719
  3. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  4. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  5. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  6. 2024InternLM2 Technical Reportcited by 397
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 108
  10. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  11. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
  12. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  13. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 32
  14. 2023ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sensecited by 27
  15. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  16. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environmentscited by 3
  2. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
  2. 2025Benchmarking Large and Small MLLMs
  3. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  4. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models
  5. 2025On Large Multimodal Models as Open-World Image Classifiers
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  7. 2024Rethinking VLMs and LLMs for Image Classification
  8. 2024NVLM: Open Frontier-Class Multimodal LLMs