mlsanityData to 7 Oct 2026

Lenna: Language Enhanced Reasoning Detection Assistant

arXiv 2312.02433 ↗First posted 5 Dec 2023cs.CVcited by 29

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

With the fast-paced development of multimodal large language models (MLLMs), we can now converse with AI systems in natural languages to understand images. However, the reasoning power and world knowledge embedded in the large language models have been much less investigated and exploited for image perception tasks. In this paper, we propose Lenna, a language-enhanced reasoning detection assistant, which utilizes the robust multimodal feature representation of MLLMs, while preserving location information for detection. This is achieved by incorporating an additional <DET> token in the MLLM vocabulary that is free of explicit semantic context but serves as a prompt for the detector to identify the corresponding position. To evaluate the reasoning capability of Lenna, we construct a ReasonDet dataset to measure its performance on reasoning-based detection. Remarkably, Lenna demonstrates outstanding performance on ReasonDet and comes with significantly low training costs. It also incurs minimal transferring overhead when extended to other tasks. Our code and model will be available at https://git.io/Lenna.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 58 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

50 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  2. 2023DetGPT: Detect What You Need via Reasoning23 May

Other references · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
36 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  8. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  14. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  15. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  16. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  17. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  18. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  19. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  20. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  21. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  22. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  23. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  24. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  25. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  26. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  27. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  28. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  29. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  30. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  31. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  32. 2022Towards Grand Unification of Object Trackingcited by 93
  33. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  34. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Modelscited by 30
  35. 2022DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Groundingcited by 24
  36. 2023InstructDET: Diversifying Referring Object Detection with Generalized Instructionscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 15; 2025: 7; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  2. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  3. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  4. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  5. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  6. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  7. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  8. 2024Toward Visual Grounding: A Surveycited by 52
  9. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  11. 2024VisionLLaMA: A Unified LLaMA Backbone for Vision Taskscited by 28
  12. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
17 more
  1. 2024Visual Text Generation in the Wildcited by 22
  2. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  3. 2024From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Informationcited by 16
  4. 2024Popeye: A Unified Visual-Language Model for Multi-Source Ship Detection from Remote Sensing Imagerycited by 14
  5. 2025STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Groundingcited by 11
  6. 2024MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Imagecited by 9
  7. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasoundcited by 8
  8. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaborationcited by 7
  9. 2025FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Modelscited by 6
  10. 2024CapeLLM: Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Modelscited by 2
  11. 2026Urban Socio-Semantic Segmentation with Vision-Language Reasoningcited by 2
  12. 2025SceneVTG++: Controllable Multilingual Visual Text Generation in the Wildcited by 1
  13. 2025Language-guided Learning for Object Detection Tackling Multiple Variations in Aerial Imagescited by 1
  14. 2026FigEx2: Visual-Conditioned Panel Detection and Captioning for Scientific Compound Figurescited by 0
  15. 2026Breaking the accuracy-resource dilemma: a lightweight adaptive video inference enhancementcited by 0
  16. 2026MedVol-R1: Reward-Driven Evidence Grounding for Volumetric Reasoning Segmentationcited by 0
  17. 2026FAS-R1: A Unified Multi-Task MLLM for Reasoning Face Anti-Spoofingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  2. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
  3. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
  4. 2026Can Linguistic Reasoning Vectors Enhance Multimodal Reasoning Ability?
  5. 2022Deep Neural Networks for Visual Reasoning
  6. 2026Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
  7. 2026Vision-aligned Latent Reasoning for Multi-modal Large Language Model
  8. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning