mlsanityData to 7 Oct 2026

Pixel Aligned Language Models

arXiv 2312.09237 ↗First posted 14 Dec 2023cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large language models have achieved great success in recent years, so as their variants in vision. Existing vision-language models can describe images in natural languages, answer visual-related questions, or perform complex reasoning about the image. However, it is yet unclear how localization tasks, such as word grounding or referring localization, can be performed using large language models. In this work, we aim to develop a vision-language model that can take locations, for example, a set of points or boxes, as either inputs or outputs. When taking locations as inputs, the model performs location-conditioned captioning, which generates captions for the indicated object or region. When generating locations as outputs, our model regresses pixel coordinates for each output word generated by the language model, and thus performs dense word grounding. Our model is pre-trained on the Localized Narrative dataset, which contains pixel-word-aligned captioning from human attention. We show our model can be applied to various location-aware vision-language tasks, including referring localization, location-conditioned captioning, and dense object captioning, archiving state-of-the-art performance on RefCOCO and Visual Genome. Project page: https://jerryxu.net/PixelLLM .

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 59 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

52 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  5. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
35 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2019Objects as Pointscited by 1,172
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  14. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  16. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  17. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  18. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  19. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  20. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  21. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  22. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  23. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  24. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  25. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  26. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  27. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  28. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  29. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  30. 2019Connecting Vision and Language with Localized Narrativescited by 225
  31. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  32. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  33. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  34. 2019Context and Attribute Grounded Dense Captioningcited by 31
  35. 2021Connecting What to Say With Where to Look by Modeling Human Attention Tracescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 2; 2026: 1.

17 papers.

Sort:

By area:

  1. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  2. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  3. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  5. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  6. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  7. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  8. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  9. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  10. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  11. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  12. 2024How Well Can Vision Language Models See Image Details?cited by 10
5 more
  1. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervisioncited by 8
  2. 2024SRL-CLIP: Efficient CLIP Video Adaptation via Structured Semantic Role Labelscited by 4
  3. 2024TraDiffusion: Trajectory-Based Training-Free Image Generationcited by 2
  4. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 1
  5. 2025QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  2. 2025Talking Points: Describing and Localizing Pixels
  3. 2026Vision-Language Grounding as Bidirectional Concept Correspondence
  4. 2025Improved Alignment of Modalities in Large Vision Language Models
  5. 2024The ART of Composition: Attention-Regularized Training for Compositional Visual Grounding
  6. 2024RegionGPT: Towards Region Understanding Vision Language Model
  7. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformers
  8. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs