mlsanityData to 14 Oct 2026

LocCa: Visual Pretraining with Location-aware Captioners

arXiv 2403.19596 ↗First posted 28 Mar 2024cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Image captioning has been shown as an effective pretraining method similar to contrastive pretraining. However, the incorporation of location-aware information into visual pretraining remains an area with limited research. In this paper, we propose a simple visual pretraining method with location-aware captioners (LocCa). LocCa uses a simple image captioner task interface, to teach a model to read out rich information, i.e. bounding box coordinates, and captions, conditioned on the image pixel input. Thanks to the multitask capabilities of an encoder-decoder architecture, we show that an image captioner can easily handle multiple tasks during pretraining. Our experiments demonstrate that LocCa outperforms standard captioners significantly on localization downstream tasks while maintaining comparable performance on holistic tasks.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 95 of its 111 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

95 of 111 references.

References · 95

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2017Mask R-CNNcited by 7,277
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
38 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2015Fast R-CNNcited by 2,933
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  13. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  14. 2019Towards VQA Models That Can Readcited by 1,718
  15. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  16. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  17. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  18. 2021Grounded Language-Image Pre-trainingcited by 1,156
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  20. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  21. 2016Modeling Context in Referring Expressionscited by 1,063
  22. 2021Scaling Vision Transformerscited by 999
  23. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  24. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  25. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  26. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  27. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  28. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  29. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  30. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  31. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  32. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  33. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  34. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  35. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  36. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  37. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  38. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 15.

35 papers.

Sort:

By area:

  1. 2025Scaling Language-Free Visual Representation Learningcited by 66
  2. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  3. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 22
  4. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 19
  5. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  6. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Modelscited by 15
  7. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  8. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  9. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
  10. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Modelscited by 11
  11. 2026The Spatial Blindspot of Vision-Language Modelscited by 8
  12. 2025EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanationscited by 7
23 more
  1. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  2. 2025Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughputcited by 4
  3. 2025Scope: Selective Cross-modal Orchestration of Visual Perception Expertscited by 2
  4. 2026Revisiting Multi-Task Visual Representation Learningcited by 2
  5. 2025Task-Agnostic Attacks Against Vision Foundation Modelscited by 1
  6. 2025Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMscited by 1
  7. 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioningcited by 1
  8. 2025Holistic Order Prediction in Natural Scenescited by 1
  9. 2025Efficient Discriminative Joint Encoders for Large Scale Vision-Language Rerankingcited by 1
  10. 2025Masked Diffusion Captioning for Visual Feature Learningcited by 1
  11. 2026VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimizationcited by 1
  12. 2026Location-Aware Pretraining for Medical Difference Visual Question Answeringcited by 1
  13. 2026PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understandingcited by 1
  14. 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perceptioncited by 1
  15. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  16. 2025Using Knowledge Graphs to harvest datasets for efficient CLIP model trainingcited by 0
  17. 2025AFRICAPTION: Establishing a New Paradigm for Image Captioning in African Languagescited by 0
  18. 2026Xray-Visual Models: Scaling Vision models on Industry Scale Datacited by 0
  19. 2026LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-raycited by 0
  20. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretrainingcited by 0
  21. 2026Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Modelscited by 0
  22. 2026MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Predictioncited by 0
  23. 2026Does Vision-Language Pretraining Granularity Matter? A Controlled Evaluation of Vision-Language Objectives Across Chest X-Ray Interpretation Taskscited by 0