ViTOC: Vision Transformer and Object-aware Captioner
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
This paper presents ViTOC (Vision Transformer and Object-aware Captioner), a novel vision-language model for image captioning that addresses the challenges of accuracy and diversity in generated descriptions. Unlike conventional approaches, ViTOC employs a dual-path architecture based on Vision Transformer and object detector, effectively fusing global visual features and local object information through learnable vectors. The model introduces an innovative object-aware prompting strategy that significantly enhances its capability in handling long-tail data. Experiments on the standard COCO dataset demonstrate that ViTOC outperforms baseline models across all evaluation metrics. Additionally, we propose a reference-free evaluation method based on CLIP to further validate the model's effectiveness. By utilizing pretrained visual model parameters, ViTOC achieves efficient end-to-end training.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 25 references that are in the map. First posted on arXiv 9 Nov 2024.
Builds on
21 of 25 references.
References · 21
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
- 2014Show and Tell: A Neural Image Caption Generatorcited by 877
9 more
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
- 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
- 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
- 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
- 2021CPTR: Full Transformer Network for Image Captioningcited by 55
- 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
- 2022End-to-End Transformer Based Model for Image Captioningcited by 37
- 2020Image Captioning through Image Transformercited by 29
- 2021Semi-Autoregressive Transformer for Image Captioningcited by 12
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022Retrieval-Augmented Transformer for Image Captioning
- 2025Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation
- 2021Scaling Up Vision-Language Pre-training for Image Captioning
- 2026Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Dataset
- 2023Vision Language Transformers: A Survey
- 2025Generating Accurate and Detailed Captions for High-Resolution Images
- 2022Deep Learning Approaches on Image Captioning: A Review
- 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning