mlsanityData to 14 Oct 2026

CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects

arXiv 2510.14904 ↗First posted 16 Oct 2025cs.CV cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Dense Video Object Captioning (DVOC) is the task of jointly detecting, tracking, and captioning object trajectories in a video, requiring the ability to understand spatio-temporal details and describe them in natural language. Due to the complexity of the task and the high cost associated with manual annotation, previous approaches resort to training strategies with limited data, potentially leading to suboptimal performance. To circumvent this issue, we propose to generate captions about spatio-temporally localized entities leveraging a state-of-the-art VLM, and extend the LVIS and LV-VIS datasets with our synthetic captions (LVISCap and LV-VISCap). Moreover, we introduce an end-to-end model, CaptionFormer, capable of jointly detecting, segmenting, tracking and captioning object trajectories. CaptionFormer achieves state-of-the-art DVOC results on three existing benchmarks, VidSTG, VLN and BenSMOT. The datasets and code are available at https://www.gabriel.fiastre.fr/captionformer/.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 77 of its 86 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

77 of 86 references.

References · 77

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2017Mask R-CNNcited by 7,277
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  13. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  14. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  15. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  16. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  17. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  18. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  19. 2019Objects as Pointscited by 1,175
  20. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  21. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 910
  22. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  23. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  24. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  25. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 823
  26. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  27. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  28. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  29. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  30. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 552
  31. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  32. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 532
  33. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  34. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  35. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  36. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  37. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  38. 2015Visual7W: Grounded Question Answering in Imagescited by 318

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Instance-Aligned Captions for Explainable Video Anomaly Detectioncited by 4
  2. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  3. 2026One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understandingcited by 1