mlsanityData to 7 Oct 2026

RefineRank: Joint Box Refinement and Ranking for Surgical Spatio-Temporal Grounding

arXiv 2608.23928 ↗First posted 25 Aug 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Surgical spatio-temporal grounding (STG) requires locating, at each video time specified by a procedural question, the object that the question asks about. Existing approaches face a trade-off: vision language models understand the question context but produce imprecise coordinates, whereas open-set detectors provide localized candidate boxes whose confidence does not reflect which box answers the question. We introduce RefineRank, which closes this gap at the candidate-box level. A compact trainable module, RefineNet, combines the language and regional features of a frozen medical vision language model with the proposals of a frozen open-set detector: it predicts a bounded coordinate correction and a quality score for every candidate box, and a fixed decoding rule returns the original or refined box with the highest score. On the MedVidBench Official Rankings (Verified), RefineRank records 0.421 STG mIoU, the highest displayed STG score, while its global multi-metric rank is 11. In a controlled evaluation on separate training and evaluation videos, coordinate correction raises the candidate oracle upper bound from 0.6772 to 0.7302, and ranking the joint pool of original and refined candidates by their RefineNet scores improves STG mIoU from 0.2719 to 0.4534, whereas separately trained selectors over the same pool reach at most 0.4186. These results show that a small box-level module can reconcile question understanding with precise localization without retraining either backbone. Code is available at [https://github.com/linzhe001/RefineRank](https://github.com/linzhe001/RefineRank).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 22 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

18 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding6 Dec
  2. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection10 Oct

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2015Fast R-CNNcited by 2,930
  4. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  5. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  8. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  10. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 408
  11. 2020VarifocalNet: An IoU-aware Dense Object Detectorcited by 230
  12. 2018Acquisition of Localization Confidence for Accurate Object Detectioncited by 174
4 more
  1. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  2. 2022Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Groundingcited by 40
  3. 2023CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Toolscited by 17
  4. 2024EgoSurgery-Tool: A Dataset of Surgical Tool and Hand Detection from Egocentric Open Surgery Videoscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs
  2. 2026Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI
  3. 2026GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation
  4. 2026Medical Image Spatial Grounding with Semantic Sampling
  5. 2026LocAnyMed: Vision-Language Grounding for Multimodal Medical Images
  6. 2026An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models
  7. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery
  8. 2026Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology