# LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization

arXiv: https://arxiv.org/abs/2505.24282 · HTML: https://mlsanity.com/paper/2505.24282/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Multilevel Language and Vision Integration for Text-to-Clip Retrieval](https://mlsanity.com/paper/1804.05113/) (2018)
- [Context-aware Biaffine Localizing Network for Temporal Sentence Grounding](https://mlsanity.com/paper/2103.11555/) (2021)
- [ExCL: Extractive Clip Localization Using Natural Language Descriptions](https://mlsanity.com/paper/1904.02755/) (2019)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [Uncovering Hidden Challenges in Query-Based Video Moment Retrieval](https://mlsanity.com/paper/2009.00325/) (2020)
- [Natural Language Video Localization: A Revisit in Span-based Question Answering Framework](https://mlsanity.com/paper/2102.13558/) (2021)
- [Memory-Guided Semantic Learning Network for Temporal Sentence Grounding](https://mlsanity.com/paper/2201.00454/) (2022)
- [Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos](https://mlsanity.com/paper/2109.06398/) (2021)
- [Reducing the Vision and Language Bias for Temporal Sentence Grounding](https://mlsanity.com/paper/2207.13457/) (2022)
- [Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding](https://mlsanity.com/paper/2109.06400/) (2021)
- [Relation-aware Video Reading Comprehension for Temporal Language Grounding](https://mlsanity.com/paper/2110.05717/) (2021)
- [Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval](https://mlsanity.com/paper/2312.12155/) (2023)
- [Frame-wise Cross-modal Matching for Video Moment Retrieval](https://mlsanity.com/paper/2009.10434/) (2020)
- [Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2404.09263/) (2024)
- [Video Activity Localisation with Uncertainties in Temporal Boundary](https://mlsanity.com/paper/2206.12923/) (2022)
- [Skimming, Locating, then Perusing: A Human-Like Framework for Natural Language Video Localization](https://mlsanity.com/paper/2207.13450/) (2022)

## Abstract (arXiv)

Language-driven action localization in videos requires not only semantic alignment between language query and video segment, but also prediction of action boundaries. However, the language query primarily describes the main content of an action and usually lacks specific details of action start and end boundaries, which increases the subjectivity of manual boundary annotation and leads to boundary uncertainty in training data. In this paper, on one hand, we propose to expand the original query by generating textual descriptions of the action start and end boundaries through LLMs, which can provide more detailed boundary cues for localization and thus reduce the impact of boundary uncertainty. On the other hand, to enhance the tolerance to boundary uncertainty during training, we propose to model probability scores of action boundaries by calculating the semantic similarities between frames and the expanded query as well as the temporal distances between frames and the annotated boundary frames. They can provide more consistent boundary supervision, thus improving the stability of training. Our method is model-agnostic and can be seamlessly and easily integrated into any existing models of language-driven action localization in an off-the-shelf manner. Experimental results on several datasets demonstrate the effectiveness of our method.