# DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding

arXiv: https://arxiv.org/abs/2312.02549 · HTML: https://mlsanity.com/paper/2312.02549/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Mega: Moving Average Equipped Gated Attention](https://mlsanity.com/paper/2209.10655/) (2022)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment](https://mlsanity.com/paper/1812.00087/) (2018)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Localizing Moments in Video with Temporal Language](https://mlsanity.com/paper/1809.01337/) (2018)
- [ExCL: Extractive Clip Localization Using Natural Language Descriptions](https://mlsanity.com/paper/1904.02755/) (2019)
- [Less is More: Learning Highlight Detection from Video Duration](https://mlsanity.com/paper/1903.00859/) (2019)
- [Contrastive Learning for Neural Topic Model](https://mlsanity.com/paper/2110.12764/) (2021)
- [MINI-Net: Multiple Instance Ranking Network for Video Highlight Detection](https://mlsanity.com/paper/2007.09833/) (2020)
- [MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer](https://mlsanity.com/paper/2305.00355/) (2023)
- [Video Question Answering via Attribute-Augmented Attention Network Learning](https://mlsanity.com/paper/1707.06355/) (2017)
- [Cross-category Video Highlight Detection via Set-based Learning](https://mlsanity.com/paper/2108.11770/) (2021)

## Built on by (most cited first, 11 of 11)

- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation](https://mlsanity.com/paper/2412.07160/) (2024)
- [Encoding and Controlling Global Semantics for Long-form Video Question Answering](https://mlsanity.com/paper/2405.19723/) (2024)
- [MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learning](https://mlsanity.com/paper/2407.03788/) (2024)
- [TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systems](https://mlsanity.com/paper/2503.06380/) (2025)
- [Multi-Scale Contrastive Learning for Video Temporal Grounding](https://mlsanity.com/paper/2412.07157/) (2024)
- [CutPaste&Find: Efficient Multimodal Hallucination Detector with Visual-aid Knowledge Base](https://mlsanity.com/paper/2502.12591/) (2025)
- [Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation](https://mlsanity.com/paper/2501.14166/) (2025)
- [READ: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling](https://mlsanity.com/paper/2312.06950/) (2023)
- [Topic-aware Causal Intervention for Counterfactual Detection](https://mlsanity.com/paper/2409.16668/) (2024)
- [Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding](https://mlsanity.com/paper/2505.12605/) (2025)

## Abstract (arXiv)

Temporal Language Grounding seeks to localize video moments that semantically correspond to a natural language query. Recent advances employ the attention mechanism to learn the relations between video moments and the text query. However, naive attention might not be able to appropriately capture such relations, resulting in ineffective distributions where target video moments are difficult to separate from the remaining ones. To resolve the issue, we propose an energy-based model framework to explicitly learn moment-query distributions. Moreover, we propose DemaFormer, a novel Transformer-based architecture that utilizes exponential moving average with a learnable damping factor to effectively encode moment-query inputs. Comprehensive experiments on four public temporal language grounding datasets showcase the superiority of our methods over the state-of-the-art baselines.