# How to Train Your Energy-Based Model for Regression

arXiv: https://arxiv.org/abs/2005.01698 · HTML: https://mlsanity.com/paper/2005.01698/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2020.

## Built on by (most cited first, 23 of 23)

- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Learning Target Candidate Association to Keep Track of What Not to Track](https://mlsanity.com/paper/2103.16556/) (2021)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Vision-Language Navigation with Energy-Based Policy](https://mlsanity.com/paper/2410.14250/) (2024)
- [Learning Tracking Representations via Dual-Branch Fully Transformer Networks](https://mlsanity.com/paper/2112.02571/) (2021)
- [The 2nd Anti-UAV Workshop & Challenge: Methods and Results](https://mlsanity.com/paper/2108.09909/) (2021)
- [Energy-Based Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2202.04933/) (2022)
- [Taming Diffusion Models for Image Restoration: A Review](https://mlsanity.com/paper/2409.10353/) (2024)
- [Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning](https://mlsanity.com/paper/2106.02938/) (2021)
- [The 3rd Anti-UAV Workshop & Challenge: Methods and Results](https://mlsanity.com/paper/2305.07290/) (2023)
- [An energy-based deep splitting method for the nonlinear filtering problem](https://mlsanity.com/paper/2203.17153/) (2022)
- [Physics-informed Neural Networks with Unknown Measurement Noise](https://mlsanity.com/paper/2211.15498/) (2022)
- [Robust Imitation Learning for Automated Game Testing](https://mlsanity.com/paper/2401.04572/) (2024)
- [Energy-Based Reward Models for Robust Language Model Alignment](https://mlsanity.com/paper/2504.13134/) (2025)
- [Learning Proposals for Practical Energy-Based Regression](https://mlsanity.com/paper/2110.11948/) (2021)
- [Conditional Energy-Based Models for Implicit Policies: The Gap between Theory and Practice](https://mlsanity.com/paper/2207.05824/) (2022)
- [Energy-Based Models for Cross-Modal Localization using Convolutional Transformers](https://mlsanity.com/paper/2306.04021/) (2023)
- [Energy Aligning for Biased Models](https://mlsanity.com/paper/2106.03343/) (2021)
- [On the connection between Noise-Contrastive Estimation and Contrastive Divergence](https://mlsanity.com/paper/2402.16688/) (2024)
- [EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering](https://mlsanity.com/paper/2206.14355/) (2022)
- [On Feature Diversity in Energy-based Models](https://mlsanity.com/paper/2306.01489/) (2023)
- [Probabilistic Matching of Real and Generated Data Statistics in Generative Adversarial Networks](https://mlsanity.com/paper/2306.10943/) (2023)
- [Learning Energy-Based Models by Self-normalising the Likelihood](https://mlsanity.com/paper/2503.07021/) (2025)

## Abstract (arXiv)

Energy-based models (EBMs) have become increasingly popular within computer vision in recent years. While they are commonly employed for generative image modeling, recent work has applied EBMs also for regression tasks, achieving state-of-the-art performance on object detection and visual tracking. Training EBMs is however known to be challenging. While a variety of different techniques have been explored for generative modeling, the application of EBMs to regression is not a well-studied problem. How EBMs should be trained for best possible regression performance is thus currently unclear. We therefore accept the task of providing the first detailed study of this problem. To that end, we propose a simple yet highly effective extension of noise contrastive estimation, and carefully compare its performance to six popular methods from literature on the tasks of 1D regression and object detection. The results of this comparison suggest that our training method should be considered the go-to approach. We also apply our method to the visual tracking task, achieving state-of-the-art performance on five datasets. Notably, our tracker achieves 63.7% AUC on LaSOT and 78.7% Success on TrackingNet. Code is available at https://github.com/fregu856/ebms_regression.