# RLSAC: Reinforcement Learning enhanced Sample Consensus for End-to-End Robust Estimation

arXiv: https://arxiv.org/abs/2308.05318 · HTML: https://mlsanity.com/paper/2308.05318/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 19 of 19)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [Graph-Cut RANSAC](https://mlsanity.com/paper/1706.00984/) (2017)
- [Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses](https://mlsanity.com/paper/1905.04132/) (2019)
- [Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task](https://mlsanity.com/paper/1912.00623/) (2019)
- [PWCLO-Net: Deep LiDAR Odometry in 3D Point Clouds Using Hierarchical Embedding Mask Optimization](https://mlsanity.com/paper/2012.00972/) (2020)
- [MODS: Fast and Robust Method for Two-View Matching](https://mlsanity.com/paper/1503.02619/) (2015)
- [Efficient 3D Deep LiDAR Odometry](https://mlsanity.com/paper/2111.02135/) (2021)
- [Progressive NAPSAC: sampling from gradually growing neighborhoods](https://mlsanity.com/paper/1906.02295/) (2019)
- [Deep Fundamental Matrix Estimation without Correspondences](https://mlsanity.com/paper/1810.01575/) (2018)
- [3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video](https://mlsanity.com/paper/2112.03045/) (2021)
- [NeFSAC: Neurally Filtered Minimal Samples](https://mlsanity.com/paper/2207.07872/) (2022)
- [Unsupervised Learning of 3D Scene Flow from Monocular Camera](https://mlsanity.com/paper/2206.03673/) (2022)
- [3D Scene Flow Estimation on Pseudo-LiDAR: Bridging the Gap on Estimating Point Motion](https://mlsanity.com/paper/2209.13130/) (2022)
- [NccFlow: Unsupervised Learning of Optical Flow With Non-occlusion from Geometry](https://mlsanity.com/paper/2107.03610/) (2021)

## Built on by (most cited first, 3 of 3)

- [Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences](https://mlsanity.com/paper/2404.06337/) (2024)
- [Deep Learning Reforms Image Matching: A Survey and Outlook](https://mlsanity.com/paper/2506.04619/) (2025)
- [DualReg: Dual-Space Filtering and Reinforcement for Rigid Registration](https://mlsanity.com/paper/2508.17034/) (2025)

## Abstract (arXiv)

Robust estimation is a crucial and still challenging task, which involves estimating model parameters in noisy environments. Although conventional sampling consensus-based algorithms sample several times to achieve robustness, these algorithms cannot use data features and historical information effectively. In this paper, we propose RLSAC, a novel Reinforcement Learning enhanced SAmple Consensus framework for end-to-end robust estimation. RLSAC employs a graph neural network to utilize both data and memory features to guide exploring directions for sampling the next minimum set. The feedback of downstream tasks serves as the reward for unsupervised training. Therefore, RLSAC can avoid differentiating to learn the features and the feedback of downstream tasks for end-to-end robust estimation. In addition, RLSAC integrates a state transition module that encodes both data and memory features. Our experimental results demonstrate that RLSAC can learn from features to gradually explore a better hypothesis. Through analysis, it is apparent that RLSAC can be easily transferred to other sampling consensus-based robust estimation tasks. To the best of our knowledge, RLSAC is also the first method that uses reinforcement learning to sample consensus for end-to-end robust estimation. We release our codes at https://github.com/IRMVLab/RLSAC.