# ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling

arXiv: https://arxiv.org/abs/2602.20166 · HTML: https://mlsanity.com/paper/2602.20166/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 20 of 20)

- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [A Survey of Label-noise Representation Learning: Past, Present and Future](https://mlsanity.com/paper/2011.04406/) (2020)
- [SoftPatch: Unsupervised Anomaly Detection with Noisy Data](https://mlsanity.com/paper/2403.14233/) (2024)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [Understanding the Dark Side of LLMs' Intrinsic Self-Correction](https://mlsanity.com/paper/2412.14959/) (2024)
- [CodeMirage: Hallucinations in Code Generated by Large Language Models](https://mlsanity.com/paper/2408.08333/) (2024)
- [BitsAI-CR: Automated Code Review via LLM in Practice](https://mlsanity.com/paper/2501.15134/) (2025)
- [AI-Assisted Assessment of Coding Practices in Modern Code Review](https://mlsanity.com/paper/2405.13565/) (2024)
- [What Types of Code Review Comments Do Developers Most Frequently Resolve?](https://mlsanity.com/paper/2510.05450/) (2025)
- [A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI](https://mlsanity.com/paper/2511.00776/) (2025)
- [iCodeReviewer: Improving Secure Code Review with Mixture of Prompts](https://mlsanity.com/paper/2510.12186/) (2025)
- [Issue-Oriented Agent-Based Framework for Automated Review Comment Generation](https://mlsanity.com/paper/2511.00517/) (2025)
- [An Empirical Study on Noisy Label Learning for Program Understanding](https://mlsanity.com/paper/2307.08990/) (2023)
- [Large Language Models as Information Sources: Distinctive Characteristics and Types of Low-Quality Information](https://mlsanity.com/paper/2511.03198/) (2025)

## Abstract (arXiv)

In many applications involving intelligent agents, the overwhelming volume of alerts (mostly false) generated by the agents may desensitize users and cause them to overlook critical issues, leading to the so-called ''alert fatigue''. A common strategy is to train a reflection model as a filter to intercept false alerts with labelled data collected from user verification feedback. However, a key challenge is the noisy nature of such data as it is often collected in production environments. As cleaning noise via manual annotation incurs high costs, this paper proposes a novel method ConceptRM for constructing a high-quality corpus to train a reflection model capable of effectively intercepting false alerts. With only a small amount of expert annotations as anchors, ConceptRM creates perturbed datasets with varying noise ratios and utilizes co-teaching to train multiple distinct models for collaborative learning. By analyzing the consensus decisions of these models, it effectively identifies reliable negative samples from a noisy dataset. Experimental results demonstrate that ConceptRM significantly enhances the interception of false alerts with minimal annotation cost, outperforming several state-of-the-art LLM baselines by up to 53.31% on in-domain datasets and 41.67% on out-of-domain datasets.