# Real-time Factuality Assessment from Adversarial Feedback

arXiv: https://arxiv.org/abs/2410.14651 · HTML: https://mlsanity.com/paper/2410.14651/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2306.05540/) (2023)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [RADAR: Robust AI-Text Detection via Adversarial Learning](https://mlsanity.com/paper/2307.03838/) (2023)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection](https://mlsanity.com/paper/2309.12247/) (2023)
- [FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19](https://mlsanity.com/paper/2006.11343/) (2020)
- [Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation](https://mlsanity.com/paper/2310.15515/) (2023)
- [Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4](https://mlsanity.com/paper/2305.14928/) (2023)
- [Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generation](https://mlsanity.com/paper/2203.05386/) (2022)
- [Fact-Enhanced Synthetic News Generation](https://mlsanity.com/paper/2012.04778/) (2020)
- [Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence](https://mlsanity.com/paper/2305.03507/) (2023)
- [Re-Search for The Truth: Multi-round Retrieval-augmented Large Language Models are Strong Fake News Detectors](https://mlsanity.com/paper/2403.09747/) (2024)

## Built on by (most cited first, 12 of 12)

- [DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality](https://mlsanity.com/paper/2603.05912/) (2026)
- [The Facade of Truth: Uncovering and Mitigating LLM Susceptibility to Deceptive Evidence](https://mlsanity.com/paper/2601.05478/) (2026)
- [Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models](https://mlsanity.com/paper/2506.17585/) (2025)
- [LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection](https://mlsanity.com/paper/2604.04815/) (2026)
- [LLM-Based Adversarial Persuasion Attacks on Fact-Checking Systems](https://mlsanity.com/paper/2601.16890/) (2026)
- [RADAR: Retrieval-Augmented Detector with Adversarial Refinement for Adaptive LLM-Generated Fake News Detection](https://mlsanity.com/paper/2601.03981/) (2026)
- [CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild](https://mlsanity.com/paper/2605.30241/) (2026)
- [ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection](https://mlsanity.com/paper/2606.24112/) (2026)
- [Lazy Grounding: Attacking Search Agents with Factual Evidence](https://mlsanity.com/paper/2608.30303/) (2026)
- [SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs](https://mlsanity.com/paper/2609.10901/) (2026)
- [FakeSpotter: A content and strategy agnostic Viral Misinformation Detection Tool](https://mlsanity.com/paper/2609.19152/) (2026)
- [VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation](https://mlsanity.com/paper/2609.35028/) (2026)

## Abstract (arXiv)

We show that existing evaluations for assessing the factuality of news from conventional sources, such as claims on fact-checking websites, result in high accuracies over time for LLM-based detectors-even after their knowledge cutoffs. This suggests that recent popular false information from such sources can be easily identified due to its likely presence in pre-training/retrieval corpora or the emergence of salient, yet shallow, patterns in these datasets. Instead, we argue that a proper factuality evaluation dataset should test a model's ability to reason about current events by retrieving and reading related evidence. To this end, we develop a novel pipeline that leverages natural language feedback from a RAG-based detector to iteratively modify real-time news into deceptive variants that challenge LLMs. Our iterative rewrite decreases the binary classification ROC-AUC by an absolute 17.5 percent for a strong RAG-based GPT-4o detector. Our experiments reveal the important role of RAG in both evaluating and generating challenging news examples, as retrieval-free LLM detectors are vulnerable to unseen events and adversarial attacks, while feedback from RAG-based evaluation helps discover more deceitful patterns.