# Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals

arXiv: https://arxiv.org/abs/2309.07601 · HTML: https://mlsanity.com/paper/2309.07601/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 70 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [Automatic Detection of Fake News](https://mlsanity.com/paper/1708.07104/) (2017)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [CoAID: COVID-19 Healthcare Misinformation Dataset](https://mlsanity.com/paper/2006.00885/) (2020)
- [A Stylometric Inquiry into Hyperpartisan and Fake News](https://mlsanity.com/paper/1702.05638/) (2017)
- [Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads](https://mlsanity.com/paper/1511.07487/) (2015)
- [This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News](https://mlsanity.com/paper/1703.09398/) (2017)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [BanFakeNews: A Dataset for Detecting Fake News in Bangla](https://mlsanity.com/paper/2004.08789/) (2020)
- [Weak Supervision for Fake News Detection via Reinforcement Learning](https://mlsanity.com/paper/1912.12520/) (2019)
- [Some Like it Hoax: Automated Fake News Detection in Social Networks](https://mlsanity.com/paper/1704.07506/) (2017)
- [Multi-Resolution Weak Supervision for Sequential Data](https://mlsanity.com/paper/1910.09505/) (2019)
- [Credibility-based Fake News Detection](https://mlsanity.com/paper/1911.00643/) (2019)

## Built on by (most cited first, 14 of 14)

- [DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection](https://mlsanity.com/paper/2402.10426/) (2024)
- [What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection](https://mlsanity.com/paper/2402.00371/) (2024)
- [User Modeling in the Era of Large Language Models: Current Research and Future Directions](https://mlsanity.com/paper/2312.11518/) (2023)
- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025)
- [Large Language Models' Detection of Political Orientation in Newspapers](https://mlsanity.com/paper/2406.00018/) (2024)
- [Tailored untruths: How personalisation challenges LLM safeguards](https://mlsanity.com/paper/2510.12993/) (2025)
- [Bangla AI: A Framework for Machine Translation Utilizing Large Language Models for Ethnic Media](https://mlsanity.com/paper/2402.14179/) (2024)
- [Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuning](https://mlsanity.com/paper/2509.07768/) (2025)
- [The Tag is the Signal: URL-Agnostic Credibility Scoring for Messages on Telegram](https://mlsanity.com/paper/2601.13294/) (2026)
- [Truth with a Twist: The Rhetoric of Persuasion in Professional vs. Community-Authored Fact-Checks](https://mlsanity.com/paper/2601.14105/) (2026)
- [MMKG-RDS: Reasoning Data Synthesis via Deep Mining of Multimodal Knowledge Graphs](https://mlsanity.com/paper/2602.23632/) (2026)
- [Are LLMs More Skeptical of Entertainment News?](https://mlsanity.com/paper/2605.01727/) (2026)
- [Can temporal article-level credibility signals improve domain-level credibility prediction?](https://mlsanity.com/paper/2607.04560/) (2026)
- [From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference](https://mlsanity.com/paper/2609.24219/) (2026)

## Abstract (arXiv)

Credibility signals represent a wide range of heuristics typically used by journalists and fact-checkers to assess the veracity of online content. Automating the extraction of credibility signals presents significant challenges due to the necessity of training high-accuracy, signal-specific extractors, coupled with the lack of sufficiently large annotated datasets. This paper introduces Pastel (Prompted weAk Supervision wiTh crEdibility signaLs), a weakly supervised approach that leverages large language models (LLMs) to extract credibility signals from web content, and subsequently combines them to predict the veracity of content without relying on human supervision. We validate our approach using four article-level misinformation detection datasets, demonstrating that Pastel outperforms zero-shot veracity detection by 38.3% and achieves 86.7% of the performance of the state-of-the-art system trained with human supervision. Moreover, in cross-domain settings where training and testing datasets originate from different domains, Pastel significantly outperforms the state-of-the-art supervised model by 63%. We further study the association between credibility signals and veracity, and perform an ablation study showing the impact of each signal on model performance. Our findings reveal that 12 out of the 19 proposed signals exhibit strong associations with veracity across all datasets, while some signals show domain-specific strengths.