# Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI

arXiv: https://arxiv.org/abs/2308.07213 · HTML: https://mlsanity.com/paper/2308.07213/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 179 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 30 of 58)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [On the Risk of Misinformation Pollution with Large Language Models](https://mlsanity.com/paper/2305.13661/) (2023)
- [Complex Claim Verification with Evidence Retrieved in the Wild](https://mlsanity.com/paper/2305.11859/) (2023)
- [Decontextualization: Making Sentences Stand-Alone](https://mlsanity.com/paper/2102.05169/) (2021)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021)
- [Extending Multilingual BERT to Low-Resource Languages](https://mlsanity.com/paper/2004.13640/) (2020)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [That is a Known Lie: Detecting Previously Fact-Checked Claims](https://mlsanity.com/paper/2005.06058/) (2020)
- [A Survey on Stance Detection for Mis- and Disinformation Identification](https://mlsanity.com/paper/2103.00242/) (2021)
- [Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society](https://mlsanity.com/paper/2005.00033/) (2020)
- [CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning](https://mlsanity.com/paper/2112.08558/) (2021)
- [Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making](https://mlsanity.com/paper/2301.05809/) (2023)
- [Generating Literal and Implied Subquestions to Fact-check Complex Claims](https://mlsanity.com/paper/2205.06938/) (2022)
- [HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution](https://mlsanity.com/paper/2307.16883/) (2023)
- [Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection](https://mlsanity.com/paper/1809.08193/) (2018)
- [Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes](https://mlsanity.com/paper/1904.09612/) (2019)
- [Explanation-Based Human Debugging of NLP Models: A Survey](https://mlsanity.com/paper/2104.15135/) (2021)

## Built on by (most cited first, 9 of 9)

- [Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking](https://mlsanity.com/paper/2502.09083/) (2025)
- [Can LLMs Automate Fact-Checking Article Writing?](https://mlsanity.com/paper/2503.17684/) (2025)
- [Diverse, but Divisive: LLMs Can Exaggerate Gender Differences in Opinion Related to Harms of Misinformation](https://mlsanity.com/paper/2401.16558/) (2024)
- [The AI-DEC: A Card-based Design Method for User-centered AI Explanations](https://mlsanity.com/paper/2405.16711/) (2024)
- [Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking](https://mlsanity.com/paper/2501.18265/) (2025)
- [Talk2Care: Facilitating Asynchronous Patient-Provider Communication with Large-Language-Model](https://mlsanity.com/paper/2309.09357/) (2023)
- [Exploring the Innovation Opportunities for Pre-trained Models](https://mlsanity.com/paper/2505.15790/) (2025)
- [Transparency as Architecture: Structural Compliance Gaps in EU AI Act Article 50 II](https://mlsanity.com/paper/2603.26983/) (2026)
- [Ctrl-F-Resist. Practices, Challenges, and Technical Needs of Civil Society Organizations Monitoring the Far-Right Online](https://mlsanity.com/paper/2609.00808/) (2026)

## Abstract (arXiv)

While many Natural Language Processing (NLP) techniques have been proposed for fact-checking, both academic research and fact-checking organizations report limited adoption of such NLP work due to poor alignment with fact-checker practices, values, and needs. To address this, we investigate a co-design method, Matchmaking for AI, to enable fact-checkers, designers, and NLP researchers to collaboratively identify what fact-checker needs should be addressed by technology, and to brainstorm ideas for potential solutions. Co-design sessions we conducted with 22 professional fact-checkers yielded a set of 11 design ideas that offer a "north star", integrating fact-checker criteria into novel NLP design concepts. These concepts range from pre-bunking misinformation, efficient and personalized monitoring misinformation, proactively reducing fact-checker potential biases, and collaborative writing fact-check reports. Our work provides new insights into both human-centered fact-checking research and practice and AI co-design research.