# Towards Trustworthy Deception Detection: Benchmarking Model Robustness across Domains, Modalities, and Languages

arXiv: https://arxiv.org/abs/2104.11761 · HTML: https://mlsanity.com/paper/2104.11761/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 63 references that are in the map.
- First posted on arXiv 23 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation](https://mlsanity.com/paper/1705.08475/) (2017)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [Towards Robust Neural Networks via Random Self-ensemble](https://mlsanity.com/paper/1712.00673/) (2017)
- [Improving the Robustness of Deep Neural Networks via Stability Training](https://mlsanity.com/paper/1604.04326/) (2016)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [Beyond News Contents: The Role of Social Context for Fake News Detection](https://mlsanity.com/paper/1712.07709/) (2017)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)
- [Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead](https://mlsanity.com/paper/1811.10154/) (2018)
- [Hierarchical Propagation Networks for Fake News Detection: Investigation and Exploitation](https://mlsanity.com/paper/1903.09196/) (2019)
- [XFake: Explainable Fake News Detector with Visualizations](https://mlsanity.com/paper/1907.07757/) (2019)
- [An Army of Me: Sockpuppets in Online Discussion Communities](https://mlsanity.com/paper/1703.07355/) (2017)
- [Fake News Identification on Twitter with Hybrid CNN and RNN Models](https://mlsanity.com/paper/1806.11316/) (2018)
- [Multilingual Stance Detection: The Catalonia Independence Corpus](https://mlsanity.com/paper/2004.00050/) (2020)
- [Discourse-Aware Rumour Stance Classification in Social Media Using Sequential Classifiers](https://mlsanity.com/paper/1712.02223/) (2017)
- [What is Interpretable? Using Machine Learning to Design Interpretable Decision-Support Systems](https://mlsanity.com/paper/1811.10799/) (2018)
- [We Built a Fake News & Click-bait Filter: What Happened Next Will Blow Your Mind!](https://mlsanity.com/paper/1803.03786/) (2018)
- [Propagation from Deceptive News Sources: Who Shares, How Much, How Evenly, and How Quickly?](https://mlsanity.com/paper/1812.03533/) (2018)
- [Open Issues in Combating Fake News: Interpretability as an Opportunity](https://mlsanity.com/paper/1904.03016/) (2019)
- [How Humans versus Bots React to Deceptive and Trusted News Sources: A Case Study of Active Users](https://mlsanity.com/paper/1807.05327/) (2018)

## Built on by (most cited first, 2 of 2)

- [Domain-Independent Deception: A New Taxonomy and Linguistic Analysis](https://mlsanity.com/paper/2402.01019/) (2024)
- [A Roadmap for Multilingual, Multimodal Domain Independent Deception Detection](https://mlsanity.com/paper/2405.03920/) (2024)

## Abstract (arXiv)

Evaluating model robustness is critical when developing trustworthy models not only to gain deeper understanding of model behavior, strengths, and weaknesses, but also to develop future models that are generalizable and robust across expected environments a model may encounter in deployment. In this paper we present a framework for measuring model robustness for an important but difficult text classification task - deceptive news detection. We evaluate model robustness to out-of-domain data, modality-specific features, and languages other than English. Our investigation focuses on three type of models: LSTM models trained on multiple datasets(Cross-Domain), several fusion LSTM models trained with images and text and evaluated with three state-of-the-art embeddings, BERT ELMo, and GloVe (Cross-Modality), and character-level CNN models trained on multiple languages (Cross-Language). Our analyses reveal a significant drop in performance when testing neural models on out-of-domain data and non-English languages that may be mitigated using diverse training data. We find that with additional image content as input, ELMo embeddings yield significantly fewer errors compared to BERT orGLoVe. Most importantly, this work not only carefully analyzes deception model robustness but also provides a framework of these analyses that can be applied to new models or extended datasets in the future.