# Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments

arXiv: https://arxiv.org/abs/2505.22137 · HTML: https://mlsanity.com/paper/2505.22137/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 30 of its 85 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](https://mlsanity.com/paper/2010.08240/) (2020)
- [Parsing Argumentation Structures in Persuasive Essays](https://mlsanity.com/paper/1604.07370/) (2016)
- [Argumentation Mining in User-Generated Web Discourse](https://mlsanity.com/paper/1601.02403/) (2016)
- [Cross-topic Argument Mining from Heterogeneous Sources Using Attention-based Neural Networks](https://mlsanity.com/paper/1802.05758/) (2018)
- [Classification and Clustering of Arguments with Contextualized Word Embeddings](https://mlsanity.com/paper/1906.09821/) (2019)
- [On the Difficulty of Evaluating Baselines: A Study on Recommender Systems](https://mlsanity.com/paper/1905.01395/) (2019)
- [Argument Mining for Understanding Peer Reviews](https://mlsanity.com/paper/1903.10104/) (2019)
- [Argument Mining with Structured SVMs and RNNs](https://mlsanity.com/paper/1704.06869/) (2017)
- [Mining Legal Arguments in Court Decisions](https://mlsanity.com/paper/2208.06178/) (2022)
- [What is the Essence of a Claim? Cross-Domain Claim Identification](https://mlsanity.com/paper/1704.07203/) (2017)
- [Fine-Grained Argument Unit Recognition and Classification](https://mlsanity.com/paper/1904.09688/) (2019)
- [Measuring the Similarity of Sentential Arguments in Dialog](https://mlsanity.com/paper/1709.01887/) (2017)
- [IAM: A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks](https://mlsanity.com/paper/2203.12257/) (2022)
- [Argument Mining Driven Analysis of Peer-Reviews](https://mlsanity.com/paper/2012.07743/) (2020)
- [Unsupervised Expressive Rules Provide Explainability and Assist Human Experts Grasping New Domains](https://mlsanity.com/paper/2010.09459/) (2020)
- [Simplistic Collection and Labeling Practices Limit the Utility of Benchmark Datasets for Twitter Bot Detection](https://mlsanity.com/paper/2301.07015/) (2023)
- [Dissecting Content and Context in Argumentative Relation Analysis](https://mlsanity.com/paper/1906.03338/) (2019)
- [Building a Web-Scale Dependency-Parsed Corpus from CommonCrawl](https://mlsanity.com/paper/1710.01779/) (2017)
- [The Discussion Tracker Corpus of Collaborative Argumentation](https://mlsanity.com/paper/2005.11344/) (2020)
- [Aspect-Based Argument Mining](https://mlsanity.com/paper/2011.00633/) (2020)
- [Topic Ontologies for Arguments](https://mlsanity.com/paper/2301.09759/) (2023)

## Built on by (most cited first, 5 of 5)

- [ArgBench: Benchmarking LLMs on Computational Argumentation Tasks](https://mlsanity.com/paper/2604.17366/) (2026)
- [Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation](https://mlsanity.com/paper/2604.19331/) (2026)
- [The Argument is the Explanation: Structured Argumentation for Trust in Agents](https://mlsanity.com/paper/2510.03442/) (2025)
- [Improving Neural Argumentative Stance Classification in Controversial Topics with Emotion-Lexicon Features](https://mlsanity.com/paper/2602.22846/) (2026)
- [Argument Structure Prediction in Online Conversations: A Comparative Study of Modeling Paradigms and Task Architectures](https://mlsanity.com/paper/2609.39225/) (2026)

## Abstract (arXiv)

Identifying arguments is a necessary prerequisite for various tasks in automated discourse analysis, particularly within contexts such as political debates, online discussions, and scientific reasoning. In addition to theoretical advances in understanding the constitution of arguments, a significant body of research has emerged around practical argument mining, supported by a growing number of publicly available datasets. On these benchmarks, BERT-like transformers have consistently performed best, reinforcing the belief that such models are broadly applicable across diverse contexts of debate. This study offers the first large-scale re-evaluation of such state-of-the-art models, with a specific focus on their ability to generalize in identifying arguments. We evaluate four transformers, three standard and one enhanced with contrastive pre-training for better generalization, on 17 English sentence-level datasets as most relevant to the task. Our findings show that, to varying degrees, these models tend to rely on lexical shortcuts tied to content words, suggesting that apparent progress may often be driven by dataset-specific cues rather than true task alignment. While the models achieve strong results on familiar benchmarks, their performance drops markedly when applied to unseen datasets. Nonetheless, incorporating both task-specific pre-training and joint benchmark training proves effective in enhancing both robustness and generalization.