# Perturbations and Subpopulations for Testing Robustness in Token-Based Argument Unit Recognition

arXiv: https://arxiv.org/abs/2209.14780 · HTML: https://mlsanity.com/paper/2209.14780/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Neural End-to-End Learning for Computational Argumentation Mining](https://mlsanity.com/paper/1704.06104/) (2017)
- [Argument Mining with Structured SVMs and RNNs](https://mlsanity.com/paper/1704.06869/) (2017)
- [Stance Detection Benchmark: How Robust Is Your Stance Detection?](https://mlsanity.com/paper/2001.01565/) (2020)
- [Fine-Grained Argument Unit Recognition and Classification](https://mlsanity.com/paper/1904.09688/) (2019)
- [Transformer-Based Models for Automatic Identification of Argument Relations: A Cross-Domain Evaluation](https://mlsanity.com/paper/2011.13187/) (2020)
- [Scientia Potentia Est -- On the Role of Knowledge in Computational Argumentation](https://mlsanity.com/paper/2107.00281/) (2021)
- [Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Study](https://mlsanity.com/paper/2110.07693/) (2021)
- [Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis](https://mlsanity.com/paper/1806.05219/) (2018)

## Abstract (arXiv)

Argument Unit Recognition and Classification aims at identifying argument units from text and classifying them as pro or against. One of the design choices that need to be made when developing systems for this task is what the unit of classification should be: segments of tokens or full sentences. Previous research suggests that fine-tuning language models on the token-level yields more robust results for classifying sentences compared to training on sentences directly. We reproduce the study that originally made this claim and further investigate what exactly token-based systems learned better compared to sentence-based ones. We develop systematic tests for analysing the behavioural differences between the token-based and the sentence-based system. Our results show that token-based models are generally more robust than sentence-based models both on manually perturbed examples and on specific subpopulations of the data.