# Assessing the Impact of Sequence Length Learning on Classification Tasks for Transformer Encoder Models

arXiv: https://arxiv.org/abs/2212.08399 · HTML: https://mlsanity.com/paper/2212.08399/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 16 Dec 2022.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://mlsanity.com/paper/2203.12942/) (2022)
- [On Adversarial Removal of Hypothesis-only Bias in Natural Language Inference](https://mlsanity.com/paper/1907.04389/) (2019)
- [Debiasing Embeddings for Reduced Gender Bias in Text Classification](https://mlsanity.com/paper/1908.02810/) (2019)

## Built on by (most cited first, 3 of 3)

- [What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval](https://mlsanity.com/paper/2605.24524/) (2026)
- [Beyond checkmate: exploring the creative chokepoints in AI text](https://mlsanity.com/paper/2501.19301/) (2025)
- [Artificial Conversations, Real Results: Fostering Language Detection with Synthetic Data](https://mlsanity.com/paper/2503.24062/) (2025)

## Abstract (arXiv)

Classification algorithms using Transformer architectures can be affected by the sequence length learning problem whenever observations from different classes have a different length distribution. This problem causes models to use sequence length as a predictive feature instead of relying on important textual information. Although most public datasets are not affected by this problem, privately owned corpora for fields such as medicine and insurance may carry this data bias. The exploitation of this sequence length feature poses challenges throughout the value chain as these machine learning models can be used in critical applications. In this paper, we empirically expose this problem and present approaches to minimize its impacts.