# The Unreasonable Effectiveness of the Baseline: Discussing SVMs in Legal Text Classification

arXiv: https://arxiv.org/abs/2109.07234 · HTML: https://mlsanity.com/paper/2109.07234/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Metrics for Multi-Class Classification: an Overview](https://mlsanity.com/paper/2008.05756/) (2020)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://mlsanity.com/paper/2103.13084/) (2021)
- [CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service](https://mlsanity.com/paper/1805.01217/) (2018)

## Built on by (most cited first, 5 of 5)

- [Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification](https://mlsanity.com/paper/2303.07142/) (2023)
- [On the Role of Negative Precedent in Legal Outcome Prediction](https://mlsanity.com/paper/2208.08225/) (2022)
- [Leveraging Task Dependency and Contrastive Learning for Case Outcome Classification on European Court of Human Rights Cases](https://mlsanity.com/paper/2302.00768/) (2023)
- [A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets](https://mlsanity.com/paper/2409.05972/) (2024)
- [LLM-Assisted Causal Structure Disambiguation and Factor Extraction for Legal Judgment Prediction](https://mlsanity.com/paper/2603.11446/) (2026)

## Abstract (arXiv)

We aim to highlight an interesting trend to contribute to the ongoing debate around advances within legal Natural Language Processing. Recently, the focus for most legal text classification tasks has shifted towards large pre-trained deep learning models such as BERT. In this paper, we show that a more traditional approach based on Support Vector Machine classifiers reaches surprisingly competitive performance with BERT-based models on the classification tasks in the LexGLUE benchmark. We also highlight that error reduction obtained by using specialised BERT-based models over baselines is noticeably smaller in the legal domain when compared to general language tasks. We present and discuss three hypotheses as potential explanations for these results to support future discussions.