# Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features

arXiv: https://arxiv.org/abs/2109.12258 · HTML: https://mlsanity.com/paper/2109.12258/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 108 references that are in the map.
- First posted on arXiv 25 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Minimizing Finite Sums with the Stochastic Average Gradient](https://mlsanity.com/paper/1309.2388/) (2013)
- [Text Readability Assessment for Second Language Learners](https://mlsanity.com/paper/1906.07580/) (2019)
- [Fast and Accurate Neural CRF Constituency Parsing](https://mlsanity.com/paper/2008.03736/) (2020)
- [Supervised and Unsupervised Neural Approaches to Text Readability](https://mlsanity.com/paper/1907.11779/) (2019)
- [Linguistic Features for Readability Assessment](https://mlsanity.com/paper/2006.00377/) (2020)
- [Trends, Limitations and Open Challenges in Automatic Readability Assessment Research](https://mlsanity.com/paper/2105.00973/) (2021)
- [ReadNet: A Hierarchical Transformer Framework for Web Article Readability Analysis](https://mlsanity.com/paper/2103.04083/) (2021)
- [Readability-based Sentence Ranking for Evaluating Text Simplification](https://mlsanity.com/paper/1603.06009/) (2016)
- [LXPER Index 2.0: Improving Text Readability Assessment Model for L2 English Students in Korea](https://mlsanity.com/paper/2010.13374/) (2020)
- [LXPER Index: a curriculum-specific text readability assessment model for EFL students in Korea](https://mlsanity.com/paper/2008.01564/) (2020)
- [A Machine Learning Approach to Persian Text Readability Assessment Using a Crowdsourced Dataset](https://mlsanity.com/paper/1810.06639/) (2018)

## Built on by (most cited first, 30 of 36)

- [LFTK: Handcrafted Features in Computational Linguistics](https://mlsanity.com/paper/2305.15878/) (2023)
- [Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach](https://mlsanity.com/paper/2310.06680/) (2023)
- [A Neural Pairwise Ranking Model for Readability Assessment](https://mlsanity.com/paper/2203.07450/) (2022)
- [ChatLog: Carefully Evaluating the Evolution of ChatGPT Across Time](https://mlsanity.com/paper/2304.14106/) (2023)
- [Learning to Paraphrase Sentences to Different Complexity Levels](https://mlsanity.com/paper/2308.02226/) (2023)
- [Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors](https://mlsanity.com/paper/2505.24523/) (2025)
- [Prompt-based Learning for Text Readability Assessment](https://mlsanity.com/paper/2302.13139/) (2023)
- [A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment](https://mlsanity.com/paper/2502.13520/) (2025)
- [Automatic Readability Assessment of German Sentences with Transformer Ensembles](https://mlsanity.com/paper/2209.04299/) (2022)
- [A Unified Neural Network Model for Readability Assessment with Feature Projection and Length-Balanced Loss](https://mlsanity.com/paper/2210.10305/) (2022)
- [Arabic Word-level Readability Visualization for Assisted Text Simplification](https://mlsanity.com/paper/2210.10672/) (2022)
- [Linguistic Blind Spots of Large Language Models](https://mlsanity.com/paper/2503.19260/) (2025)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [EqualizeIR: Mitigating Linguistic Biases in Retrieval Models](https://mlsanity.com/paper/2504.07115/) (2025)
- [Uniform Complexity for Text Generation](https://mlsanity.com/paper/2204.05185/) (2022)
- [Traditional Readability Formulas Compared for English](https://mlsanity.com/paper/2301.02975/) (2023)
- [Language Models Don't Learn the Physical Manifestation of Language](https://mlsanity.com/paper/2402.11349/) (2024)
- [Multi-Objective Linguistic Control of Large Language Models](https://mlsanity.com/paper/2406.16229/) (2024)
- [Ling-CL: Understanding NLP Models through Linguistic Curricula](https://mlsanity.com/paper/2310.20121/) (2023)
- [Complexity-Guided Curriculum Learning for Text Graphs](https://mlsanity.com/paper/2311.13472/) (2023)
- [Guidelines for Fine-grained Sentence-level Arabic Readability Annotation](https://mlsanity.com/paper/2410.08674/) (2024)
- [Zero-shot Large Language Models for Automatic Readability Assessment](https://mlsanity.com/paper/2604.24470/) (2026)
- [BasahaCorpus: An Expanded Linguistic Resource for Readability Assessment in Central Philippine Languages](https://mlsanity.com/paper/2310.11584/) (2023)
- [Strategies for Arabic Readability Modeling](https://mlsanity.com/paper/2407.03032/) (2024)
- [Auto-Select Reading Passages in English Assessment Tests?](https://mlsanity.com/paper/2205.06961/) (2022)
- [Efficient Machine Translation Corpus Generation](https://mlsanity.com/paper/2306.11838/) (2023)
- [FPT: Feature Prompt Tuning for Few-shot Readability Assessment](https://mlsanity.com/paper/2404.02772/) (2024)
- [Exploring Linguistic Features for Turkish Text Readability](https://mlsanity.com/paper/2306.03774/) (2023)
- [Cross-corpus Readability Compatibility Assessment for English Texts](https://mlsanity.com/paper/2306.09704/) (2023)
- [Measuring Adversarial Datasets](https://mlsanity.com/paper/2311.03566/) (2023)

## Abstract (arXiv)

We report two essential improvements in readability assessment: 1. three novel features in advanced semantics and 2. the timely evidence that traditional ML models (e.g. Random Forest, using handcrafted features) can combine with transformers (e.g. RoBERTa) to augment model performance. First, we explore suitable transformers and traditional ML models. Then, we extract 255 handcrafted linguistic features using self-developed extraction software. Finally, we assemble those to create several hybrid models, achieving state-of-the-art (SOTA) accuracy on popular datasets in readability assessment. The use of handcrafted features help model performance on smaller datasets. Notably, our RoBERTA-RF-T1 hybrid achieves the near-perfect classification accuracy of 99%, a 20.3% increase from the previous SOTA.