# What do writing features tell us about AI papers?

arXiv: https://arxiv.org/abs/2107.06310 · HTML: https://mlsanity.com/paper/2107.06310/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 19 of its 66 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [CausaLM: Causal Model Explanation Through Counterfactual Language Models](https://mlsanity.com/paper/2005.13407/) (2020)
- [How to Make Causal Inferences Using Texts](https://mlsanity.com/paper/1802.02163/) (2018)
- [Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates](https://mlsanity.com/paper/2005.00649/) (2020)
- [Challenges of Using Text Classifiers for Causal Inference](https://mlsanity.com/paper/1810.00956/) (2018)
- [What Can We Do to Improve Peer Review in NLP?](https://mlsanity.com/paper/2010.03863/) (2020)
- [Does My Rebuttal Matter? Insights from a Major NLP Conference](https://mlsanity.com/paper/1903.11367/) (2019)
- [On Testing for Biases in Peer Review](https://mlsanity.com/paper/1912.13188/) (2019)
- [SChuBERT: Scholarly Document Chunks with BERT-encoding boost Citation Count Prediction](https://mlsanity.com/paper/2012.11740/) (2020)
- [Automatic Academic Paper Rating Based on Modularized Hierarchical Convolutional Neural Network](https://mlsanity.com/paper/1805.03977/) (2018)
- [Deep Paper Gestalt](https://mlsanity.com/paper/1812.08775/) (2018)
- [Textual analysis of artificial intelligence manuscripts reveals features associated with peer review outcome](https://mlsanity.com/paper/1911.02648/) (2019)
- [What Makes a Scientific Paper be Accepted for Publication?](https://mlsanity.com/paper/2104.07112/) (2021)
- [Detecting and Explaining Causes From Text For a Time Series Event](https://mlsanity.com/paper/1707.08852/) (2017)
- [De-anonymization of authors through arXiv submissions during double-blind review](https://mlsanity.com/paper/2007.00177/) (2020)

## Abstract (arXiv)

As the numbers of submissions to conferences grow quickly, the task of assessing the quality of academic papers automatically, convincingly, and with high accuracy attracts increasing attention. We argue that studying interpretable dimensions of these submissions could lead to scalable solutions. We extract a collection of writing features, and construct a suite of prediction tasks to assess the usefulness of these features in predicting citation counts and the publication of AI-related papers. Depending on the venues, the writing features can predict the conference vs. workshop appearance with F1 scores up to 60-90, sometimes even outperforming the content-based tf-idf features and RoBERTa. We show that the features describe writing style more than content. To further understand the results, we estimate the causal impact of the most indicative features. Our analysis on writing features provides a perspective to assessing and refining the writing of academic articles at scale.