# Fine-Tuning Language Models for Scientific Writing Support

arXiv: https://arxiv.org/abs/2306.10974 · HTML: https://mlsanity.com/paper/2306.10974/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 44 references that are in the map.
- First posted on arXiv 19 Jun 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Building a Conversational Agent Overnight with Dialogue Self-Play](https://mlsanity.com/paper/1801.04871/) (2018)
- [Neural Paraphrase Generation with Stacked Residual LSTM Networks](https://mlsanity.com/paper/1610.03098/) (2016)
- [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://mlsanity.com/paper/2003.10555/) (2020)
- [unarXive 2022: All arXiv Publications Pre-Processed for NLP, Including Structured Full-Text and Citation Network](https://mlsanity.com/paper/2303.14957/) (2023)
- [Unsupervised Paraphrase Generation using Pre-trained Language Models](https://mlsanity.com/paper/2006.05477/) (2020)
- [Ethical ChatGPT: Concerns, Challenges, and Commandments](https://mlsanity.com/paper/2305.10646/) (2023)
- [Text Style Transfer: A Review and Experimental Evaluation](https://mlsanity.com/paper/2010.12742/) (2020)
- [Are We Really Making Much Progress in Text Classification? A Comparative Review](https://mlsanity.com/paper/2204.03954/) (2022)
- [Memorization of Named Entities in Fine-tuned BERT Models](https://mlsanity.com/paper/2212.03749/) (2022)

## Built on by (most cited first, 2 of 2)

- [Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning](https://mlsanity.com/paper/2412.08587/) (2024)
- [Identifying Reliable Evaluation Metrics for Scientific Text Revision](https://mlsanity.com/paper/2506.04772/) (2025)

## Abstract (arXiv)

We support scientific writers in determining whether a written sentence is scientific, to which section it belongs, and suggest paraphrasings to improve the sentence. Firstly, we propose a regression model trained on a corpus of scientific sentences extracted from peer-reviewed scientific papers and non-scientific text to assign a score that indicates the scientificness of a sentence. We investigate the effect of equations and citations on this score to test the model for potential biases. Secondly, we create a mapping of section titles to a standard paper layout in AI and machine learning to classify a sentence to its most likely section. We study the impact of context, i.e., surrounding sentences, on the section classification performance. Finally, we propose a paraphraser, which suggests an alternative for a given sentence that includes word substitutions, additions to the sentence, and structural changes to improve the writing style. We train various large language models on sentences extracted from arXiv papers that were peer reviewed and published at A*, A, B, and C ranked conferences. On the scientificness task, all models achieve an MSE smaller than 2%. For the section classification, BERT outperforms WideMLP and SciBERT in most cases. We demonstrate that using context enhances the classification of a sentence, achieving up to a 90% F1-score. Although the paraphrasing models make comparatively few alterations, they produce output sentences close to the gold standard. Large fine-tuned models such as T5 Large perform best in experiments considering various measures of difference between input sentence and gold standard. Code is provided under https://github.com/JustinMuecke/SciSen.