# SciGisPy: a Novel Metric for Biomedical Text Simplification via Gist Inference Score

arXiv: https://arxiv.org/abs/2410.09632 · HTML: https://mlsanity.com/paper/2410.09632/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 12 Oct 2024.

## Builds on (most cited first, 16 of 16)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [BLEU is Not Suitable for the Evaluation of Text Simplification](https://mlsanity.com/paper/1810.05995/) (2018)
- [Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection](https://mlsanity.com/paper/2106.01071/) (2021)
- [Paragraph-level Simplification of Medical Texts](https://mlsanity.com/paper/2104.05767/) (2021)
- [BioSentVec: creating sentence embeddings for biomedical texts](https://mlsanity.com/paper/1810.09302/) (2018)
- [PHEE: A Dataset for Pharmacovigilance Event Extraction from Text](https://mlsanity.com/paper/2210.12560/) (2022)
- [Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies](https://mlsanity.com/paper/2102.08366/) (2021)
- [NapSS: Paragraph-level Medical Text Simplification via Narrative Prompting and Sentence-matching Summarization](https://mlsanity.com/paper/2302.05574/) (2023)
- [A Disentangled Adversarial Neural Topic Model for Separating Opinions from Plots in User Reviews](https://mlsanity.com/paper/2010.11384/) (2020)
- [Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study](https://mlsanity.com/paper/2402.15663/) (2024)
- [TDAM: a Topic-Dependent Attention Model for Sentiment Analysis](https://mlsanity.com/paper/1908.06435/) (2019)
- [GisPy: A Tool for Measuring Gist Inference Score in Text](https://mlsanity.com/paper/2205.12484/) (2022)
- [Topical Phrase Extraction from Clinical Reports by Incorporating both Local and Global Context](https://mlsanity.com/paper/1911.10180/) (2019)

## Built on by (most cited first, 3 of 3)

- [Explaining Matters: Leveraging Definitions and Semantic Expansion for Sexism Detection](https://mlsanity.com/paper/2506.06238/) (2025)
- [SafeSpeech: A Comprehensive and Interactive Tool for Analysing Sexist and Abusive Language in Conversations](https://mlsanity.com/paper/2503.06534/) (2025)
- [IntrEx: A Dataset for Modeling Engagement in Educational Conversations](https://mlsanity.com/paper/2509.06652/) (2025)

## Abstract (arXiv)

Biomedical literature is often written in highly specialized language, posing significant comprehension challenges for non-experts. Automatic text simplification (ATS) offers a solution by making such texts more accessible while preserving critical information. However, evaluating ATS for biomedical texts is still challenging due to the limitations of existing evaluation metrics. General-domain metrics like SARI, BLEU, and ROUGE focus on surface-level text features, and readability metrics like FKGL and ARI fail to account for domain-specific terminology or assess how well the simplified text conveys core meanings (gist). To address this, we introduce SciGisPy, a novel evaluation metric inspired by Gist Inference Score (GIS) from Fuzzy-Trace Theory (FTT). SciGisPy measures how well a simplified text facilitates the formation of abstract inferences (gist) necessary for comprehension, especially in the biomedical domain. We revise GIS for this purpose by introducing domain-specific enhancements, including semantic chunking, Information Content (IC) theory, and specialized embeddings, while removing unsuitable indexes. Our experimental evaluation on the Cochrane biomedical text simplification dataset demonstrates that SciGisPy outperforms the original GIS formulation, with a significant increase in correctly identified simplified texts (84% versus 44.8%). The results and a thorough ablation study confirm that SciGisPy better captures the essential meaning of biomedical content, outperforming existing approaches.