# MultiSChuBERT: Effective Multimodal Fusion for Scholarly Document Quality Prediction

arXiv: https://arxiv.org/abs/2308.07971 · HTML: https://mlsanity.com/paper/2308.07971/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks](https://mlsanity.com/paper/1903.05987/) (2019)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)
- [Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions](https://mlsanity.com/paper/2107.11508/) (2021)
- [SChuBERT: Scholarly Document Chunks with BERT-encoding boost Citation Count Prediction](https://mlsanity.com/paper/2012.11740/) (2020)
- [Automatic Academic Paper Rating Based on Modularized Hierarchical Convolutional Neural Network](https://mlsanity.com/paper/1805.03977/) (2018)
- [A Joint Model for Multimodal Document Quality Assessment](https://mlsanity.com/paper/1901.01010/) (2019)
- [Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction](https://mlsanity.com/paper/2005.00129/) (2020)
- [NwQM: A neural quality assessment framework for Wikipedia](https://mlsanity.com/paper/2010.06969/) (2020)
- [Cognitive Representation Learning of Self-Media Online Article Quality](https://mlsanity.com/paper/2008.05658/) (2020)

## Built on by (most cited first, 4 of 4)

- [Automatic Evaluation Metrics for Artificially Generated Scientific Research](https://mlsanity.com/paper/2503.05712/) (2025)
- [STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond](https://mlsanity.com/paper/2409.05367/) (2024)
- [ForeCite: Adapting Pre-Trained Language Models to Predict Future Citation Rates of Academic Papers](https://mlsanity.com/paper/2505.08941/) (2025)
- [Explaining word embeddings with perfect fidelity: Case study in research impact prediction](https://mlsanity.com/paper/2409.15912/) (2024)

## Abstract (arXiv)

Automatic assessment of the quality of scholarly documents is a difficult task with high potential impact. Multimodality, in particular the addition of visual information next to text, has been shown to improve the performance on scholarly document quality prediction (SDQP) tasks. We propose the multimodal predictive model MultiSChuBERT. It combines a textual model based on chunking full paper text and aggregating computed BERT chunk-encodings (SChuBERT), with a visual model based on Inception V3.Our work contributes to the current state-of-the-art in SDQP in three ways. First, we show that the method of combining visual and textual embeddings can substantially influence the results. Second, we demonstrate that gradual-unfreezing of the weights of the visual sub-model, reduces its tendency to ovefit the data, improving results. Third, we show the retained benefit of multimodality when replacing standard BERT_textrmBASE embeddings with more recent state-of-the-art text embedding models. Using BERT_textrmBASE embeddings, on the (log) number of citations prediction task with the ACL-BiblioMetry dataset, our MultiSChuBERT (text+visual) model obtains an R2 score of 0.454 compared to 0.432 for the SChuBERT (text only) model. Similar improvements are obtained on the PeerRead accept/reject prediction task. In our experiments using SciBERT, scincl, SPECTER and SPECTER2.0 embeddings, we show that each of these tailored embeddings adds further improvements over the standard BERT_textrmBASE embeddings, with the SPECTER2.0 embeddings performing best.