# Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weighting

arXiv: https://arxiv.org/abs/2402.14184 · HTML: https://mlsanity.com/paper/2402.14184/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 24 of 24)

- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers](https://mlsanity.com/paper/1805.08206/) (2018)
- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [Quantifying Uncertainties in Natural Language Processing Tasks](https://mlsanity.com/paper/1811.07253/) (2018)
- [Representation Topology Divergence: A Method for Comparing Neural Network Representations](https://mlsanity.com/paper/2201.00058/) (2021)
- [Artificial Text Detection via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2109.04825/) (2021)
- [On Power Laws in Deep Ensembles](https://mlsanity.com/paper/2007.08483/) (2020)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [Disrupting Deep Uncertainty Estimation Without Harming Accuracy](https://mlsanity.com/paper/2110.13741/) (2021)
- [Acceptability Judgements via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2205.09630/) (2022)
- [Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems](https://mlsanity.com/paper/1908.05287/) (2019)
- [Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanations](https://mlsanity.com/paper/2105.06791/) (2021)
- [Exploring Transformers in Emotion Recognition: a comparison of BERT, DistillBERT, RoBERTa, XLNet and ELECTRA](https://mlsanity.com/paper/2104.02041/) (2021)
- [Ensemble approach for natural language question answering problem](https://mlsanity.com/paper/1908.09720/) (2019)
- [Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?](https://mlsanity.com/paper/2005.07292/) (2020)

## Built on by (most cited first, 1 of 1)

- [Topological Data Analysis Applications in Natural Language Processing: A Survey](https://mlsanity.com/paper/2411.10298/) (2024)

## Abstract (arXiv)

In machine learning, ensembles are important tools for improving the model performance. In natural language processing specifically, ensembles boost the performance of a method due to multiple large models available in open source. However, existing approaches mostly rely on simple averaging of predictions by ensembles with equal weights for each model, ignoring differences in the quality and conformity of models. We propose to estimate weights for ensembles of NLP models using not only knowledge of their individual performance but also their similarity to each other. By adopting distance measures based on Topological Data Analysis (TDA), we improve our ensemble. The quality improves for both text classification accuracy and relevant uncertainty estimation.