# Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective

arXiv: https://arxiv.org/abs/2411.14654 · HTML: https://mlsanity.com/paper/2411.14654/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Enhancing Link Prediction with Fuzzy Graph Attention Networks and Dynamic Negative Sampling](https://mlsanity.com/paper/2411.07482/) (2024)
- [SHAPNN: Shapley Value Regularized Tabular Neural Network](https://mlsanity.com/paper/2309.08799/) (2023)

## Built on by (most cited first, 12 of 12)

- [An overview of model uncertainty and variability in LLM-based sentiment analysis. Challenges, mitigation strategies and the role of explainability](https://mlsanity.com/paper/2504.04462/) (2025)
- [Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models](https://mlsanity.com/paper/2510.03339/) (2025)
- [Controlling Repetition in Protein Language Models](https://mlsanity.com/paper/2602.00782/) (2026)
- [Multi-view Fuzzy Graph Attention Networks for Enhanced Graph Learning](https://mlsanity.com/paper/2412.17271/) (2024)
- [Towards Efficient LLM-aware Heterogeneous Graph Learning](https://mlsanity.com/paper/2511.17923/) (2025)
- [FGATT: A Robust Framework for Wireless Data Imputation Using Fuzzy Graph Attention Networks and Transformer Encoders](https://mlsanity.com/paper/2412.01979/) (2024)
- [Text Embedded Swin-UMamba for DeepLesion Segmentation](https://mlsanity.com/paper/2508.06453/) (2025)
- [Towards Improved Sentence Representations using Token Graphs](https://mlsanity.com/paper/2603.03389/) (2026)
- [On the Persistent Effects of Lexicality in Large Language Models](https://mlsanity.com/paper/2606.02750/) (2026)
- [Looking around you: external information enhances representations for event sequences](https://mlsanity.com/paper/2502.10205/) (2025)
- [Mechanistic Decomposition of Sentence Representations](https://mlsanity.com/paper/2506.04373/) (2025)
- [Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation](https://mlsanity.com/paper/2606.07057/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have revolutionized natural language processing (NLP) by delivering state-of-the-art performance across a variety of tasks. Among these, Transformer-based models like BERT and GPT rely on pooling layers to aggregate token-level embeddings into sentence-level representations. Common pooling mechanisms such as Mean, Max, and Weighted Sum play a pivotal role in this aggregation process. Despite their widespread use, the comparative performance of these strategies on different LLM architectures remains underexplored. To address this gap, this paper investigates the effects of these pooling mechanisms on two prominent LLM families -- BERT and GPT, in the context of sentence-level sentiment analysis. Comprehensive experiments reveal that each pooling mechanism exhibits unique strengths and weaknesses depending on the task's specific requirements. Our findings underline the importance of selecting pooling methods tailored to the demands of particular applications, prompting a re-evaluation of common assumptions regarding pooling operations. By offering actionable insights, this study contributes to the optimization of LLM-based models for downstream tasks.