# Modeling Selective Feature Attention for Representation-based Siamese Text Matching

arXiv: https://arxiv.org/abs/2404.16776 · HTML: https://mlsanity.com/paper/2404.16776/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 40 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 2 of 2)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)

## Built on by (most cited first, 2 of 2)

- [Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion](https://mlsanity.com/paper/2505.17793/) (2025)
- [Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries](https://mlsanity.com/paper/2508.18212/) (2025)

## Abstract (arXiv)

Representation-based Siamese networks have risen to popularity in lightweight text matching due to their low deployment and inference costs. While word-level attention mechanisms have been implemented within Siamese networks to improve performance, we propose Feature Attention (FA), a novel downstream block designed to enrich the modeling of dependencies among embedding features. Employing "squeeze-and-excitation" techniques, the FA block dynamically adjusts the emphasis on individual features, enabling the network to concentrate more on features that significantly contribute to the final classification. Building upon FA, we introduce a dynamic "selection" mechanism called Selective Feature Attention (SFA), which leverages a stacked BiGRU Inception structure. The SFA block facilitates multi-scale semantic extraction by traversing different stacked BiGRU layers, encouraging the network to selectively concentrate on semantic information and embedding features across varying levels of abstraction. Both the FA and SFA blocks offer a seamless integration capability with various Siamese networks, showcasing a plug-and-play characteristic. Experimental evaluations conducted across diverse text matching baselines and benchmarks underscore the indispensability of modeling feature attention and the superiority of the "selection" mechanism.