mlsanityData to 7 Oct 2026

A Structured Self-attentive Sentence Embedding

arXiv 1703.03130 ↗First posted 9 Mar 2017cs.CL cs.AI cs.LGcited by 318

Not on the map. Why?

Abstract (arXiv)

This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification, and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 112; 2022: 79; 2023: 38; 2024: 41; 2025: 33; 2026: 15.

318 papers.

Sort:

By area:

  1. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  2. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  3. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  4. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  5. 2021Dive into Deep Learningcited by 111
  6. 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
  7. 2022Learning to Generate Realistic LiDAR Point Cloudscited by 91
  8. 2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 87
  9. 2021Few-Shot Graph Learning for Molecular Property Predictioncited by 81
  10. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  11. 2021FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoderscited by 75
  12. 2021Is Attention Better Than Matrix Decomposition?cited by 75
38 more
  1. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  2. 2023Transformers as Support Vector Machinescited by 67
  3. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  4. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  5. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  6. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  7. 2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52
  8. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  9. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  10. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  11. 2022Class-Incremental Learning for Action Recognition in Videoscited by 46
  12. 20223D Siamese Transformer Network for Single Object Tracking on Point Cloudscited by 44
  13. 2021PVT: Point-Voxel Transformer for Point Cloud Learningcited by 42
  14. 2022SemAttack: Natural Textual Attacks via Different Semantic Spacescited by 42
  15. 2022Simplicial Attention Networkscited by 40
  16. 2024Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slicescited by 40
  17. 2021Sparse-Interest Network for Sequential Recommendationcited by 37
  18. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  19. 2021Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Trackingcited by 35
  20. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  21. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  22. 2021Explainable AI: current status and future directionscited by 32
  23. 2021Deep Learning-based Action Detection in Untrimmed Videos: A Surveycited by 32
  24. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 31
  25. 2022Unsupervised Temporal Video Grounding with Deep Semantic Clusteringcited by 30
  26. 2022Probabilistic Compositional Embeddings for Multimodal Image Retrievalcited by 29
  27. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  28. 2022Fine-Grained Distillation for Long Document Retrievalcited by 29
  29. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  30. 2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
  31. 2023Generalized Simplicial Attention Neural Networkscited by 27
  32. 2021Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answeringcited by 26
  33. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  34. 2022A Unified Review of Deep Learning for Automated Medical Codingcited by 26
  35. 2021Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognitioncited by 25
  36. 2021Dense Interaction Learning for Video-based Person Re-identificationcited by 24
  37. 2021Self-Supervised Euphemism Detection and Identification for Content Moderationcited by 24
  38. 2021SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Miningcited by 24

50 most cited of 318 All 318 (JSON)