mlsanityData to 7 Oct 2026

Nugget: Neural Agglomerative Embeddings of Text

arXiv 2310.01732 ↗First posted 3 Oct 2023cs.CL cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Embedding text sequences is a widespread requirement in modern language understanding. Existing approaches focus largely on constant-size representations. This is problematic, as the amount of information contained in text often varies with the length of the input. We propose a solution called Nugget, which encodes language into a representation based on a dynamically selected subset of input tokens. These nuggets are learned through tasks like autoencoding and machine translation, and intuitively segment language into meaningful units. We demonstrate Nugget outperforms related approaches in tasks involving semantic comparison. Finally, we illustrate these compact units allow for expanding the contextual window of a language model (LM), suggesting new future LMs that can condition on significantly larger amounts of content.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

35 of 49 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2018Deep contextualized word representationscited by 2,285
23 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  3. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  8. 2015Generating Sentences from a Continuous Spacecited by 607
  9. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  10. 2015Skip-Thought Vectorscited by 321
  11. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  12. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  13. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  14. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  15. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  16. 2019Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translationcited by 56
  17. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 55
  18. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  19. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  20. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  21. 2019ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translationcited by 38
  22. 2022The NLP Task Effectiveness of Long-Range Transformerscited by 22
  23. 2022A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddingscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 4; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  2. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  3. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  4. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  5. 2025Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacitycited by 27
  6. 2023Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representationscited by 19
  7. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMscited by 19
  8. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 15
  9. 2023Optimizing Retrieval-augmented Reader Models via Token Eliminationcited by 10
  10. 2023Toucan: Token-Aware Character Level Language Modelingcited by 7
  11. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLMcited by 7
  12. 2024From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compressioncited by 7
7 more
  1. 2024Generation with Dynamic Vocabularycited by 5
  2. 2025Conflict-Aware Soft Prompting for Retrieval-Augmented Generationcited by 5
  3. 2024CoLLEGe: Concept Embedding Generation for Large Language Modelscited by 3
  4. 2024Streaming Sequence Transduction through Dynamic Compressioncited by 2
  5. 2026Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compressioncited by 2
  6. 2025DeCAL Tokenwise Compressioncited by 0
  7. 2026TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Codingcited by 0