mlsanityData to 14 Oct 2026

Better Embeddings with Coupled Adam

arXiv 2502.08441 ↗First posted 12 Feb 2025cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Despite their remarkable capabilities, LLMs learn word representations that exhibit the undesirable yet poorly understood feature of anisotropy. In this paper, we argue that the second moment in Adam is a cause of anisotropic embeddings, and suggest a modified optimizer called Coupled Adam to mitigate the problem. Our experiments demonstrate that Coupled Adam significantly improves the quality of embeddings, while also leading to better upstream and downstream performance on large enough datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 36 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

19 of 36 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2016An overview of gradient descent optimization algorithmscited by 1,033
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  12. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
7 more
  1. 2016Using the Output Embedding to Improve Language Modelscited by 272
  2. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  3. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  4. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 172
  5. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 53
  6. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  7. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Output Embedding Centering for Stable LLM Pretrainingcited by 4
  2. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 3
  3. 2026Weight Tying Biases Token Embeddings Towards the Output Spacecited by 3
  4. 2026Why Adaptive Optimizers Underestimate Rare Tokenscited by 0