mlsanityData to 7 Oct 2026

Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

arXiv 2412.13663 ↗First posted 18 Dec 2024cs.CL cs.AIcited by 527

Not on the map. Why?

Cited by 29 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.

Key facts. Cited by 527 papers in the map (data through 7 Oct 2026). Builds on 76 of its 100 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

76 of 100 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  10. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  13. 2024Qwen2 Technical Reportcited by 2,076
  14. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  15. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  16. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  17. 2020GLU Variants Improve Transformercited by 1,724
  18. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  19. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  20. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  21. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  22. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  23. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  24. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  25. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  26. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  27. 2020On Layer Normalization in the Transformer Architecturecited by 906
  28. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  29. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  30. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  31. 2016Language Modeling with Gated Convolutional Networkscited by 688
  32. 2024A Survey on Large Language Models for Code Generationcited by 646
  33. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  34. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  35. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  36. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  37. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  38. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 256; 2026: 270.

527 papers.

Sort:

By area:

  1. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  2. 2025REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjectscited by 70
  3. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 68
  4. 2025MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutorscited by 47
  5. 2025LLM Agents Making Agent Toolscited by 45
  6. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 39
  7. 2025ThinkSwitcher: When to Think Hard, When to Think Fastcited by 37
  8. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
  9. 2025LettuceDetect: A Hallucination Detection Framework for RAG Applicationscited by 32
  10. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 28
  11. 2025Clinical ModernBERT: An efficient and long context encoder for biomedical textcited by 26
  12. 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 26
38 more
  1. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 24
  2. 2025Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systemscited by 24
  3. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspectivecited by 23
  4. 2025SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Predictioncited by 21
  5. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 21
  6. 2024GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Searchcited by 20
  7. 2025Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?cited by 20
  8. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  9. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 20
  10. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  11. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  12. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  13. 2025Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIcited by 19
  14. 2025Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMscited by 17
  15. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  16. 2025Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Modelscited by 16
  17. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  18. 2025Should We Still Pretrain Encoders with Masked Language Modeling?cited by 15
  19. 2025Machine-generated text detection prevents language model collapsecited by 14
  20. 2025NeoBERT: A Next-Generation BERTcited by 14
  21. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Modelscited by 13
  22. 2025When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQAcited by 13
  23. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Languagecited by 13
  24. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 13
  25. 2025AdaSplash: Adaptive Sparse Flash Attentioncited by 12
  26. 2025When to Reason: Semantic Router for vLLMcited by 12
  27. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
  28. 2026dLLM: Simple Diffusion Language Modelingcited by 12
  29. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  30. 2025ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performancecited by 11
  31. 2025Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generationcited by 11
  32. 2026Shaping capabilities with token-level data filteringcited by 11
  33. 2025Large language models as uncertainty-calibrated optimizers for experimental discoverycited by 10
  34. 2025Out of Style: RAG's Fragility to Linguistic Variationcited by 10
  35. 2025CSPLADE: Learned Sparse Retrieval with Causal Language Modelscited by 10
  36. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  37. 2025Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authorscited by 10
  38. 2025The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer trainingcited by 9

50 most cited of 527 All 527 (JSON)