mlsanityData to 14 Oct 2026

NeoBERT: A Next-Generation BERT

arXiv 2502.19587 ↗First posted 26 Feb 2025cs.CL cs.AIcited by 16

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent innovations in architecture, pre-training, and fine-tuning have led to the remarkable in-context learning and reasoning abilities of large auto-regressive language models such as LLaMA and DeepSeek. In contrast, encoders like BERT and RoBERTa have not seen the same level of progress despite being foundational for many downstream NLP applications. To bridge this gap, we introduce NeoBERT, a next-generation encoder that redefines the capabilities of bidirectional models by integrating state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies. NeoBERT is designed for seamless adoption: it serves as a plug-and-play replacement for existing base models, relies on an optimal depth-to-width ratio, and leverages an extended context length of 4,096 tokens. Despite its compact 250M parameter footprint, it achieves state-of-the-art results on the massive MTEB benchmark, outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under identical fine-tuning conditions. In addition, we rigorously evaluate the impact of each modification on GLUE and design a uniform fine-tuning and evaluation framework for MTEB. We release all code, data, checkpoints, and training scripts to accelerate research and real-world adoption.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 48 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

38 of 48 references.

References · 38

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2020GLU Variants Improve Transformercited by 1,804
26 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2020On Layer Normalization in the Transformer Architecturecited by 922
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  10. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  11. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  13. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  14. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  15. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  16. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  17. 2023AnglE-optimized Text Embeddingscited by 99
  18. 2021GooAQ: Open Question Answering with Diverse Answer Typescited by 53
  19. 2019AmazonQA: A Review-Based Question Answering Taskcited by 43
  20. 2020The Depth-to-Width Interplay in Self-Attentioncited by 42
  21. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
  22. 2024GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuningcited by 40
  23. 2022Reasoning over Public and Private Data in Retrieval-Based Systemscited by 21
  24. 2024Contextual Document Embeddingscited by 19
  25. 2019Episodic Memory Reader: Learning What to Remember for Question Answering from Streaming Datacited by 14
  26. 2024Enhancing Training Efficiency Using Packing with Flash Attentioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 45
  2. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 11
  3. 2025Reasoning-Intensive Regressioncited by 2
  4. 2026Decoupling Vision, Language, and Action for Efficient Multi-Task Robot Policiescited by 2
  5. 2025TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkishcited by 1
  6. 2026Long-Context Encoder Models for Polish Language Understandingcited by 1
  7. 2026Separating Representation from Reconstruction Enables Scalable Text Encoderscited by 1
  8. 2025ABCDWaveNet: Advancing Robust Road Ponding Detection in Fog through Dynamic Frequency-Spatial Synergycited by 0
  9. 2025Chinese ModernBERT with Whole-Word Maskingcited by 0
  10. 2025ProtoSiTex: Learning Semi-Interpretable Prototypes for Multi-label Text Classificationcited by 0
  11. 2026Large-Scale Aspect-Based Sentiment Analysis with Reasoning-Infused LLMscited by 0
  12. 2026Avey-Bcited by 0
4 more
  1. 2026Is She Even Relevant? When BERT Ignores Explicit Gender Cuescited by 0
  2. 2026moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERTcited by 0
  3. 2026SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metricscited by 0
  4. 2026UltraBERT: Beyond the Limit of BERT Pre-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021RefBERT: Compressing BERT by Referencing to Pre-computed Representations
  2. 2021RomeBERT: Robust Training of Multi-Exit BERT
  3. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
  4. 2021Elbert: Fast Albert with Confidence-Window Based Early Exit
  5. 2022MKQ-BERT: Quantized BERT with 4-bits Weights and Activations
  6. 2022BEBERT: Efficient and Robust Binary Ensemble BERT
  7. 2022BiBERT: Accurate Fully Binarized BERT
  8. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baseline