mlsanityData to 14 Oct 2026

LMEB: Long-horizon Memory Embedding Benchmark

arXiv 2603.12572 ↗First posted 13 Mar 2026cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Embedding models are increasingly used as retrievers in memory-augmented systems, but are typically evaluated on traditional passage retrieval benchmarks. It remains unclear whether models that excel at passage retrieval can retrieve fragmented, context-dependent, and temporally distant memories. To address this gap, we introduce the Long-horizon Memory Embedding Benchmark (LMEB), comprising 22 English datasets and 193 zero-shot retrieval tasks across four memory types: episodic, dialogue, semantic, and procedural. We benchmark 30 embedding models ranging from 33M to 12B parameters. We find that no single model dominates across memory types and that evaluation scores on LMEB correlate only weakly with those on the MTEB (English, v2) retrieval subset. These findings underscore the need for benchmarks tailored to long-horizon memory retrieval. LMEB comes with a versatile, open-source evaluation framework and a public leaderboard to guide model selection and advance research on memory embeddings.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 66 of its 84 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

66 of 84 references.

References · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  6. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  8. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  10. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  11. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  12. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
38 more
  1. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  2. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  3. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  4. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  5. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  6. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  7. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  8. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
  9. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  10. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  11. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
  12. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  13. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  14. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  15. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  16. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  17. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  18. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  19. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
  20. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  21. 2022Task-aware Retrieval with Instructionscited by 107
  22. 2025Memp: Exploring Agent Procedural Memorycited by 103
  23. 2025MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agentscited by 84
  24. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 76
  25. 2025Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolutioncited by 73
  26. 2025SWE-Exp: Experience-Driven Software Issue Resolutioncited by 65
  27. 2025ViDoRe Benchmark V2: Raising the Bar for Visual Retrievalcited by 61
  28. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  29. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 47
  30. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 46
  31. 2025KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Modelcited by 36
  32. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  33. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  34. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 36
  35. 2025REALTALK: A 21-Day Real-World Dataset for Long-Term Conversationcited by 35
  36. 2024PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answeringcited by 34
  37. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 32
  38. 2025PeerQA: A Scientific Question Answering Dataset from Peer Reviewscited by 22

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleaguescited by 23
  2. 2026ToolOmni: Enabling Open-World Tool Use via Agentic learning with Proactive Retrieval and Grounded Executioncited by 7
  3. 2026Memory Beyond Recall: A Dual-Process Cognitive Memory System for Self-Evolving LLM Agentscited by 2
  4. 2026EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memorycited by 2
  5. 2026MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarioscited by 0
  6. 2026Lychee-Reranker: Fast but Not Late Interaction for Compressed Document Rerankingcited by 0
  7. 2026Beyond the Final Embedding: Retrieving Memories from Token-Level Evidencecited by 0