mlsanityData to 7 Oct 2026

Wontopos Tablet 2: Measuring Multilingual and Multimodal Memory Retrieval Without Lexical Matching

arXiv 2608.23920 ↗First posted 24 Aug 2026cs.IR cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We measure tablet-2, a production long-term memory engine for language models, on the text benchmarks the field already uses and on cross-lingual retrieval of photographs stored with no text at all. Its retrieval path contains no lexical matching, no keyword scoring, and no language model of its own. On LongMemEval-S (500 questions) it scores 95.7% [93.4, 97.1]; on BEAM-1M (700 questions, 2.21M stored memories) 67.5% [64.8, 70.2]. Those are question-sampling intervals, not the run-to-run spread, which is an order of magnitude narrower. Most of the paper is about how little they mean alone. Holding engine, corpus, settings and judge fixed, changing only the reader moves LongMemEval-S by 2.0 points; changing only the re-ask budget moves BEAM-1M by 8.9. Neither is stated in the reports we compare against, and the second exceeds most gaps there, so we give that table as a placement and not a ranking. For the multimodal axis we run two controls. Against BM25, configured as strongly as we could, we reach 95.2% mean recall@5 over 70 store-and-query language cells where BM25 reaches 19.0% and is exactly zero in 54. On captionless photographs a lexical method has no document to score at all. Open dense baselines on 300 Crossmodal-3600 photographs in 14 languages show that density confers no language independence: one scores 91.0% on English and 4.7% on Russian from identical image vectors, and a multilingual variant collapses on Telugu and Swahili. Our spread across languages is 14.0 against their 27.5 and 27.7. Three results run against us and are reported at equal weight: low-resource languages degrade sharply (Swahili 53.0%, Telugu 64.0%), attaching captions lowers cross-lingual retrieval by 11.4 points, and one setting omitted into one stage of our own retrieval cost 37 points of Korean top-1 accuracy while leaving nine languages untouched.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

29 of 41 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
17 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  2. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  3. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  4. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  5. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  6. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  7. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  8. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  9. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  10. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  11. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 160
  12. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  13. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  14. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  15. 2025Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMscited by 52
  16. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  17. 2023"Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generationcited by 12

Built on by

Not cited yet.