mlsanityData to 7 Oct 2026

Multimodal Evaluation of Russian-language Architectures

arXiv 2511.15552 ↗First posted 19 Nov 2025cs.CL cs.AI cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce MERA Multi, an open multimodal evaluation framework for Russian-spoken architectures. The benchmark is instruction-based and encompasses default text, image, audio, and video modalities, comprising 18 newly constructed evaluation tasks for both general-purpose models and modality-specific architectures (imageto-text, video-to-text, and audio-to-text). Our contributions include: (i) a universal taxonomy of multimodal abilities; (ii) 18 datasets created entirely from scratch with attention to Russian cultural and linguistic specificity, unified prompts, and metrics; (iii) baseline results for both closed-source and open-source models; (iv) a methodology for preventing benchmark leakage, including watermarking for private sets. While our current focus is on Russian, the proposed benchmark provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic language family.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 86 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

54 of 86 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025On Path to Multimodal Generalist: General-Level and General-Bench7 May
  2. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs3 Mar
  3. 2024Core Knowledge Deficits in Multi-Modal Language Models6 Oct
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 47

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
35 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  7. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  10. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  11. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  12. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  13. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  14. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  15. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  16. 2017PoseTrack: A Benchmark for Human Pose Estimation and Trackingcited by 173
  17. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  18. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  19. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  20. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  21. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  22. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 67
  23. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  24. 2021From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understandingcited by 43
  25. 2022HaGRID - HAnd Gesture Recognition Image Datasetcited by 43
  26. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 35
  27. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  28. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  29. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  30. 2024Semantic Membership Inference Attack against Large Language Modelscited by 17
  31. 2025Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answeringcited by 14
  32. 2022TAPE: Assessing Few-shot Russian Language Understandingcited by 11
  33. 2020RWCP-SSD-Onomatopoeia: Onomatopoeic Word Dataset for Environmental Sound Synthesiscited by 2
  34. 2022Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP modelscited by 2
  35. 2025Don't Fight Hallucinations, Use Them: Estimating Image Realism using NLI over Atomic Factscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025FiMMIA: scaling semantic perturbation-based membership inference across modalitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026What We are Missing in Multimodal LLM Evaluation?
  2. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  3. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  6. 2023Multimodal Large Language Models: A Survey
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning