mlsanityData to 14 Oct 2026

FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions

arXiv 2509.17177 ↗First posted 21 Sep 2025cs.CL cs.CV cs.LGcited by 1

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We conduct a moderate-scale contamination-free (to some extent) evaluation of current large reasoning models (LRMs) with some preliminary findings. We also release ROME, our evaluation benchmark for vision language models intended to test reasoning from visual clues. We attach links to the benchmark, evaluation data, and other updates on this website: https://flageval-baai.github.io/LRM-Eval/

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 81 references that are in the map. First posted on arXiv 21 Sep 2025.

Builds on

36 of 81 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers30 Jun
  2. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models29 May
  3. 2025RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs22 May
  4. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models19 May
  5. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning19 Feb
  6. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts26 Dec

Other references · 30

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2024OpenAI o1 System Cardcited by 1,244
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  4. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  5. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  6. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  7. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  8. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  9. 2024Measuring short-form factuality in large language modelscited by 373
  10. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  11. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  12. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
18 more
  1. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  2. 2025Why Language Models Hallucinatecited by 220
  3. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  4. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  5. 2025MindCube: Spatial Mental Modeling from Limited Viewscited by 95
  6. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 92
  7. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 72
  8. 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMscited by 53
  9. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 52
  10. 2025Inverse Scaling in Test-Time Computecited by 50
  11. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 44
  12. 2025The Hallucination Tax of Reinforcement Finetuningcited by 40
  13. 2025Understanding Tool-Integrated Reasoningcited by 38
  14. 2025AI Sees Your Location, But With A Bias Toward The Wealthy Worldcited by 14
  15. 2025Can Language Models Follow Multiple Turns of Entangled Instructions?cited by 14
  16. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  17. 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Modelscited by 8
  18. 2025Humans Perceive Wrong Narratives from AI Reasoning Textscited by 6

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBenchcited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?
  2. 2024A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models
  3. 2025mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models
  4. 2025Visual Reasoning Evaluation of Grok, Deepseek Janus, Gemini, Qwen, Mistral, and ChatGPT
  5. 2024Trust but Verify: Programmatic VLM Evaluation in the Wild
  6. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
  7. 2025VER-Bench: Evaluating MLLMs on Reasoning with Fine-Grained Visual Evidence
  8. 2023GLoRE: Evaluating Logical Reasoning of Large Language Models