mlsanityData to 7 Oct 2026

DarkQA: Benchmarking Vision-Language Models on Visual-Primitive Question Answering in Low-Light Indoor Scenes

arXiv 2512.24985 ↗First posted 31 Dec 2025cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision Language Models (VLMs) are increasingly adopted as central reasoning modules for embodied agents. Existing benchmarks evaluate their capabilities under ideal, well-lit conditions, yet robust 24/7 operation demands performance under a wide range of visual degradations, including low-light conditions at night or in dark environments, a core necessity that has been largely overlooked. To address this underexplored challenge, we present DarkQA, an open-source benchmark for evaluating perceptual primitives under multi-level low-light conditions in embodied scenarios. DarkQA evaluates single-view egocentric observations across controlled degradation levels, isolating low-light perceptual failures before they are entangled with complex embodied tasks. The benchmark contains 9.4K deterministically generated and verifiable question-image pairs spanning five visual-primitive families. A key design feature of DarkQA is its physical fidelity: visual degradations are modeled in linear RAW space, simulating physics-based illumination drop and sensor noise followed by an ISP-inspired rendering pipeline; we further validate the synthesis against real paired low-light camera data. We evaluate representative VLMs and Low-Light Image Enhancement (LLIE) preprocessing methods. Results show consistent VLM degradation under low illumination and sensor noise, while LLIE provides severity-dependent but unstable recovery. We demonstrate the utility of DarkQA by evaluating a wide range of state-of-the-art VLMs and Low-Light Image Enhancement (LLIE) models, and systematically reveal VLMs' limitations when operating under these challenging visual conditions. Our code and benchmark dataset will be released upon acceptance. Project website: https://darkqa-benchmark.github.io

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 39 references that are in the map. First posted on arXiv 31 Dec 2025.

Builds on

32 of 39 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  8. 2025Qwen3-VL Technical Reportcited by 2,534
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  12. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
20 more
  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  2. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  3. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  4. 2020Zero-Reference Deep Curve Estimation for Low-Light Image Enhancementcited by 541
  5. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  6. 2017Embodied Question Answeringcited by 335
  7. 2020Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancementcited by 303
  8. 2018Unprocessing Images for Learned Raw Denoisingcited by 250
  9. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  10. 2023Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancementcited by 228
  11. 2017Benchmarking Denoising Algorithms with Real Photographscited by 209
  12. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  13. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  14. 2021Physics-based Noise Modeling for Extreme Low-light Photographycited by 67
  15. 2021Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Darkcited by 57
  16. 2024DarkIR: Robust Low-Light Image Restorationcited by 48
  17. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
  18. 2022Episodic Memory Question Answeringcited by 32
  19. 2023Human Pose Estimation in Extremely Low-Light Conditionscited by 27
  20. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
  2. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  3. 2024Q-Bench+: A Benchmark for Multi-modal Foundation Models on Low-level Vision from Single Images to Pairs
  4. 2026Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models
  5. 2026StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models
  6. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  7. 2025Analyzing the Sensitivity of Vision Language Models in Visual Question Answering
  8. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding