mlsanityData to 14 Oct 2026

Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control

arXiv 2512.24826 ↗First posted 31 Dec 2025cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Cross-modal systems trained on 2D visual inputs are presented with a dimensional shift when processing 3D scenes. An in-scene camera bridges the dimensionality gap but requires learning a control module. We introduce a new method that improves multivariate mutual information estimates by regret minimisation with derivative-free optimisation. Our algorithm enables off-the-shelf cross-modal systems trained on 2D visual inputs to adapt online to object occlusions and differentiate features. The pairing of expressive measures and value-based optimisation assists control of an in-scene camera to learn directly from the noisy outputs of vision-language models. The resulting pipeline improves performance in cross-modal tasks on multi-object 3D scenes without resorting to pretraining or finetuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 74 references that are in the map. First posted on arXiv 31 Dec 2025.

Builds on

32 of 74 references.

References · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  6. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  7. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  8. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  9. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  10. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  11. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  12. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
20 more
  1. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  2. 2019Conditional Learning of Fair Representationscited by 78
  3. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 76
  4. 2024The Neglected Tails in Vision-Language Modelscited by 63
  5. 2024Multi-modal Situated Reasoning in 3D Scenescited by 51
  6. 2020Optimizing Molecules using Efficient Queries from Property Evaluationscited by 50
  7. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 50
  8. 2019Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Videocited by 45
  9. 2017Estimating Mutual Information for Discrete-Continuous Mixturescited by 34
  10. 2019ShapeCaptioner: Generative Caption Network for 3D Shapes by Learning a Mapping from Parts Detected in Multiple Views to Sentencescited by 30
  11. 2016Efficient multivariate entropy estimation via k-nearest neighbour distancescited by 28
  12. 2021A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decompositioncited by 26
  13. 2022Training Efficient Controllers via Analytic Policy Gradientcited by 22
  14. 2019Active Perception based Formation Control for Multiple Aerial Vehiclescited by 8
  15. 2021Vision-based system identification and 3D keypoint discovery using dynamics constraintscited by 5
  16. 2017Unsupervised Learning via Total Correlation Explanationcited by 3
  17. 2015The Information Sievecited by 1
  18. 2022Signal in Noise: Exploring Meaning Encoded in Random Character Sequences with Character-Aware Language Modelscited by 1
  19. 2023Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptionscited by 1
  20. 2023System Identification and Control of Front-Steered Ackermann Vehicles through Differentiable Physicscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
  2. 2024Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels
  3. 2026Cross-View Action Consistency for Camera-Robust Vision-Language-Action Policies
  4. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning
  5. 2025LLaVA3: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs
  6. 2025LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion
  7. 2026G3VLA: Geometric inductive bias for Vision-Language-Action Models
  8. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding