# OSEF: One-Step Evidence Fusion for Cross-Video Scene Procedure Planning

arXiv: https://arxiv.org/abs/2607.29401 · HTML: https://mlsanity.com/paper/2607.29401/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 53 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos](https://mlsanity.com/paper/1804.09626/) (2018)
- [Unsupervised Learning from Narrated Instruction Videos](https://mlsanity.com/paper/1506.09215/) (2015)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [Procedure Planning in Instructional Videos](https://mlsanity.com/paper/1907.01172/) (2019)
- [Hierarchical Video-Moment Retrieval and Step-Captioning](https://mlsanity.com/paper/2303.16406/) (2023)
- [Reparameterizable Subset Sampling via Continuous Relaxations](https://mlsanity.com/paper/1901.10517/) (2019)
- [Video Corpus Moment Retrieval with Contrastive Learning](https://mlsanity.com/paper/2105.06247/) (2021)
- [Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning](https://mlsanity.com/paper/2110.01770/) (2021)
- [P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision](https://mlsanity.com/paper/2205.02300/) (2022)
- [PDPP: Projected Diffusion for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2303.14676/) (2023)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval](https://mlsanity.com/paper/2109.10016/) (2021)
- [SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.01599/) (2024)
- [Towards Efficient Partially Relevant Video Retrieval with Active Moment Discovering](https://mlsanity.com/paper/2504.10920/) (2025)
- [Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos](https://mlsanity.com/paper/2403.02782/) (2024)
- [Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learning](https://mlsanity.com/paper/2310.00608/) (2023)
- [Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos](https://mlsanity.com/paper/2409.20557/) (2024)
- [MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks](https://mlsanity.com/paper/2506.07016/) (2025)
- [Learning to Locate Visual Answer in Video Corpus Using Question](https://mlsanity.com/paper/2210.05423/) (2022)
- [VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding](https://mlsanity.com/paper/2410.08593/) (2024)
- [ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.08591/) (2024)

## Built on by (most cited first, 3 of 3)

- [RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion with Matrix-Valued Trust for Multimodal Prediction under Modality Uncertainty](https://mlsanity.com/paper/2609.10798/) (2026)
- [Can Language Models Learn to Reject Their Own Bad Reasoning Steps?](https://mlsanity.com/paper/2610.05976/) (2026)
- [What Breaks in Multi-Image Schema Application? Factorial Oracle Interventions Reveal an Instance-Information Bottleneck](https://mlsanity.com/paper/2610.19611/) (2026)

## Abstract (arXiv)

Video Scene Procedure Planning (VSPP) supplies the target start-goal observations in advance, leaving open how a planner should act when the evidence must itself be retrieved. We introduce Cross-Video Scene Procedure Planning (CVSPP): given an answer-redacted start-goal query and K candidate videos, a model must retrieve the supporting video, localize the relevant window, and predict the action sequence. Two obstacles couple here. Same-task demonstrations share stages and windows, and an early hard selection passes the wrong scene chain to the planner. We build an eleven-source benchmark with typed negative roles, a fail-closed answer-leakage gate, and separate Evidence- and Plan-axis metrics. On its 14 source-horizon cells we adapt nine planner families against a majority-sequence floor. We then present One-Step Evidence Fusion (OSEF), which scores a query-conditioned cell-and-span lattice over all candidates and feeds the full soft lattice to the planner through a token-global adapter, cropping no window beforehand. OSEF ranks first on all six cells the benchmark certifies as method-rankable. On four matched same-task COIN and CrossTask cells it improves exact-video-and-plan success by 2.9-10.7 points over an enhanced hard-selection SOTA, and a component study assigns the largest single increment to the token-global interface. Five converted-source cells sit at or near the majority-sequence floor, the benchmark's remaining headroom. The supplementary package includes model constructors and evaluation code.