# JIR-Arena: The First Benchmark Dataset for Just-in-time Information Recommendation

arXiv: https://arxiv.org/abs/2505.13550 · HTML: https://mlsanity.com/paper/2505.13550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 69 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 30 of 30)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Learning to Optimize Tensor Programs](https://mlsanity.com/paper/1805.08166/) (2018)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents](https://mlsanity.com/paper/2101.11796/) (2021)
- [Precision and Recall for Time Series](https://mlsanity.com/paper/1803.03639/) (2018)
- [The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes](https://mlsanity.com/paper/2012.14210/) (2020)
- [Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generation](https://mlsanity.com/paper/2307.09688/) (2023)
- [Learning Temporal Point Processes via Reinforcement Learning](https://mlsanity.com/paper/1811.05016/) (2018)
- [Sparse Attentive Backtracking: Temporal CreditAssignment Through Reminding](https://mlsanity.com/paper/1809.03702/) (2018)
- [Inquisitive Question Generation for High Level Text Comprehension](https://mlsanity.com/paper/2010.01657/) (2020)
- [A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks](https://mlsanity.com/paper/1802.06153/) (2018)
- [User Simulation for Evaluating Information Access Systems](https://mlsanity.com/paper/2306.08550/) (2023)
- [IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics](https://mlsanity.com/paper/2007.03953/) (2020)
- [From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing](https://mlsanity.com/paper/2410.04772/) (2024)
- [Answerer in Questioner's Mind: Information Theoretic Approach to Goal-Oriented Visual Dialog](https://mlsanity.com/paper/1802.03881/) (2018)
- [Geometry-Based Data Generation](https://mlsanity.com/paper/1802.04927/) (2018)
- [ProCIS: A Benchmark for Proactive Retrieval in Conversations](https://mlsanity.com/paper/2405.06460/) (2024)
- [Generalisation of structural knowledge in the hippocampal-entorhinal system](https://mlsanity.com/paper/1805.09042/) (2018)
- [Bayesian Nonparametric Spectral Estimation](https://mlsanity.com/paper/1809.02196/) (2018)
- [A probabilistic population code based on neural samples](https://mlsanity.com/paper/1811.09739/) (2018)
- [Human Languages in Source Code: Auto-Translation for Localized Instruction](https://mlsanity.com/paper/1909.04556/) (2019)
- [My Mouse, My Rules: Privacy Issues of Behavioral User Profiling via Mouse Tracking](https://mlsanity.com/paper/2101.09087/) (2021)
- [Defect Prediction Guided Search-Based Software Testing](https://mlsanity.com/paper/2109.12645/) (2021)
- [TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment](https://mlsanity.com/paper/2501.00522/) (2024)

## Abstract (arXiv)

Just-in-time Information Recommendation (JIR) is a service designed to deliver the most relevant information precisely when users need it, , addressing their knowledge gaps with minimal effort and boosting decision-making and efficiency in daily life. Advances in device-efficient deployment of foundation models and the growing use of intelligent wearable devices have made always-on JIR assistants feasible. However, there has been no systematic effort to formally define JIR tasks or establish evaluation frameworks. To bridge this gap, we present the first mathematical definition of JIR tasks and associated evaluation metrics. Additionally, we introduce JIR-Arena, a multimodal benchmark dataset featuring diverse, information-request-intensive scenarios to evaluate JIR systems across critical dimensions: i) accurately inferring user information needs, ii) delivering timely and relevant recommendations, and iii) avoiding irrelevant content that may distract users. Developing a JIR benchmark dataset poses challenges due to subjectivity in estimating user information needs and uncontrollable system variables affecting reproducibility. To address these, JIR-Arena: i) combines input from multiple humans and large AI models to approximate information need distributions; ii) assesses JIR quality through information retrieval outcomes using static knowledge base snapshots; and iii) employs a multi-turn, multi-entity validation framework to improve objectivity and generality. Furthermore, we implement a baseline JIR system capable of processing real-time information streams aligned with user inputs. Our evaluation of this baseline system on JIR-Arena indicates that while foundation model-based JIR systems simulate user needs with reasonable precision, they face challenges in recall and effective content retrieval. To support future research in this new area, we fully release our code and data.