mlsanityData to 7 Oct 2026

Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback

arXiv 2402.04867 ↗First posted 7 Feb 2024cs.IRcited by 9

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

In the rapidly evolving landscape of information retrieval, search engines strive to provide more personalized and relevant results to users. Query suggestion systems play a crucial role in achieving this goal by assisting users in formulating effective queries. However, existing query suggestion systems mainly rely on textual inputs, potentially limiting user search experiences for querying images. In this paper, we introduce a novel Multimodal Query Suggestion (MMQS) task, which aims to generate query suggestions based on user query images to improve the intentionality and diversity of search results. We present the RL4Sugg framework, leveraging the power of Large Language Models (LLMs) with Multi-Agent Reinforcement Learning from Human Feedback to optimize the generation process. Through comprehensive experiments, we validate the effectiveness of RL4Sugg, demonstrating a 18% improvement compared to the best existing approach. Moreover, the MMQS has been transferred into real-world search engine products, which yield enhanced user engagement. Our research advances query suggestion systems and provides a new perspective on multimodal information retrieval.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 53 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

26 of 53 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
14 more
  1. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  2. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  4. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  5. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  6. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  7. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  10. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  11. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 34
  12. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  13. 2021Pre-trained Language Model for Web-scale Retrieval in Baidu Searchcited by 31
  14. 2014Query Rewriting and Optimization for Ontological Databasescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 1.

9 papers.

  1. 2024M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitionscited by 38
  2. 2025P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmarkcited by 15
  3. 2025From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational Systemcited by 9
  4. 2024CARROT: A Learned Cost-Constrained Retrieval Optimization System for RAGcited by 5
  5. 2024Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Interventioncited by 5
  6. 2024Enhancing Discoverability in Enterprise Conversational Systems with Proactive Question Suggestionscited by 2
  7. 2025AgentCTG: Harnessing Multi-Agent Collaboration for Fine-Grained Precise Control in Text Generationcited by 0
  8. 2025Paper2Web: Let's Make Your Paper Alive!cited by 0
  9. 2026Generative Query Suggestion via Intent Coverage and Query-Level Credit Assignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning
  2. 2025MMSearch-R1: Incentivizing LMMs to Search
  3. 2026Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
  4. 2025CogPlanner: Unveiling the Potential of Agentic Multimodal Retrieval Augmented Generation with Planning
  5. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval
  6. 2026SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach
  7. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
  8. 2025Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning