mlsanityData to 7 Oct 2026

Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation

arXiv 2307.13368 ↗First posted 25 Jul 2023cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

We introduce a novel speaker model textscKefa for navigation instruction generation. The existing speaker models in Vision-and-Language Navigation suffer from the large domain gap of vision features between different environments and insufficient temporal grounding capability. To address the challenges, we propose a Knowledge Refinement Module to enhance the feature representation with external knowledge facts, and an Adaptive Temporal Alignment method to enforce fine-grained alignment between the generated instructions and the observation sequences. Moreover, we propose a new metric SPICE-D for navigation instruction evaluation, which is aware of the correctness of direction phrases. The experimental results on R2R and UrbanWalk datasets show that the proposed KEFA speaker achieves state-of-the-art instruction generation performance for both indoor and outdoor scenes.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 40 of its 70 references that are in the map. First posted on arXiv 25 Jul 2023.

Builds on

40 of 70 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
28 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  3. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  5. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  6. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  7. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  8. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  9. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  10. 2018Visual Semantic Navigation using Scene Priorscited by 127
  11. 2019Text Generation from Knowledge Graphs with Graph Transformerscited by 122
  12. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  13. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 101
  14. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  15. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  16. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  17. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
  18. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  19. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  20. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  21. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
  22. 2016Image Captioning and Visual Question Answering Based on Attributes and External Knowledgecited by 44
  23. 2020More Grounded Image Captioning by Distilling Image-Text Matching Modelcited by 43
  24. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41
  25. 2020Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answeringcited by 32
  26. 2021Action Unit Memory Network for Weakly Supervised Temporal Action Localizationcited by 26
  27. 2022FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigationcited by 22
  28. 2019Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 3.

11 papers.

  1. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  2. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  3. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  4. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  5. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  6. 2025AeroDuo: Aerial Duo for UAV-based Vision and Language Navigationcited by 13
  7. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  8. 2025GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoningcited by 6
  9. 2026Advancing DialNav through Automatic Embodied Dialog Augmentationcited by 0
  10. 2026VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environmentscited by 0
  11. 2026Time-Aware Assistive Navigationcited by 0