# SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark

arXiv: https://arxiv.org/abs/2110.10661 · HTML: https://mlsanity.com/paper/2110.10661/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 65 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020)
- [Grounded Language Learning in a Simulated 3D World](https://mlsanity.com/paper/1706.06551/) (2017)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text](https://mlsanity.com/paper/2005.09382/) (2020)
- [Unified Pragmatic Models for Generating and Following Instructions](https://mlsanity.com/paper/1711.04987/) (2017)
- [Grounded Language Learning Fast and Slow](https://mlsanity.com/paper/2009.01719/) (2020)

## Built on by (most cited first, 11 of 11)

- [Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals](https://mlsanity.com/paper/2302.04449/) (2023)
- [NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation](https://mlsanity.com/paper/2411.08579/) (2024)
- [Policy Improvement using Language Feedback Models](https://mlsanity.com/paper/2402.07876/) (2024)
- [Language-Guided World Models: A Model-Based Approach to AI Control](https://mlsanity.com/paper/2402.01695/) (2024)
- [SPRING: Studying the Paper and Reasoning to Play Games](https://mlsanity.com/paper/2305.15486/) (2023)
- [AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback](https://mlsanity.com/paper/2309.17176/) (2023)
- [Improving Policy Learning via Language Dynamics Distillation](https://mlsanity.com/paper/2210.00066/) (2022)
- [Conceptual Reinforcement Learning for Language-Conditioned Tasks](https://mlsanity.com/paper/2303.05069/) (2023)
- [AI Finds A Way](https://mlsanity.com/paper/2608.23875/) (2026)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments](https://mlsanity.com/paper/2407.09287/) (2024)

## Abstract (arXiv)

Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection of diverse grounded language learning environments under a common interface. SILG consists of grid-world environments that require generalization to new dynamics, entities, and partially observed worlds (RTFM, Messenger, NetHack), as well as symbolic counterparts of visual worlds that require interpreting rich natural language with respect to complex scenes (ALFWorld, Touchdown). Together, these environments provide diverse grounding challenges in richness of observation space, action space, language specification, and plan complexity. In addition, we propose the first shared model architecture for RL on these environments, and evaluate recent advances such as egocentric local convolution, recurrent state-tracking, entity-centric attention, and pretrained LM using SILG. Our shared architecture achieves comparable performance to environment-specific architectures. Moreover, we find that many recent modelling advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for a multi-environment benchmark. Finally, the best models significantly underperform humans on SILG, which suggests ample room for future work. We hope SILG enables the community to quickly identify new methodologies for language grounding that generalize to a diverse set of environments and their associated challenges.