mlsanityData to 14 Oct 2026

JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups

arXiv 2404.04458 ↗First posted 6 Apr 2024cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Understanding human social behaviour is crucial in computer vision and robotics. Micro-level observations like individual actions fall short, necessitating a comprehensive approach that considers individual behaviour, intra-group dynamics, and social group levels for a thorough understanding. To address dataset limitations, this paper introduces JRDB-Social, an extension of JRDB. Designed to fill gaps in human understanding across diverse indoor and outdoor social contexts, JRDB-Social provides annotations at three levels: individual attributes, intra-group interactions, and social group context. This dataset aims to enhance our grasp of human social dynamics for robotic applications. Utilizing the recent cutting-edge multi-modal large language models, we evaluated our benchmark to explore their capacity to decipher social human behaviour.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 6 Apr 2024.

Builds on

35 of 53 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  9. 2017Dense-Captioning Events in Videoscited by 793
  10. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
23 more
  1. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  4. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  5. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  6. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  7. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  8. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  9. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  10. 2017MovieGraphs: Towards Understanding Human-Centric Situations from Videoscited by 67
  11. 2020PANDA: A Gigapixel-level Human-centric Video Datasetcited by 45
  12. 2020Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videoscited by 39
  13. 2021JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detectioncited by 39
  14. 2018ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wildcited by 34
  15. 2015SALSA: A Novel Dataset for Multimodal Group Behavior Analysiscited by 27
  16. 2022JRDB-Pose: A Large-scale Dataset for Multi-Person Pose Estimation and Trackingcited by 25
  17. 2020Learning to Discretely Compose Reasoning Module Networks for Video Captioningcited by 20
  18. 2021Who's Waldo? Linking People Across Text and Imagescited by 16
  19. 2015Describing Common Human Visual Actions in Imagescited by 12
  20. 2021Human Interaction Recognition Framework based on Interacting Body Part Attentioncited by 11
  21. 2015Facial Descriptors for Human Interaction Recognition In Still Imagescited by 7
  22. 2020I can attend a meeting too! Towards a human-like telepresence avatar robot to attend meeting on your behalfcited by 3
  23. 2022Detecting Human-to-Human-or-Object (H2O) Interactions with DIABOLOcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025SocialGesture: Delving into Multi-person Gesture Understandingcited by 11
  2. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  3. 2024CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognitioncited by 3
  4. 2025MINGLE: VLMs for Semantically Complex Region Detection in Urban Scenescited by 3
  5. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  6. 2024MRAC Track 1: 2nd Workshop on Multimodal, Generative and Responsible Affective Computingcited by 1
  7. 2026JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Roboticscited by 1
  8. 2026EmoScene: A Dual-space Dataset for Controllable Affective Image Generationcited by 1
  9. 2025Whom to Respond To? A Transformer-Based Model for Multi-Party Social Robot Interactioncited by 0
  10. 2026Group-DINOmics: Incorporating People Dynamics into DINO for Self-supervised Group Activity Feature Learningcited by 0
  11. 2026STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognitioncited by 0
  12. 2026Gaze-Anchored Social Net: Decoding Implicit Relations via Joint Modelingcited by 0
1 more
  1. 2026De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shiftcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Social Human Robot Embodied Conversation (SHREC) Dataset: Benchmarking Foundational Models' Social Reasoning
  2. 2026Social Graph Mamba: Forecasting Pedestrian Movements Based on Social Context
  3. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning
  4. 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces
  5. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
  6. 2024Towards Social AI: A Survey on Understanding Social Interactions
  7. 2026Adaptive Companionship for Group-Following Robots: Handling Dynamically Changing Group Formations
  8. 2025Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understanding