Paper Reading AI Learner

MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation

2025-12-11 18:59:44
Henghui Ding, Chang Liu, Shuting He, Kaining Ying, Xudong Jiang, Chen Change Loy, Yu-Gang Jiang

Abstract

This paper proposes a large-scale multi-modal dataset for referring motion expression video segmentation, focusing on segmenting and tracking target objects in videos based on language description of objects' motions. Existing referring video segmentation datasets often focus on salient objects and use language expressions rich in static attributes, potentially allowing the target object to be identified in a single frame. Such datasets underemphasize the role of motion in both videos and languages. To explore the feasibility of using motion expressions and motion reasoning clues for pixel-level video understanding, we introduce MeViS, a dataset containing 33,072 human-annotated motion expressions in both text and audio, covering 8,171 objects in 2,006 videos of complex scenarios. We benchmark 15 existing methods across 4 tasks supported by MeViS, including 6 referring video object segmentation (RVOS) methods, 3 audio-guided video object segmentation (AVOS) methods, 2 referring multi-object tracking (RMOT) methods, and 4 video captioning methods for the newly introduced referring motion expression generation (RMEG) task. The results demonstrate weaknesses and limitations of existing methods in addressing motion expression-guided video understanding. We further analyze the challenges and propose an approach LMPM++ for RVOS/AVOS/RMOT that achieves new state-of-the-art results. Our dataset provides a platform that facilitates the development of motion expression-guided video understanding algorithms in complex video scenes. The proposed MeViS dataset and the method's source code are publicly available at this https URL

Abstract (translated)

这篇论文提出了一种大规模的多模态数据集,用于根据对象运动的语言描述对视频进行参考性动作表达分割。该数据集着重于依据语言描述中的物体运动来分割和追踪视频中的目标对象。现有的参考视频分割数据集往往关注显眼的对象,并使用富含静态属性的语言表达方式,这使得在单帧图像中识别出目标对象成为可能。然而,这些数据集低估了动作在视频及语言理解中的作用。 为了探索利用动作表达和运动推理线索进行像素级视频理解的可能性,我们引入了一个名为MeViS的数据集,它包含了33,072个人工注释的动作表达文本与音频,涵盖了复杂场景中2,006个视频里的8,171个物体。我们在MeViS支持的4项任务上基准测试了15种现有方法,包括6种参考视频对象分割(RVOS)方法、3种基于语音引导的视频对象分割(AVOS)方法、2种参考多目标跟踪(RMOT)方法以及针对新提出的参考动作表达生成(RMEG)任务的4种视频描述方法。测试结果揭示了现有方法在处理由动作表达指导的视频理解时所存在的弱点和局限性。 我们进一步分析了面临的挑战,并提出了一种名为LMPM++的新方法,该方法适用于RVOS、AVOS及RMOT,在这些领域实现了新的最佳性能。我们的数据集为复杂场景下基于动作表达的视频理解算法的发展提供了一个平台。MeViS数据集和所提方法的源代码已在以下网址公开发布:[URL]

URL

https://arxiv.org/abs/2512.10945

PDF

https://arxiv.org/pdf/2512.10945.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot