Abstract
Understanding dynamic 3D human representation has become increasingly critical in virtual and extended reality applications. However, existing human part segmentation methods are constrained by reliance on closed-set datasets and prolonged inference times, which significantly restrict their applicability. In this paper, we introduce the first 4D human parsing framework that simultaneously addresses these challenges by reducing the inference time and introducing open-vocabulary capabilities. Building upon state-of-the-art open-vocabulary 3D human parsing techniques, our approach extends the support to 4D human-centric video with three key innovations: 1) We adopt mask-based video object tracking to efficiently establish spatial and temporal correspondences, avoiding the necessity of segmenting all frames. 2) A novel Mask Validation module is designed to manage new target identification and mitigate tracking failures. 3) We propose a 4D Mask Fusion module, integrating memory-conditioned attention and logits equalization for robust embedding fusion. Extensive experiments demonstrate the effectiveness and flexibility of the proposed method on 4D human-centric parsing tasks, achieving up to 93.3% acceleration compared to the previous state-of-the-art method, which was limited to parsing fixed classes.
Abstract (translated)
理解动态三维人体表示在虚拟现实和扩展现实应用中变得越来越重要。然而，现有的人体部位分割方法由于依赖于封闭数据集以及较长的推理时间而受到限制，这大大限制了它们的应用范围。本文介绍了第一个4D人体解析框架，该框架通过减少推理时间和引入开放词汇能力同时解决了这些挑战。 我们的方法基于最先进的开放词汇3D人体解析技术，并在三个方面进行了创新以支持4D以人为中心的视频： 1. 我们采用基于掩码的视频对象跟踪来高效地建立空间和时间上的对应关系，避免了需要对所有帧进行分割的需求。 2. 设计了一个新颖的Mask验证模块，用于管理新目标识别并减轻追踪失败的问题。 3. 提出了一种4D掩码融合模块，该模块结合记忆条件注意力机制和逻辑值平衡化以实现稳健的嵌入融合。 广泛的实验展示了所提出方法在4D人体为中心解析任务中的有效性和灵活性，在与以前最先进方法相比时实现了高达93.3%的速度提升。此前最先进的方法仅限于对固定类别的解析。
URL
https://arxiv.org/abs/2507.09880
