Abstract
Zero-shot skeleton-based action recognition aims to recognize unseen actions by transferring knowledge from seen categories through semantic descriptions. Most existing methods typically align skeleton features with textual embeddings within a shared latent space. However, the absence of contextual cues, such as objects involved in the action, introduces an inherent gap between skeleton and semantic representations, making it difficult to distinguish visually similar actions. To address this, we propose SkeletonContext, a prompt-based framework that enriches skeletal motion representations with language-driven contextual semantics. Specifically, we introduce a Cross-Modal Context Prompt Module, which leverages a pretrained language model to reconstruct masked contextual prompts under guidance derived from LLMs. This design effectively transfers linguistic context to the skeleton encoder for instance-level semantic grounding and improved cross-modal alignment. In addition, a Key-Part Decoupling Module is incorporated to decouple motion-relevant joint features, ensuring robust action understanding even in the absence of explicit object interactions. Extensive experiments on multiple benchmarks demonstrate that SkeletonContext achieves state-of-the-art performance under both conventional and generalized zero-shot settings, validating its effectiveness in reasoning about context and distinguishing fine-grained, visually similar actions.
Abstract (translated)
零样本骨骼动作识别旨在通过从已见类别向未见类别的知识迁移,实现对未见动作的识别。现有方法通常将骨骼特征与文本嵌入在共享潜在空间中对齐。然而,动作所涉及物体等上下文线索的缺失,导致了骨骼与语义表示之间存在固有鸿沟,使得视觉上相似的动作难以区分。针对此问题,我们提出了基于提示的框架SkeletonContext(骨架上下文),该框架利用语言驱动的上下文语义来丰富骨骼运动表示。具体而言,我们引入了跨模态上下文提示模块,该模块借助预训练语言模型,在大语言模型导出的指导下重建被掩码的上下文提示。此设计能有效将语言上下文迁移至骨骼编码器,实现实例级语义锚定并改善跨模态对齐。此外,我们还引入了关键部位解耦模块,用于解耦与运动相关的关节点特征,确保在缺乏明确物体交互的情况下仍能实现鲁棒的动作理解。在多个基准数据集上的大量实验表明,SkeletonContext在传统与广义零样本设置下均达到最先进性能,验证了其在上下文推理与区分细粒度视觉相似动作方面的有效性。
URL
https://arxiv.org/abs/2603.29692