Abstract
In recent years, there has been a growing interest in developing effective alignment pipelines to generate unified representations from different modalities for multi-modal fusion and generation. As an important component of Human-Centric applications, Human Pose representations are critical in many downstream tasks, such as Human Pose Estimation, Action Recognition, Human-Computer Interaction, Object tracking, etc. Human Pose representations or embeddings can be extracted from images, 2D keypoints, 3D skeletons, mesh models, and lots of other modalities. Yet, there are limited instances where the correlation among all of those representations has been clearly researched using a contrastive paradigm. In this paper, we propose UniHPR, a unified Human Pose Representation learning pipeline, which aligns Human Pose embeddings from images, 2D and 3D human poses. To align more than two data representations at the same time, we propose a novel singular value-based contrastive learning loss, which better aligns different modalities and further boosts performance. To evaluate the effectiveness of the aligned representation, we choose 2D and 3D Human Pose Estimation (HPE) as our evaluation tasks. In our evaluation, with a simple 3D human pose decoder, UniHPR achieves remarkable performance metrics: MPJPE 49.9mm on the Human3.6M dataset and PA-MPJPE 51.6mm on the 3DPW dataset with cross-domain evaluation. Meanwhile, we are able to achieve 2D and 3D pose retrieval with our unified human pose representations in Human3.6M dataset, where the retrieval error is 9.24mm in MPJPE.
Abstract (translated)
近年来，人们对开发有效的对齐管道产生了越来越浓厚的兴趣，旨在从不同模态生成统一的表示形式，以进行多模态融合和生成。作为以人为中心的应用程序的重要组成部分，人体姿态表示在许多下游任务中至关重要，例如人体姿态估计、动作识别、人机交互、目标跟踪等。人体姿态表示或嵌入可以从图像、二维关键点、三维骨架、网格模型以及其他众多模态中提取出来。然而，在使用对比学习范式明确研究这些表示之间的相关性方面，案例相对有限。 在本文中，我们提出了UniHPR，一种统一的人体姿态表示学习管道，它对齐来自图像、二维和三维人体姿势的嵌入式表示。为了同时对齐超过两种数据表示形式，我们提出了一种基于奇异值的对比学习损失函数，这种方法能够更好地将不同模态进行对齐，并进一步提升性能表现。 为了评估对齐后的表示的有效性，我们将2D和3D人体姿态估计（HPE）作为评价任务。在我们的测试中，通过使用简单的三维人体姿势解码器，UniHPR取得了显著的性能指标：在Human3.6M数据集上实现了49.9毫米的MPJPE，在3DPW数据集中实现了51.6毫米的PA-MPJPE（跨域评估）。同时，我们能够在Human3.6M数据集上利用统一的人体姿态表示进行2D和3D姿势检索，并且在MPJPE指标下，检索误差仅为9.24毫米。
URL
https://arxiv.org/abs/2510.19078
