Abstract
Gait recognition is a crucial biometric identification technique. Camera-based gait recognition has been widely applied in both research and industrial fields. LiDAR-based gait recognition has also begun to evolve most recently, due to the provision of 3D structural information. However, in certain applications, cameras fail to recognize persons, such as in low-light environments and long-distance recognition scenarios, where LiDARs work well. On the other hand, the deployment cost and complexity of LiDAR systems limit its wider application. Therefore, it is essential to consider cross-modality gait recognition between cameras and LiDARs for a broader range of applications. In this work, we propose the first cross-modality gait recognition framework between Camera and LiDAR, namely CL-Gait. It employs a two-stream network for feature embedding of both modalities. This poses a challenging recognition task due to the inherent matching between 3D and 2D data, exhibiting significant modality discrepancy. To align the feature spaces of the two modalities, i.e., camera silhouettes and LiDAR points, we propose a contrastive pre-training strategy to mitigate modality discrepancy. To make up for the absence of paired camera-LiDAR data for pre-training, we also introduce a strategy for generating data on a large scale. This strategy utilizes monocular depth estimated from single RGB images and virtual cameras to generate pseudo point clouds for contrastive pre-training. Extensive experiments show that the cross-modality gait recognition is very challenging but still contains potential and feasibility with our proposed model and pre-training strategy. To the best of our knowledge, this is the first work to address cross-modality gait recognition.
Abstract (translated)
行走识别是一种关键的生物识别技术。基于相机的行走识别已经在研究和工业领域得到了广泛应用。最近,基于激光雷达的行走识别也开始发展,因为提供了3D结构信息。然而,在某些应用中,相机无法识别人员,例如在低光环境和远距离识别场景中,激光雷达能更好地工作。另一方面,激光雷达系统的部署成本和复杂性限制了其更广泛的应用。因此,在考虑摄像头和激光雷达之间的跨模态行走识别框架时,在更广泛的应用范围内,跨模态行走识别是至关重要的。 在这项工作中,我们提出了第一个基于摄像头和激光雷达之间的跨模态行走识别框架,即CL-Gait。它采用两个流网络对两种模态的特征进行嵌入。由于3D和2D数据之间固有的匹配,这导致识别任务具有挑战性,并表现出显著的模态差异。为了调整两种模态特征空间的对比度,我们提出了一个先验策略来减轻模态差异。为了弥补预训练中缺乏成对相机-激光雷达数据,我们还引入了一种生成大规模数据策略。该策略利用从单色图像和虚拟相机估计的单目深度来生成对比性预训练伪点云。大量实验证明,跨模态行走识别非常具有挑战性,但仍然包含我们提出的模型和预训练策略的潜力和可行性。据我们所知,这是第一个关注跨模态行走识别的工作。
URL
https://arxiv.org/abs/2407.02038