Paper Reading AI Learner

Camera-LiDAR Cross-modality Gait Recognition

2024-07-03 01:50:48
Wenxuan Guo, Yingping Liang, Zhiyu Pan, Ziheng Xi, Jianjiang Feng, Jie Zhou

Abstract

Gait recognition is a crucial biometric identification technique. Camera-based gait recognition has been widely applied in both research and industrial fields. LiDAR-based gait recognition has also begun to evolve most recently, due to the provision of 3D structural information. However, in certain applications, cameras fail to recognize persons, such as in low-light environments and long-distance recognition scenarios, where LiDARs work well. On the other hand, the deployment cost and complexity of LiDAR systems limit its wider application. Therefore, it is essential to consider cross-modality gait recognition between cameras and LiDARs for a broader range of applications. In this work, we propose the first cross-modality gait recognition framework between Camera and LiDAR, namely CL-Gait. It employs a two-stream network for feature embedding of both modalities. This poses a challenging recognition task due to the inherent matching between 3D and 2D data, exhibiting significant modality discrepancy. To align the feature spaces of the two modalities, i.e., camera silhouettes and LiDAR points, we propose a contrastive pre-training strategy to mitigate modality discrepancy. To make up for the absence of paired camera-LiDAR data for pre-training, we also introduce a strategy for generating data on a large scale. This strategy utilizes monocular depth estimated from single RGB images and virtual cameras to generate pseudo point clouds for contrastive pre-training. Extensive experiments show that the cross-modality gait recognition is very challenging but still contains potential and feasibility with our proposed model and pre-training strategy. To the best of our knowledge, this is the first work to address cross-modality gait recognition.

Abstract (translated)

行走识别是一种关键的生物识别技术。基于相机的行走识别已经在研究和工业领域得到了广泛应用。最近,基于激光雷达的行走识别也开始发展,因为提供了3D结构信息。然而,在某些应用中,相机无法识别人员,例如在低光环境和远距离识别场景中,激光雷达能更好地工作。另一方面,激光雷达系统的部署成本和复杂性限制了其更广泛的应用。因此,在考虑摄像头和激光雷达之间的跨模态行走识别框架时,在更广泛的应用范围内,跨模态行走识别是至关重要的。 在这项工作中,我们提出了第一个基于摄像头和激光雷达之间的跨模态行走识别框架,即CL-Gait。它采用两个流网络对两种模态的特征进行嵌入。由于3D和2D数据之间固有的匹配,这导致识别任务具有挑战性,并表现出显著的模态差异。为了调整两种模态特征空间的对比度,我们提出了一个先验策略来减轻模态差异。为了弥补预训练中缺乏成对相机-激光雷达数据,我们还引入了一种生成大规模数据策略。该策略利用从单色图像和虚拟相机估计的单目深度来生成对比性预训练伪点云。大量实验证明,跨模态行走识别非常具有挑战性,但仍然包含我们提出的模型和预训练策略的潜力和可行性。据我们所知,这是第一个关注跨模态行走识别的工作。

URL

https://arxiv.org/abs/2407.02038

PDF

https://arxiv.org/pdf/2407.02038.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot