In this paper, we address the challenge of making ViT models more robust to unseen affine transformations. Such robustness becomes useful in various recognition tasks such as face recognition when image alignment failures occur. We propose a novel method called KP-RPE, which leverages key points (e.g.~facial landmarks) to make ViT more resilient to scale, translation, and pose variations. We begin with the observation that Relative Position Encoding (RPE) is a good way to bring affine transform generalization to ViTs. RPE, however, can only inject the model with prior knowledge that nearby pixels are more important than far pixels. Keypoint RPE (KP-RPE) is an extension of this principle, where the significance of pixels is not solely dictated by their proximity but also by their relative positions to specific keypoints within the image. By anchoring the significance of pixels around keypoints, the model can more effectively retain spatial relationships, even when those relationships are disrupted by affine transformations. We show the merit of KP-RPE in face and gait recognition. The experimental results demonstrate the effectiveness in improving face recognition performance from low-quality images, particularly where alignment is prone to failure. Code and pre-trained models are available.
在本文中,我们解决了使ViT模型对未见到的平移变换更加鲁棒的问题。这种鲁棒性在各种识别任务中变得有用,例如在面部识别中,当图像对齐失败时。我们提出了一种名为KP-RPE的新方法,它利用关键点(例如~面部关键点)使ViT更加弹性,对平移和姿态变化具有鲁棒性。我们首先观察到,相对位置编码(RPE)是将平移变换推广到ViT的好的方法。然而,RPE只能将模型注入到附近像素比远距离像素更重要这一先验知识。关键点RPE(KP-RPE)是这一原则的扩展,其中像素的重要性不仅由其邻近性决定,还由其与图像中特定关键点之间的相对位置决定。通过将像素的重要性锚定在关键点上,模型可以在平移变换破坏时更有效地保留空间关系。我们在面部和步态识别中展示了KP-RPE的优点。实验结果表明,从低质量图像中提高面部识别性能的有效方法,特别是在对齐容易失败的地方。代码和预训练模型可获得。
https://arxiv.org/abs/2403.14852
Gait recognition stands as one of the most pivotal remote identification technologies and progressively expands across research and industrial communities. However, existing gait recognition methods heavily rely on task-specific upstream driven by supervised learning to provide explicit gait representations, which inevitably introduce expensive annotation costs and potentially cause cumulative errors. Escaping from this trend, this work explores effective gait representations based on the all-purpose knowledge produced by task-agnostic Large Vision Models (LVMs) and proposes a simple yet efficient gait framework, termed BigGait. Specifically, the Gait Representation Extractor (GRE) in BigGait effectively transforms all-purpose knowledge into implicit gait features in an unsupervised manner, drawing from design principles of established gait representation construction approaches. Experimental results on CCPG, CAISA-B* and SUSTech1K indicate that BigGait significantly outperforms the previous methods in both self-domain and cross-domain tasks in most cases, and provides a more practical paradigm for learning the next-generation gait representation. Eventually, we delve into prospective challenges and promising directions in LVMs-based gait recognition, aiming to inspire future work in this emerging topic. The source code will be available at this https URL.
翻译: Gait识别是远程识别技术中最关键的一个,并随着研究和工业社区的不断发展而扩展。然而,现有的gait识别方法在很大程度上依赖于任务特定的上游驱动的监督学习来提供明确的gait表示,这无疑会带来昂贵的注释成本,并可能导致累积错误。为了摆脱这一趋势,本文探讨了基于任务无关的大型视觉模型(LVMs)产生的全知论的有效gait表示,并提出了一个简单而有效的gait框架,称为BigGait。具体来说,BigGait中的Gait表示提取器(GRE)以非监督方式将全知论转换为隐含的gait特征,并从现有gait表示构建方法的设计原则中汲取了设计原则。在CCPG、CAISA-B*和SUSTech1K等实验中,BigGait在自领域和跨领域任务中的表现明显优于以前的方法,并为学习下一代的gait表示提供了一个更实际的范例。最后,我们深入探讨了基于LVMs的gait识别中的潜在挑战和有前景的 direction,旨在激发未来在这个新兴领域的研究工作。源代码将在此链接中提供。
https://arxiv.org/abs/2402.19122
Gait, an unobtrusive biometric, is valued for its capability to identify individuals at a distance, across external outfits and environmental conditions. This study challenges the prevailing assumption that vision-based gait recognition, in particular skeleton-based gait recognition, relies primarily on motion patterns, revealing a significant role of the implicit anthropometric information encoded in the walking sequence. We show through a comparative analysis that removing height information leads to notable performance degradation across three models and two benchmarks (CASIA-B and GREW). Furthermore, we propose a spatial transformer model processing individual poses, disregarding any temporal information, which achieves unreasonably good accuracy, emphasizing the bias towards appearance information and indicating spurious correlations in existing benchmarks. These findings underscore the need for a nuanced understanding of the interplay between motion and appearance in vision-based gait recognition, prompting a reevaluation of the methodological assumptions in this field. Our experiments indicate that "in-the-wild" datasets are less prone to spurious correlations, prompting the need for more diverse and large scale datasets for advancing the field.
翻译:Gait,一种不显眼的生物识别技术,因其能够在距离、外部服装和环境条件下识别个体的能力而受到重视。这项研究挑战了普遍认为,视觉为基础的步态识别,特别是基于骨骼的步态识别,主要依赖于运动模式,揭示了在步行序列中编码的隐含人体测量信息的重要作用。我们通过比较分析展示了,去除身高信息会导致三种模型和两个基准(CASIA-B和GREW)的性能显著下降。此外,我们提出了一个忽略任何时间信息的空间转换器模型来处理个人动作,实现了前所未有的准确性,强调了面向外观信息的偏差,并指出了现有基准中的伪相关关系。这些发现强调了在视觉为基础的步态识别中,需要对运动和外观之间的相互作用进行深入的理解,这促使我们在该领域重新评估方法论假设。我们的实验表明,“野外”数据集不太容易受到伪相关关系的影响,因此需要更大、更多样化的数据集来推动该领域的进步。
https://arxiv.org/abs/2402.08320
Gait recognition is a promising biometric method that aims to identify pedestrians from their unique walking patterns. Silhouette modality, renowned for its easy acquisition, simple structure, sparse representation, and convenient modeling, has been widely employed in controlled in-the-lab research. However, as gait recognition rapidly advances from in-the-lab to in-the-wild scenarios, various conditions raise significant challenges for silhouette modality, including 1) unidentifiable low-quality silhouettes (abnormal segmentation, severe occlusion, or even non-human shape), and 2) identifiable but challenging silhouettes (background noise, non-standard posture, slight occlusion). To address these challenges, we revisit gait recognition pipeline and approach gait recognition from a quality perspective, namely QAGait. Specifically, we propose a series of cost-effective quality assessment strategies, including Maxmial Connect Area and Template Match to eliminate background noises and unidentifiable silhouettes, Alignment strategy to handle non-standard postures. We also propose two quality-aware loss functions to integrate silhouette quality into optimization within the embedding space. Extensive experiments demonstrate our QAGait can guarantee both gait reliability and performance enhancement. Furthermore, our quality assessment strategies can seamlessly integrate with existing gait datasets, showcasing our superiority. Code is available at this https URL.
行走识别是一种有前景的生物识别方法,旨在通过独特的行走模式识别行人。轮廓表示形式以其易得性、简单的结构、稀疏表示和方便建模而闻名,在实验室控制研究中被广泛应用。然而,随着行走识别从实验室环境迅速转移到现实环境,轮廓表示形式面临着一系列具有挑战性的条件,包括1)无法识别的低质量轮廓(异常分割、严重遮挡或甚至非人类形状),2)可以识别但具有挑战性的轮廓(背景噪声、不标准姿势、轻微遮挡)。为了应对这些挑战,我们重新审视了行走识别流程,并从质量角度出发进行行走识别,即QAGait。具体来说,我们提出了一系列具有成本效益的质评估策略,包括Maxmial Connect Area和模板匹配以消除背景噪声和无法识别的轮廓,以及Alignment策略来处理不标准的姿势。我们还提出了两个质量感知的损失函数,将轮廓质量整合到嵌入空间中的优化。大量实验证明,我们的QAGait可以确保行走的可靠性和性能提升。此外,我们的质评估策略可以无缝地整合到现有的行走数据集中,展示出我们优越的质量。代码可在此处访问:https://www. this URL。
https://arxiv.org/abs/2401.13531
Existing gait recognition benchmarks mostly include minor clothing variations in the laboratory environments, but lack persistent changes in appearance over time and space. In this paper, we propose the first in-the-wild benchmark CCGait for cloth-changing gait recognition, which incorporates diverse clothing changes, indoor and outdoor scenes, and multi-modal statistics over 92 days. To further address the coupling effect of clothing and viewpoint variations, we propose a hybrid approach HybridGait that exploits both temporal dynamics and the projected 2D information of 3D human meshes. Specifically, we introduce a Canonical Alignment Spatial-Temporal Transformer (CA-STT) module to encode human joint position-aware features, and fully exploit 3D dense priors via a Silhouette-guided Deformation with 3D-2D Appearance Projection (SilD) strategy. Our contributions are twofold: we provide a challenging benchmark CCGait that captures realistic appearance changes across an expanded and space, and we propose a hybrid framework HybridGait that outperforms prior works on CCGait and Gait3D benchmarks. Our project page is available at this https URL.
现有的步伐识别基准通常包括实验室环境中的轻微服装变化,但缺乏随时间和空间持续变化的视觉效果。在本文中,我们提出了第一个在野外的衣物更换的CCGait基准,该基准包括多样化的服装变化、室内和室外场景以及超过92天的多模态统计。为了进一步解决衣物和观点变化的影响,我们提出了HybridGait混合方法,该方法利用了时间和3D人体网格的投影2D信息。具体来说,我们引入了一个规范的alignment-spatial-temporalTransformer(CA-STT)模块来编码人类关节位置相关的特征,并完全利用通过3D-2D外观投影策略指导的轮廓引导变形。我们的贡献是双重的:我们提供一个挑战性的衣物更换基准,涵盖了扩展和空间的现实外观变化,并提出了一种混合框架HybridGait,在CCGait和Gait3D基准上超过了以前的工作。我们的项目页面可以通过这个链接获得。
https://arxiv.org/abs/2401.00271
Re-identifying participants in ultra-distance running competitions can be daunting due to the extensive distances and constantly changing terrain. To overcome these challenges, computer vision techniques have been developed to analyze runners' faces, numbers on their bibs, and clothing. However, our study presents a novel gait-based approach for runners' re-identification (re-ID) by leveraging various pre-trained human action recognition (HAR) models and loss functions. Our results show that this approach provides promising results for re-identifying runners in ultra-distance competitions. Furthermore, we investigate the significance of distinct human body movements when athletes are approaching their endurance limits and their potential impact on re-ID accuracy. Our study examines how the recognition of a runner's gait is affected by a competition's critical point (CP), defined as a moment of severe fatigue and the point where the finish line comes into view, just a few kilometers away from this location. We aim to determine how this CP can improve the accuracy of athlete re-ID. Our experimental results demonstrate that gait recognition can be significantly enhanced (up to a 9% increase in mAP) as athletes approach this point. This highlights the potential of utilizing gait recognition in real-world scenarios, such as ultra-distance competitions or long-duration surveillance tasks.
在超长距离跑步比赛中重新识别参与者可能会让人望而生畏,因为比赛距离广阔,地形不断变化。为克服这些挑战,计算机视觉技术被开发出来分析跑步者的面容、号码和服装。然而,我们的研究提出了一种新颖的基于步态的跑步者重新识别(RE-ID)方法,通过利用各种预训练的人动作识别(HAR)模型和损失函数。我们的研究结果表明,这种方法在超长距离比赛中识别跑步者具有相当大的潜力。此外,我们研究了当运动员接近其极限时,不同的人体运动对重新识别准确度的影响,以及这种影响如何影响跑步者步态识别的准确性。我们的研究探讨了运动员的关键点(CP),即疲劳程度严重时的时刻,距离终点仅有一两公里,此位置为CP。我们旨在确定此CP如何提高运动员重新识别的准确性。我们的实验结果表明,当运动员接近这个点时,步态识别可以显著增强(MAP增加9%)。这表明在现实场景中,如超长距离比赛或长时间监视任务,可以利用步态识别技术。
https://arxiv.org/abs/2401.00080
Gait recognition is a biometric technology that has received extensive attention. Most existing gait recognition algorithms are unimodal, and a few multimodal gait recognition algorithms perform multimodal fusion only once. None of these algorithms may fully exploit the complementary advantages of the multiple modalities. In this paper, by considering the temporal and spatial characteristics of gait data, we propose a multi-stage feature fusion strategy (MSFFS), which performs multimodal fusions at different stages in the feature extraction process. Also, we propose an adaptive feature fusion module (AFFM) that considers the semantic association between silhouettes and skeletons. The fusion process fuses different silhouette areas with their more related skeleton joints. Since visual appearance changes and time passage co-occur in a gait period, we propose a multiscale spatial-temporal feature extractor (MSSTFE) to learn the spatial-temporal linkage features thoroughly. Specifically, MSSTFE extracts and aggregates spatial-temporal linkages information at different spatial scales. Combining the strategy and modules mentioned above, we propose a multi-stage adaptive feature fusion (MSAFF) neural network, which shows state-of-the-art performance in many experiments on three datasets. Besides, MSAFF is equipped with feature dimensional pooling (FD Pooling), which can significantly reduce the dimension of the gait representations without hindering the accuracy. this https URL
翻译:步态识别是一种生物识别技术,已受到广泛关注。现有的步态识别算法大多是单模态的,而一些多模态步态识别算法仅在特征提取过程中执行一次多模态融合。这些算法可能没有充分利用多个模态的互补优势。在本文中,我们通过考虑步态数据的时序和空间特性,提出了一个多级特征融合策略(MSFFS),在特征提取过程的不同阶段执行多模态融合。此外,我们还提出了一个自适应特征融合模块(AFFM),考虑了轮廓和骨架之间的语义关联。融合过程将不同轮廓区域的特征与它们更相关的骨架关节融合在一起。由于在步行周期中视觉外观变化和时间流逝是同时发生的,我们提出了一个多尺度空间时元特征提取器(MSSTFE),以深入学习空间时元特征。具体来说,MSSTFE在不同的空间尺度上提取和聚合空间时元连接信息。结合上述策略和模块,我们提出了一个多级自适应特征融合(MSAFF)神经网络,在多个数据集上的实验表现均达到了最先进的水平。此外,MSAFF配备了特征维度池化(FD Pooling)模块,这可以在不降低准确性的情况下显著减少步行表示的维度。
https://arxiv.org/abs/2312.14410
Gait datasets are essential for gait research. However, this paper observes that present benchmarks, whether conventional constrained or emerging real-world datasets, fall short regarding covariate diversity. To bridge this gap, we undertake an arduous 20-month effort to collect a cross-covariate gait recognition (CCGR) dataset. The CCGR dataset has 970 subjects and about 1.6 million sequences; almost every subject has 33 views and 53 different covariates. Compared to existing datasets, CCGR has both population and individual-level diversity. In addition, the views and covariates are well labeled, enabling the analysis of the effects of different factors. CCGR provides multiple types of gait data, including RGB, parsing, silhouette, and pose, offering researchers a comprehensive resource for exploration. In order to delve deeper into addressing cross-covariate gait recognition, we propose parsing-based gait recognition (ParsingGait) by utilizing the newly proposed parsing data. We have conducted extensive experiments. Our main results show: 1) Cross-covariate emerges as a pivotal challenge for practical applications of gait recognition. 2) ParsingGait demonstrates remarkable potential for further advancement. 3) Alarmingly, existing SOTA methods achieve less than 43% accuracy on the CCGR, highlighting the urgency of exploring cross-covariate gait recognition. Link: this https URL.
翻译: 步态数据对于步态研究至关重要。然而,本文观察到,无论是传统的约束数据还是新兴的实时现实数据,都与协方差多样性存在不足。为了填补这一空白,我们进行了长达20个月的艰苦努力,收集了一个跨协方差步态识别(CCGR)数据集。CCGR数据集包括970个受试者,大约1600万条序列;几乎每个受试者都有33个视图和53个不同的协方差。与现有数据集相比,CCGR在人口水平和个体水平上具有多样性。此外,视图和协方差都有良好的标注,使得研究人员可以分析不同因素的影响。CCGR提供了多种类型的步态数据,包括RGB、解析、细枝和姿态,为研究人员提供了一个全面的资源。为了更深入地研究跨协方差步态识别,我们提出了基于解析的步态识别(ParsingGait)方法,利用新提出的解析数据。我们进行了广泛的实验。我们的主要结果表明:1)跨协方差步态识别成为 practical applications of gait recognition的一个关键挑战。 2)ParsingGait 展示了进一步发展的显著潜力。3)令人担忧的是,现有 SOTA 方法在 CCGR 上 achieves less than 43% 的准确率,凸显了探索跨协方差步态识别的紧迫性。链接:https://this URL。
https://arxiv.org/abs/2312.14404
While gait recognition has seen many advances in recent years, the occlusion problem has largely been ignored. This problem is especially important for gait recognition from uncontrolled outdoor sequences at range - since any small obstruction can affect the recognition system. Most current methods assume the availability of complete body information while extracting the gait features. When parts of the body are occluded, these methods may hallucinate and output a corrupted gait signature as they try to look for body parts which are not present in the input at all. To address this, we exploit the learned occlusion type while extracting identity features from videos. Thus, in this work, we propose an occlusion aware gait recognition method which can be used to model intrinsic occlusion awareness into potentially any state-of-the-art gait recognition method. Our experiments on the challenging GREW and BRIAR datasets show that networks enhanced with this occlusion awareness perform better at recognition tasks than their counterparts trained on similar occlusions.
虽然最近几年在步态识别方面取得了许多进展,但遮挡问题却被大大忽视了。对于未受控的户外序列中的步态识别,这个问题尤为重要,因为任何小的遮挡都可能影响识别系统。大多数现有方法在提取步态特征时假定具有完整的身体信息。当身体部分被遮挡时,这些方法可能会出现幻觉,并输出一个污染的步态签名,因为他们试图寻找在输入中根本不存在的身体部位。为了解决这个问题,我们利用学习到的遮挡类型来提取身份特征。因此,在本文中,我们提出了一个具有遮挡意识的步态识别方法,可以用于将内在遮挡意识建模为可能实现任何最先进的步态识别方法的状态。我们对具有挑战性的GREW和BRIAR数据集的实验结果表明,与训练在类似遮挡条件下的网络相比,具有遮挡意识的网络在识别任务上表现更好。
https://arxiv.org/abs/2312.02290
Gait recognition holds the promise to robustly identify subjects based on walking patterns instead of appearance information. In recent years, this field has been dominated by learning methods based on two principal input representations: dense silhouette masks or sparse pose keypoints. In this work, we propose a novel, point-based Contour-Pose representation, which compactly expresses both body shape and body parts information. We further propose a local-to-global architecture, called GaitContour, to leverage this novel representation and efficiently compute subject embedding in two stages. The first stage consists of a local transformer that extracts features from five different body regions. The second stage then aggregates the regional features to estimate a global human gait representation. Such a design significantly reduces the complexity of the attention operation and improves efficiency and performance simultaneously. Through large scale experiments, GaitContour is shown to perform significantly better than previous point-based methods, while also being significantly more efficient than silhouette-based methods. On challenging datasets with significant distractors, GaitContour can even outperform silhouette-based methods.
基于行走模式的识别具有识别基于步行模式而非外观信息的受试者的潜力。在过去的几年里,这个领域主导着基于两个主要输入表示的学习方法:密集轮廓掩码或稀疏姿态关键点。在这项工作中,我们提出了一个新颖的基于点的轮廓-姿态表示,该表示既包含了身体形状信息,又包含了身体部位信息。我们进一步提出了一个局部到全局架构,称为GaitContour,以利用这个新颖表示并高效地计算受试者嵌入。第一阶段包括从五个不同的身体区域提取特征的局部Transformer。第二阶段然后对区域特征进行聚合,以估计全局人类步态表示。这种设计显著减少了注意操作的复杂性,同时提高了效率和性能。通过大型的实验,GaitContour证明了比之前基于点的方法显著更好的性能,同时比基于轮廓的方法更高效。在具有巨大干扰者的具有挑战性的数据集中,GaitContour甚至超过了基于轮廓的方法。
https://arxiv.org/abs/2311.16497
The choice of the representations is essential for deep gait recognition methods. The binary silhouettes and skeletal coordinates are two dominant representations in recent literature, achieving remarkable advances in many scenarios. However, inherent challenges remain, in which silhouettes are not always guaranteed in unconstrained scenes, and structural cues have not been fully utilized from skeletons. In this paper, we introduce a novel skeletal gait representation named Skeleton Map, together with SkeletonGait, a skeleton-based method to exploit structural information from human skeleton maps. Specifically, the skeleton map represents the coordinates of human joints as a heatmap with Gaussian approximation, exhibiting a silhouette-like image devoid of exact body structure. Beyond achieving state-of-the-art performances over five popular gait datasets, more importantly, SkeletonGait uncovers novel insights about how important structural features are in describing gait and when do they play a role. Furthermore, we propose a multi-branch architecture, named SkeletonGait++, to make use of complementary features from both skeletons and silhouettes. Experiments indicate that SkeletonGait++ outperforms existing state-of-the-art methods by a significant margin in various scenarios. For instance, it achieves an impressive rank-1 accuracy of over $85\%$ on the challenging GREW dataset. All the source code will be available at this https URL.
选择表示方式对深度步态识别方法至关重要。二值轮廓和骨架坐标是最近文献中两种主导表示方式,在许多场景中取得了显著的进步。然而,仍然存在一些固有挑战,其中轮廓并不总是保证在约束场景中,从骨架中也没有完全利用到结构信息。在本文中,我们提出了一个新颖的骨架步态表示名为骨架映射,以及一个基于骨架的人体骨架图挖掘方法SkeletonGait。具体来说,骨架映射用高斯近似的二维图像表示人类关节的坐标,呈现出类似轮廓的图像,缺乏精确的身体结构。除了在五个流行的步态数据集上实现最先进的性能,更重要的是,SkeletonGait揭示了描述步态和何时结构特征发挥作用的新见解。此外,我们提出了一个多分支架构,名为SkeletonGait++,以利用骨架和轮廓的互补特征。实验表明,SkeletonGait++在各种场景中都显著优于现有最先进的方法。例如,它在具有挑战性的GREW数据集上实现了令人印象深刻的排名前1%的准确性。所有源代码都将公开发布在这个https URL上。
https://arxiv.org/abs/2311.13444
Gait recognition has achieved promising advances in controlled settings, yet it significantly struggles in unconstrained environments due to challenges such as view changes, occlusions, and varying walking speeds. Additionally, efforts to fuse multiple modalities often face limited improvements because of cross-modality incompatibility, particularly in outdoor scenarios. To address these issues, we present a multi-modal Hierarchy in Hierarchy network (HiH) that integrates silhouette and pose sequences for robust gait recognition. HiH features a main branch that utilizes Hierarchical Gait Decomposer (HGD) modules for depth-wise and intra-module hierarchical examination of general gait patterns from silhouette data. This approach captures motion hierarchies from overall body dynamics to detailed limb movements, facilitating the representation of gait attributes across multiple spatial resolutions. Complementing this, an auxiliary branch, based on 2D joint sequences, enriches the spatial and temporal aspects of gait analysis. It employs a Deformable Spatial Enhancement (DSE) module for pose-guided spatial attention and a Deformable Temporal Alignment (DTA) module for aligning motion dynamics through learned temporal offsets. Extensive evaluations across diverse indoor and outdoor datasets demonstrate HiH's state-of-the-art performance, affirming a well-balanced trade-off between accuracy and efficiency.
翻译: 平衡设置中,平衡计取得了进展,但在无约束的环境中,由于诸如视野变化、遮挡和不同行走速度等问题的存在,它 significantly 挣扎。此外,由于跨模态不兼容,将多个模式融合的努力通常面临有限的改进,特别是在户外场景中。为解决这些问题,我们提出了一个多模态层次结构层次网络(HiH),该网络整合了轮廓和姿态序列以实现稳健的步态识别。HiH 具有主分支和辅助分支。主分支利用分层步态分解器(HGD)模块对轮廓数据进行深度和内部模块层次检查,以捕捉整体身体动态到详细肢体运动的运动层次结构。这种方法从总体身体动态到详细肢体运动捕捉运动层次结构,从而在多个空间分辨率上表示步态属性。补充的是,辅助分支基于二维关节序列,丰富了步态分析的时空方面。它采用了一个可塑的空间增强(DSE)模块进行姿态引导的空间关注,和一个可塑的时间对齐(DTA)模块通过学习到的时间偏移来对运动动态进行对齐。在多样室内和室外数据集上进行广泛的评估证明HiH 实现了最先进的性能,确实实现了准确性和效率之间的良好平衡。
https://arxiv.org/abs/2311.11210
Human silhouette extraction is a fundamental task in computer vision with applications in various downstream tasks. However, occlusions pose a significant challenge, leading to incomplete and distorted silhouettes. To address this challenge, we introduce POISE: Pose Guided Human Silhouette Extraction under Occlusions, a novel self-supervised fusion framework that enhances accuracy and robustness in human silhouette prediction. By combining initial silhouette estimates from a segmentation model with human joint predictions from a 2D pose estimation model, POISE leverages the complementary strengths of both approaches, effectively integrating precise body shape information and spatial information to tackle occlusions. Furthermore, the self-supervised nature of \POISE eliminates the need for costly annotations, making it scalable and practical. Extensive experimental results demonstrate its superiority in improving silhouette extraction under occlusions, with promising results in downstream tasks such as gait recognition. The code for our method is available this https URL.
人体轮廓提取是计算机视觉中一个基本任务,在各种下游任务中有应用。然而,遮挡带来的挑战相当大,导致轮廓不完整和扭曲。为解决这个问题,我们引入了POISE:在遮挡下的人体轮廓提取,一种新颖的自监督融合框架,可以提高人体轮廓预测的准确性和鲁棒性。通过将来自分割模型的初始轮廓估计与来自2D姿势估计模型的人体关节预测相结合,POISE有效地利用了两种方法的互补优势,将精确的身体形状信息和空间信息结合起来解决遮挡问题。此外,自监督的 nature of POISE 消除了需要昂贵注释的需求,使得它具有可扩展性和实用性。大量的实验结果表明,在遮挡条件下,POISE在改善轮廓提取方面具有优越性,同时在下游任务(如步态识别)中取得了有益的结果。我们的方法的代码可以从https://URL中获取。
https://arxiv.org/abs/2311.05077
Gait analysis leverages unique walking patterns for person identification and assessment across multiple domains. Among the methods used for gait analysis, skeleton-based approaches have shown promise due to their robust and interpretable features. However, these methods often rely on hand-crafted spatial-temporal graphs that are based on human anatomy disregarding the particularities of the dataset and task. This paper proposes a novel method to simplify the spatial-temporal graph representation for gait-based gender estimation, improving interpretability without losing performance. Our approach employs two models, an upstream and a downstream model, that can adjust the adjacency matrix for each walking instance, thereby removing the fixed nature of the graph. By employing the Straight-Through Gumbel-Softmax trick, our model is trainable end-to-end. We demonstrate the effectiveness of our approach on the CASIA-B dataset for gait-based gender estimation. The resulting graphs are interpretable and differ qualitatively from fixed graphs used in existing models. Our research contributes to enhancing the explainability and task-specific adaptability of gait recognition, promoting more efficient and reliable gait-based biometrics.
步态分析利用独特的步态模式跨多个领域进行人员识别和评估。在步态分析中使用的方法中,基于骨骼的方法表现出了很大的潜力,因为它们具有稳健和可解释的特点。然而,这些方法通常依赖于手工构建的空间和时间图形,这些图形是基于人类解剖学而不考虑数据集和任务特定性的。本文提出了一种新方法,以简化基于步态性别估计的空间和时间图形表示,同时提高解释性,而不会丢失性能。我们的方法和两个模型一起工作,一个向前,一个向后,可以调整每个行走实例的相邻矩阵,从而消除图形的固定性质。通过使用直穿 Gumbel-Softmax技巧,我们的模型可以 end-to-end 训练。我们使用 CASIA-B 数据集展示了我们方法的有效性, resulting 图形具有可解释性,与当前模型使用的固定图形 qualitative 不同。我们的研究有助于增强解释性和任务特定适应性的步态识别,促进更高效和可靠的基于步态的生物特征。
https://arxiv.org/abs/2310.03396
Most current gait recognition methods suffer from poor interpretability and high computational cost. To improve interpretability, we investigate gait features in the embedding space based on Koopman operator theory. The transition matrix in this space captures complex kinematic features of gait cycles, namely the Koopman operator. The diagonal elements of the operator matrix can represent the overall motion trend, providing a physically meaningful descriptor. To reduce the computational cost of our algorithm, we use a reversible autoencoder to reduce the model size and eliminate convolutional layers to compress its depth, resulting in fewer floating-point operations. Experimental results on multiple datasets show that our method reduces computational cost to 1% compared to state-of-the-art methods while achieving competitive recognition accuracy 98% on non-occlusion datasets.
目前的步进识别方法通常存在 poor interpretability 和 high computational cost 的问题,为了改善 interpretability,我们基于 Koopman 操作理论研究了步进特征在嵌入空间中的表示。在这个空间中,过渡矩阵捕获了步进周期中的复杂运动特征,即 Koopman 操作。操作矩阵的对角元素可以表示整个运动趋势,提供了具有物理意义的描述符。为了降低算法的计算成本,我们使用可逆自编码器减少模型大小,消除卷积层以压缩深度,从而减少了浮点操作。多个数据集的实验结果显示,与我们最先进的方法相比,我们的算法将计算成本降低到 1% 以下,而在包含遮挡数据集上的竞争性识别准确率达到 98%。
https://arxiv.org/abs/2309.14764
Gait recognition (GR) is a growing biometric modality used for person identification from a distance through visual cameras. GR provides a secure and reliable alternative to fingerprint and face recognition, as it is harder to distinguish between false and authentic signals. Furthermore, its resistance to spoofing makes GR suitable for all types of environments. With the rise of deep learning, steadily improving strides have been made in GR technology with promising results in various contexts. As video surveillance becomes more prevalent, new obstacles arise, such as ensuring uniform performance evaluation across different protocols, reliable recognition despite shifting lighting conditions, fluctuations in gait patterns, and protecting privacy.This survey aims to give an overview of GR and analyze the environmental elements and complications that could affect it in comparison to other biometric recognition systems. The primary goal is to examine the existing deep learning (DL) techniques employed for human GR that may generate new research opportunities.
步识别(GR)是一种正在增长的生物特征识别方式,通过视觉摄像机用于远距离人员身份识别。GR提供了指纹和面部识别的可靠和安全替代品,因为更难区分虚假和真实信号。此外,它的抗伪造能力使GR适用于各种环境。随着深度学习的兴起,GR技术稳步前进,在各种情况下取得了令人瞩目的成果。随着视频监控越来越普遍,出现了新的问题,例如确保不同协议下一致的性能评估、即使在不同照明条件下也能可靠识别、步态模式的不规则变化以及保护隐私。本调查旨在提供一个概述GR的情况,并分析与环境元素和复杂性相比可能对其产生影响的其他生物特征识别系统。其主要目标是审查现有的人类步识别(GR)技术,可能为新的研究机会提供支持。
https://arxiv.org/abs/2309.10144
Powered ankle prostheses effectively assist people with lower limb amputation to perform daily activities. High performance prostheses with adjustable compliance and capability to predict and implement amputee's intent are crucial for them to be comparable to or better than a real limb. However, current designs fail to provide simple yet effective compliance of the joint with full potential of modification, and lack accurate gait prediction method in real time. This paper proposes an innovative design of powered ankle prosthesis with serial elastic actuator (SEA), and puts forward a MLP based gait recognition method that can accurately and continuously predict more gait parameters for motion sensing and control. The prosthesis mimics biological joint with similar weight, torque, and power which can assist walking of up to 4 m/s. A new design of planar torsional spring is proposed for the SEA, which has better stiffness, endurance, and potential of modification than current designs. The gait recognition system simultaneously generates locomotive speed, gait phase, ankle angle and angular velocity only utilizing signals of single IMU, holding advantage in continuity, adaptability for speed range, accuracy, and capability of multi-functions.
动力脚踝假肢有效地协助人们完成腿部缺失的人的日常活动。具有可调节 compliance 和能够预测和实现缺失腿意图的性能出色的假肢是至关重要的,以便它们能够与真实的腿部相媲美或更好。然而,目前的设计未能提供具有完整潜力的关节 compliance 的简单而有效的 compliance,也没有实时准确的步态预测方法。本文提出了一种创新性的设计,即使用 serial elastic actuator (SEA) 动力脚踝假肢,并提出了基于 MLP 的步态识别方法,能够准确和连续地预测更多的步态参数,用于运动感知和控制。假肢模拟生物关节,具有相似的重量、扭矩和功率,可以协助步行速度达到 4 米/秒。为 SEA 提出了一种新的平面 torsional spring 设计,其 stiffness、耐久性和修改潜力都比目前的设计更好。步态识别系统同时生成运动速度、步态阶段、脚踝角度和角速度,仅利用单个惯性测量单元(IMU)的信号,具有连续、适应速度范围、准确性和多功能能力的优势。
https://arxiv.org/abs/2309.08323
We present FastPoseGait, an open-source toolbox for pose-based gait recognition based on PyTorch. Our toolbox supports a set of cutting-edge pose-based gait recognition algorithms and a variety of related benchmarks. Unlike other pose-based projects that focus on a single algorithm, FastPoseGait integrates several state-of-the-art (SOTA) algorithms under a unified framework, incorporating both the latest advancements and best practices to ease the comparison of effectiveness and efficiency. In addition, to promote future research on pose-based gait recognition, we provide numerous pre-trained models and detailed benchmark results, which offer valuable insights and serve as a reference for further investigations. By leveraging the highly modular structure and diverse methods offered by FastPoseGait, researchers can quickly delve into pose-based gait recognition and promote development in the field. In this paper, we outline various features of this toolbox, aiming that our toolbox and benchmarks can further foster collaboration, facilitate reproducibility, and encourage the development of innovative algorithms for pose-based gait recognition. FastPoseGait is available at this https URL and is actively maintained. We will continue updating this report as we add new features.
我们提出了 FastPoseGait,一个基于 PyTorch 的开源工具集,用于基于姿态的步态识别。该工具集支持一组最新的基于姿态的步态识别算法和相关基准。与专注于单一算法的其他基于姿态的项目不同,FastPoseGait在一个统一框架下集成了多个最先进的算法,包括最新的进展和最佳实践,以 ease 于效率和效力的比较。此外,为了促进基于姿态的步态识别研究的未来发展,我们提供了许多预训练模型和详细的基准结果,提供了有价值的 insights 并作为进一步研究的参考。通过利用 FastPoseGait 提供的高模块化结构和多种方法,研究人员可以快速深入基于姿态的步态识别,促进该领域的发展。在本文中,我们描述了该工具集的各种特性,旨在使我们的工具集和基准进一步促进合作、促进可重复性和鼓励开发基于姿态的步态识别的创新算法。FastPoseGait 可以在这个 https URL 上可用,并正在积极维护。我们将随着添加新特性继续更新本报告。
https://arxiv.org/abs/2309.00794
Binary silhouettes and keypoint-based skeletons have dominated human gait recognition studies for decades since they are easy to extract from video frames. Despite their success in gait recognition for in-the-lab environments, they usually fail in real-world scenarios due to their low information entropy for gait representations. To achieve accurate gait recognition in the wild, this paper presents a novel gait representation, named Gait Parsing Sequence (GPS). GPSs are sequences of fine-grained human segmentation, i.e., human parsing, extracted from video frames, so they have much higher information entropy to encode the shapes and dynamics of fine-grained human parts during walking. Moreover, to effectively explore the capability of the GPS representation, we propose a novel human parsing-based gait recognition framework, named ParsingGait. ParsingGait contains a Convolutional Neural Network (CNN)-based backbone and two light-weighted heads. The first head extracts global semantic features from GPSs, while the other one learns mutual information of part-level features through Graph Convolutional Networks to model the detailed dynamics of human walking. Furthermore, due to the lack of suitable datasets, we build the first parsing-based dataset for gait recognition in the wild, named Gait3D-Parsing, by extending the large-scale and challenging Gait3D dataset. Based on Gait3D-Parsing, we comprehensively evaluate our method and existing gait recognition methods. The experimental results show a significant improvement in accuracy brought by the GPS representation and the superiority of ParsingGait. The code and dataset are available at this https URL .
二进制轮廓和关键点基于 skeleton 的骨骼结构已经主导了数十年的人步态识别研究,因为它们可以从视频帧中轻松提取。尽管在实验室环境下的人步态识别取得了成功,但在现实世界中通常失败,因为它们在步态表示方面的信息熵较低。为了实现野生状态下准确的步态识别,本文提出了一种新的步态表示方法,称为步态解析序列(GPS), GPS 是由精细的人类分割序列(即人类解析)提取的视频帧序列,因此它们具有更高的信息熵,以编码步行时精细人类部件的形状和动态。此外,为了更好地探索 GPS 表示的能力,我们提出了一种基于人类解析的步态识别框架,称为 ParsingGait。ParsingGait 包含一个卷积神经网络(CNN)基线和一个轻量级头,第一个头从 GPS 中提取全局语义特征,而另一个头通过学习部分级别的特征相互信息,通过图卷积网络模型模拟人类步行的详细动态。此外,由于缺少适当的数据集,我们建立了第一个基于解析的步态识别数据集,称为步态3D-解析,通过扩展大型且具有挑战性的步态3D数据集。基于步态3D-解析,我们全面地评估了我们的方法和现有的步态识别方法。实验结果显示,GPS 表示带来了显著的精度提高,以及 ParsingGait 的优越性。代码和数据集可在 this https URL 上获取。
https://arxiv.org/abs/2308.16739
Gait recognition is a promising biometric technology for identification due to its non-invasiveness and long-distance. However, external variations such as clothing changes and viewpoint differences pose significant challenges to gait recognition. Silhouette-based methods preserve body shape but neglect internal structure information, while skeleton-based methods preserve structure information but omit appearance. To fully exploit the complementary nature of the two modalities, a novel triple branch gait recognition framework, TriGait, is proposed in this paper. It effectively integrates features from the skeleton and silhouette data in a hybrid fusion manner, including a two-stream network to extract static and motion features from appearance, a simple yet effective module named JSA-TC to capture dependencies between all joints, and a third branch for cross-modal learning by aligning and fusing low-level features of two modalities. Experimental results demonstrate the superiority and effectiveness of TriGait for gait recognition. The proposed method achieves a mean rank-1 accuracy of 96.0% over all conditions on CASIA-B dataset and 94.3% accuracy for CL, significantly outperforming all the state-of-the-art methods. The source code will be available at this https URL.
步态识别是一种有前途的生物特征识别技术,因为它非侵入性和远距离。然而,外部变化,例如服装变化和视角差异,对步态识别提出了巨大的挑战。基于轮廓的方法维持身体形状,但忽略了内部结构信息,而基于骨骼的方法维持结构信息,但忽略了外观。为了充分利用两种模式学的特征互补性,在本文中提出了一种全新的三分支步态识别框架,名为TriGait。它通过混合融合方式有效地整合了骨骼和轮廓数据的特征,包括一个二流网络提取外观静态和运动特征,一个简单但有效的模块名为JSA-TC,用于捕捉所有关节之间的依赖关系,以及第三个分支,通过对齐和融合两种模式学的低层次特征,进行跨模态学习。实验结果证明了TriGait对步态识别的优越性和有效性。该方法在CASIA-B数据集上在所有条件上都实现了100%的平均准确率,而在CL数据集中实现了94.3%的准确率,显著超越了所有最先进的方法。源代码将在这个httpsURL上提供。
https://arxiv.org/abs/2308.13340