Estimating human gaze target from visible images is a critical task for robots to understand human attention, yet the development of generalizable neural architectures and training paradigms remains challenging. While recent advances in pre-trained vision foundation models offer promising avenues for locating gaze targets, the integration of multi-modal cues -- including eyes, head poses, gestures, and contextual features -- demands adaptive and efficient decoding mechanisms. Inspired by Mixture-of-Experts (MoE) for adaptive domain expertise in large vision-language models, we propose GazeMoE, a novel end-to-end framework that selectively leverages gaze-target-related cues from a frozen foundation model through MoE modules. To address class imbalance in gaze target classification (in-frame vs. out-of-frame) and enhance robustness, GazeMoE incorporates a class-balancing auxiliary loss alongside strategic data augmentations, including region-specific cropping and photometric transformations. Extensive experiments on benchmark datasets demonstrate that our GazeMoE achieves state-of-the-art performance, outperforming existing methods on challenging gaze estimation tasks. The code and pre-trained models are released at this https URL
从可见图像中估计人类视线目标是机器人理解人类注意力的关键任务,然而开发通用的神经网络架构和训练模式仍然具有挑战性。虽然近期在预训练视觉基础模型方面的进展为定位注视点提供了有前景的方法,但整合多模态线索——包括眼睛、头部姿态、手势及上下文特征——则需要灵活高效的解码机制。 受混合专家(Mixture-of-Experts, MoE)在大型视觉语言模型中适应领域专业知识的启发,我们提出了GazeMoE,这是一种新颖的端到端框架。该框架通过MoE模块选择性地利用一个冻结的基础模型中的与注视点相关的线索。为了应对视线目标分类中的类别不平衡问题(即画面内与画面外)以及增强其鲁棒性,GazeMoE引入了类平衡辅助损失,并采用了特定区域裁剪和光度变换等策略性的数据增强方法。 在基准数据集上进行的广泛实验表明,我们的GazeMoE框架实现了最先进的性能,在具有挑战性的视线估计任务中超越现有方法。相关代码与预训练模型可以在以下网址获取:[此 URL]
https://arxiv.org/abs/2603.06256
Effective communication is vital in healthcare, especially across language barriers, where non-verbal cues and gestures are critical. This paper presents a privacy-preserving vision-language framework for medical interpreter robots that detects specific speech acts (consent and instruction) and generates corresponding robotic gestures. Built on locally deployed open-source models, the system utilizes a Large Language Model (LLM) with few-shot prompting for intent detection. We also introduce a novel dataset of clinical conversations annotated for speech acts and paired with gesture clips. Our identification module achieved 0.90 accuracy, 0.93 weighted precision, and a 0.91 weighted F1-Score. Our approach significantly improves computational efficiency and, in user studies, outperforms the speech-gesture generation baseline in human-likeness while maintaining comparable appropriateness.
有效的沟通在医疗保健中至关重要,尤其是在语言障碍的情况下,非言语线索和手势在此时尤为重要。本文介绍了一种隐私保护的视觉-语言框架,用于医学翻译机器人,该框架能够检测特定的语言行为(如同意和指示)并生成相应的机器人动作。此系统基于本地部署的开源模型构建,并使用少量样本提示的大规模语言模型(LLM)进行意图识别。我们还引入了一个新的临床对话数据集,其中包含了针对言语行为标注的数据以及对应的肢体动作片段。 我们的身份识别模块达到了0.90的准确性、0.93加权精确度和0.91加权F1值。我们的方法显著提高了计算效率,并且在用户研究中,与现有的基于语音-手势生成的方法相比,在人机相似度方面表现更佳,同时保持了相当的适当性。 这段文字总结了一种用于医疗翻译机器人的系统框架及其性能特征,旨在通过有效的视觉和语言交互来克服语言障碍。
https://arxiv.org/abs/2603.05751
Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens. Once trained, the model predicts gestures and phases for video frames, enabling the construction of a structured surgical timeline. This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.
视频手术时间线是辅助工具手术中非常重要的组成部分,因为它可以帮助外科医生快速聚焦于程序的关键部分。目前的方法包括外科医生填写术后报告,但这种报告通常不够详细;或者手动标注手术视频,这是一项耗时且繁琐的工作。我们提出了一种介于这两种方法之间的新方案:直接从手术视频自动创建手术时间线和叙述。 为了实现这一目标,我们采用了一个基于CLIP的多模态框架,该框架将手术视频帧与文本描述手势进行对齐。具体来说,我们使用CLIP视觉编码器来提取手术视频帧的表示,并利用文本编码器将其相应的手势句子嵌入到共享的嵌入空间中。然后,我们微调模型以改善视频手势和文本标记之间的对齐关系。一旦训练完成,该模型可以预测视频帧的手势及阶段,从而构建结构化的手术时间线。 这种方法通过利用预训练的多模态表示来连接视觉手势与文本叙述,极大地减少了外科医生对手术视频手动审查和标注的需求。
https://arxiv.org/abs/2603.05732
Establishing common ground, a shared set of beliefs and mutually recognized facts, is fundamental to collaboration, yet remains a challenge for current AI systems, especially in multimodal, multiparty settings, where the collaborators bring different information to the table. We introduce the Distributed Partial Information Puzzle (DPIP), a collaborative construction task that elicits rich multimodal communication under epistemic asymmetry. We present a multimodal dataset of these interactions, annotated and temporally aligned across speech, gesture, and action modalities to support reasoning over propositional content and belief dynamics. We then evaluate two paradigms for modeling common ground (CG): (1) state-of-the-art large language models (LLMs), prompted to infer shared beliefs from multimodal updates, and (2) an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task. Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs' abilities to track both task progression and belief state.
建立共同基础,即共享的一套信念和双方认可的事实,是合作的基础,但对当前的人工智能系统来说仍然是一个挑战,尤其是在多模态、多方参与的场景中,参与者会带来不同的信息。我们引入了分布式部分信息拼图(Distributed Partial Information Puzzle, DPIP),这是一个在认识论不对称性条件下激发丰富多模态交流的合作构建任务。我们提供了一套此类互动的多模态数据集,并对言语、手势和行动等不同模态进行了标注和时间同步,以支持命题内容和信念动态的推理。接着,我们评估了两种建立共同基础(CG)的方法:(1) 基于最先进的大型语言模型(LLMs),通过提示它们从多模态更新中推断共享信念;以及 (2) 一种基于动态认识逻辑(Dynamic Epistemic Logic, DEL)的公理化流程,该流程逐步完成同样的任务。在标注的DPIP数据上的结果显示,它对现代大型语言模型跟踪任务进展和信念状态的能力提出了挑战。
https://arxiv.org/abs/2603.05450
To assist humans in open-world environments, robots must interpret ambiguous instructions to locate desired objects. Foundation model-based approaches excel at multimodal grounding, but they lack a principled mechanism for modeling uncertainty in long-horizon tasks. In contrast, Partially Observable Markov Decision Processes (POMDPs) provide a systematic framework for planning under uncertainty but are often limited in supported modalities and rely on restrictive environment assumptions. We introduce LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP), a modular POMDP system that integrates language, gesture, and visual observations for open-world object search. Unlike prior work, LEGS-POMDP explicitly models two sources of partial observability: uncertainty over the target object's identity and its spatial location. In simulation, multimodal fusion significantly outperforms unimodal baselines, achieving an average success rate of 89\% across challenging environments and object categories. Finally, we demonstrate the full system on a quadruped mobile manipulator, where real-world experiments qualitatively validate robust multimodal perception and uncertainty reduction under ambiguous instructions.
为了在开放环境中协助人类,机器人必须能够解析模糊指令来定位所需物体。基于基础模型的方法在多模态语义关联方面表现出色,但在长时间任务中缺乏一种系统的方式来建模不确定性。相比之下,部分可观察马尔科夫决策过程(POMDP)提供了一种在不确定条件下进行规划的系统框架,但通常支持的模态有限,并依赖于严格的环境假设。我们引入了一个模块化POMDP系统——LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP),该系统结合了语言、手势和视觉观察来实现开放世界中的物体搜索功能。 不同于以往的工作,LEGS-POMDP 明确建模了两种部分可观察性的来源:目标对象身份的不确定性及其空间位置的不确定性。在模拟环境中,多模式融合显著优于单模式基线方法,在具有挑战性环境及不同类别对象的情况下,平均成功率为89%。 最后,我们在四足移动机器人上展示了完整的系统,并通过实际实验定性地验证了其强大的多模态感知能力以及在模糊指令下不确定性减少的能力。
https://arxiv.org/abs/2603.04705
Recent advancements have made robot navigation more intuitive by transitioning from traditional 2D displays to spatially aware Mixed Reality (MR) systems. However, current MR interfaces often rely on manual "air tap" gestures for goal placement, which can be repetitive and physically demanding, especially for beginners. This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS). This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction. By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology. Comprehensive experiments comparing MRPoS against conventional gesture-based systems demonstrate that our approach significantly reduces task completion time and workload, providing a more accessible and efficient interface. For additional material, please check: this https URL
https://arxiv.org/abs/2603.13313
Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions. Current large language model (LLM)-based conversational agents lack embodiment and the expressive gestures essential for natural interaction. Existing solutions for ECAs often produce rigid, low-diversity motions, that are unsuitable for human-like interaction. Alternatively, generative methods for co-speech gesture synthesis yield natural body gestures but depend on future speech context and require long run-times. To bridge this gap, we present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue. We employ body-part aware gesture codecs that encode hierarchical motion details into multi-level discrete tokens. These tokens are then autoregressively generated by a two-dimensional causal framework conditioned on LLM-based speech-text embeddings, modeling both temporal dynamics and part-level motion hierarchy in real time. Further, we introduce auxiliary objectives to encourage expressive and diverse gestures while preventing convergence to static poses. Comparative evaluations demonstrate that our causal and real-time approach produces natural and contextually aligned gestures against recent baselines. We urge the reader to explore demo videos on this https URL.
实体对话代理(ECAs)旨在通过语音、手势和面部表情模拟面对面的人类互动。当前基于大型语言模型(LLM)的对话代理缺乏身体表现力以及自然互动所必需的表情动作。现有的ECAs解决方案通常会产生僵硬且多样性较低的动作,不适合模仿人类交互。另一方面,用于同步生成自然肢体动作的方法虽然依赖于未来语境中的言语内容,并能产生自然的身体手势,但需要较长的时间来运行。 为了弥合这一差距,我们介绍了MIBURI,这是一个在线、因果框架,用于实时合成与对话语音相匹配的富有表现力的全身动作和面部表情。我们采用肢体部位感知的手势编解码器,将分层运动细节编码为多级离散令牌,并通过基于LLM的言语-文本嵌入条件,在二维因果框架内自回归生成这些令牌,从而实时建模时间动态和部分级别的运动层次结构。此外,为了鼓励产生富有表现力且多样化的手势并防止收敛到静态姿态,我们引入了辅助目标。 对比评估表明,我们的因果性和实时性方法在最近的基准测试中产生的动作自然且与上下文相关。我们建议读者访问以下链接查看演示视频:[此处插入实际URL](请根据实际情况提供具体链接)。
https://arxiv.org/abs/2603.03282
Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential. In this context, hands-free teleoperation enhances operator mobility and situational awareness, thereby improving safety in hazardous environments. While vision-based gesture recognition has been explored as one method for hands-free teleoperation, its performance often deteriorates under occlusions, lighting variations, and cluttered backgrounds, limiting its applicability in real-world operations. To overcome these limitations, we propose a multimodal gesture recognition framework that integrates inertial data (accelerometer, gyroscope, and orientation) from Apple Watches on both wrists with capacitive sensing signals from custom gloves. We design a late fusion strategy based on the log-likelihood ratio (LLR), which not only enhances recognition performance but also provides interpretability by quantifying modality-specific contributions. To support this research, we introduce a new dataset of 20 distinct gestures inspired by aircraft marshalling signals, comprising synchronized RGB video, IMU, and capacitive sensor data. Experimental results demonstrate that our framework achieves performance comparable to a state-of-the-art vision-based baseline while significantly reducing computational cost, model size, and training time, making it well suited for real-time robot control. We therefore underscore the potential of sensor-based multimodal fusion as a robust and interpretable solution for gesture-driven mobile robot and drone teleoperation.
人类操作员仍然经常暴露在危险环境中,如灾区和工业设施中,在这些地方,直观且可靠的移动机器人和无人驾驶飞行器(UAV)远程操控是至关重要的。在这种情况下,无手操控制(hands-free teleoperation)能够增强操作人员的机动性和情境意识,从而提高危险环境中的安全性。虽然基于视觉的手势识别已被探索为一种无手操控的方法,但由于遮挡、光照变化和杂乱背景的影响,其性能经常下降,限制了其实用性。 为了克服这些局限性,我们提出了一种多模态手势识别框架,该框架集成了来自双腕Apple Watch的惯性数据(加速度计、陀螺仪和方向)以及定制手套提供的电容感应信号。我们设计了一种基于对数似然比(LLR)的晚期融合策略,这种策略不仅提升了识别性能,还通过量化各模式的具体贡献提供了可解释性。 为了支持这项研究,我们引入了一个新的数据集,该数据集中包含20个不同的手势,这些手势源自飞机引导信号。这个新数据集包括同步的RGB视频、IMU和电容传感器数据。实验结果表明,我们的框架在性能上与最先进的基于视觉的方法相媲美,同时显著降低了计算成本、模型大小和训练时间,使其非常适合实时机器人控制。 因此,我们强调传感器驱动多模态融合作为手势驱动移动机器人和无人机远程操控的稳健且可解释解决方案的巨大潜力。
https://arxiv.org/abs/2602.23694
Generating realistic conversational gestures are essential for achieving natural, socially engaging interactions with digital humans. However, existing methods typically map a single audio stream to a single speaker's motion, without considering social context or modeling the mutual dynamics between two people engaging in conversation. We present DyaDiT, a multi-modal diffusion transformer that generates contextually appropriate human motion from dyadic audio signals. Trained on Seamless Interaction Dataset, DyaDiT takes dyadic audio with optional social-context tokens to produce context-appropriate motion. It fuses information from both speakers to capture interaction dynamics, uses a motion dictionary to encode motion priors, and can optionally utilize the conversational partner's gestures to produce more responsive motion. We evaluate DyaDiT on standard motion generation metrics and conduct quantitative user studies, demonstrating that it not only surpasses existing methods on objective metrics but is also strongly preferred by users, highlighting its robustness and socially favorable motion generation. Code and models will be released upon acceptance.
生成逼真的对话手势对于实现自然、社交互动的数字人类交互至关重要。然而,现有方法通常将单一音频流映射到单个发言人的动作上,而不考虑社会背景或建模两个参与对话的人之间的相互动态关系。我们提出了DyaDiT,这是一种多模式扩散变压器,可以从双人音频信号生成上下文相关的肢体动作。在Seamless Interaction Dataset(无缝交互数据集)上训练后,DyaDiT可以接收带有可选社交背景标记的双人音频,并产生相应的适当动作。它融合了双方的信息以捕捉互动动态,使用动作词典来编码动作先验知识,并且可以选择利用对话伙伴的手势生成更灵敏的动作。 我们在标准动作生成指标上评估了DyaDiT,并进行了定量用户研究,证明了它不仅在客观指标上超越现有方法,而且也因其实用性和社交友好性而被用户强烈偏爱。这表明其稳健的上下文适应能力以及能够产生高度自然的社会互动行为。代码和模型将在接受后发布。 这段文字描述了一种新的人工智能系统DyaDiT的功能与优势,该系统能够生成更符合社会情境的动作,并且在用户体验方面有着显著的优势。
https://arxiv.org/abs/2602.23165
We present, to our knowledge, the first sign language-driven Vision-Language-Action (VLA) framework for intuitive and inclusive human-robot interaction. Unlike conventional approaches that rely on gloss annotations as intermediate supervision, the proposed system adopts a gloss-free paradigm and directly maps visual sign gestures to semantic instructions. This design reduces annotation cost and avoids the information loss introduced by gloss representations, enabling more natural and scalable multimodal interaction. In this work, we focus on a real-time alphabet-level finger-spelling interface that provides a robust and low-latency communication channel for robotic control. Compared with large-scale continuous sign language recognition, alphabet-level interaction offers improved reliability, interpretability, and deployment feasibility in safety-critical embodied environments. The proposed pipeline transforms continuous gesture streams into coherent language commands through geometric normalization, temporal smoothing, and lexical refinement, ensuring stable and consistent interaction. Furthermore, the framework is designed to support future integration of transformer-based gloss-free sign language models, enabling scalable word-level and sentence-level semantic understanding. Experimental results demonstrate the effectiveness of the proposed system in grounding sign-derived instructions into precise robotic actions under diverse interaction scenarios. These results highlight the potential of the framework to advance accessible, scalable, and multimodal embodied intelligence.
我们提出了一种开创性的手语驱动的视觉-语言-动作(VLA)框架,旨在实现直观且包容的人机交互。与传统的依赖于词素注释作为中间监督的方法不同,所提出的系统采用无词素范式,并直接将视觉手势映射为语义指令。这种设计减少了标注成本,避免了由词素表示引入的信息损失,从而实现了更自然和可扩展的多模态互动。在本研究中,我们专注于一个实时的手字母级手指拼写界面,该界面提供了一个可靠且低延迟的机器人控制通信通道。 与大规模连续手语识别相比,字母级别的交互在安全关键性的实体环境中提供了更高的可靠性、可解释性和部署可行性。所提出的流水线通过几何标准化、时间平滑和词汇精炼将连续手势流转换为连贯的语言命令,确保了稳定且一致的互动体验。此外,该框架设计支持未来基于变压器的无词素手语模型的集成,从而能够实现单词级和句子级的语义理解。 实验结果证明,在各种交互场景中,所提出的系统能有效地将手势指令转化为精确的机器人动作,突显了该框架在推进可访问性、可扩展性和多模态实体智能方面的潜力。
https://arxiv.org/abs/2602.22514
This work presents iMiGUE-Speech, an extension of the iMiGUE dataset that provides a spontaneous affective corpus for studying emotional and affective states. The new release focuses on speech and enriches the original dataset with additional metadata, including speech transcripts, speaker-role separation between interviewer and interviewee, and word-level forced alignments. Unlike existing emotional speech datasets that rely on acted or laboratory-elicited emotions, iMiGUE-Speech captures spontaneous affect arising naturally from real match outcomes. To demonstrate the utility of the dataset and establish initial benchmarks, we introduce two evaluation tasks for comparative assessment: speech emotion recognition and transcript-based sentiment analysis. These tasks leverage state-of-the-art pre-trained representations to assess the dataset's ability to capture spontaneous affective states from both acoustic and linguistic modalities. iMiGUE-Speech can also be synchronously paired with micro-gesture annotations from the original iMiGUE dataset, forming a uniquely multimodal resource for studying speech-gesture affective dynamics. The extended dataset is available at this https URL.
这项工作介绍了iMiGUE-Speech,这是对iMiGUE数据集的扩展,提供了用于研究情绪和情感状态的自发性情感语料库。新版本专注于语音,并通过添加额外的元数据来丰富原始数据集,包括语音转录、访谈者与受访者的角色分离以及词级别强制对齐信息。 不同于现有的基于表演或实验室诱发的情绪语音数据集,iMiGUE-Speech 捕捉到了自然来源于真实比赛结果中的自发性情感。为了展示该数据集的实用性并建立初步基准,我们引入了两个用于比较评估的任务:语音情绪识别和基于转录的情感分析。这些任务利用最先进的预训练表示来评估数据集捕捉来自声学和语言模态的自发情感状态的能力。 iMiGUE-Speech还可以与原始iMiGUE数据集中微手势注释同步配对,形成一个独特的多模式资源,用于研究语音-手势情感动态。扩展后的数据集可以在[此链接](https://this https URL/)获取。
https://arxiv.org/abs/2602.21464
As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see this https URL for details.
随着虚拟现实(VR)、远程呈现和数字人类应用的发展,实体代理的角色变得越来越重要。这些代理的动作需要超越与语音同步的手势,它们应该能够面向用户、响应用户的移动,并保持自然的眼神交流。目前的方法缺乏这种空间感知能力。我们填补了这一空白,推出了首个实时的、完全因果关系的空间感知对话动作生成方法,能够在流媒体VR头显上部署。 我们的方法基于给定用户的位置和双向音频信息来生成全身运动,使手势与语音同步的同时根据用户的移动调整代理的方向。我们的架构结合了一个因果Transformer-based变分自编码器(VAE)以及交错的隐变量令牌以支持流式推理,还有一个受控于用户轨迹和音频的数据匹配模型。 为了适应不同的眼神交流偏好,我们引入了一个眼神评分机制,并使用无分类指导来分离学习与控制:模型从数据中捕捉自然的空间对应关系,而用户则可以在推断时调整目光接触的强度。在Embody 3D数据集上,我们的方法实现了超过每秒300帧的速度(比非因果基准快三倍)并捕获了自然对话中的微妙空间动态变化。 我们通过一个实时VR系统验证了这一方案的有效性,将具有空间感知能力的对话代理带到了现实世界的部署中。详情请参阅此[链接](https://example.com)。
https://arxiv.org/abs/2602.18432
In this work, we present HiAER-Spike, a modular, reconfigurable, event-driven neuromorphic computing platform designed to execute large spiking neural networks with up to 160 million neurons and 40 billion synapses - roughly twice the neurons of a mouse brain at faster than real time. This system, assembled at the UC San Diego Supercomputer Center, comprises a co-designed hard- and software stack that is optimized for run-time massively parallel processing and hierarchical address-event routing (HiAER) of spikes while promoting memory-efficient network storage and execution. The architecture efficiently handles both sparse connectivity and sparse activity for robust and low-latency event-driven inference for both edge and cloud computing. A Python programming interface to HiAER-Spike, agnostic to hardware-level detail, shields the user from complexity in the configuration and execution of general spiking neural networks with minimal constraints in topology. The system is made easily available over a web portal for use by the wider community. In the following, we provide an overview of the hard- and software stack, explain the underlying design principles, demonstrate some of the system's capabilities and solicit feedback from the broader neuromorphic community. Examples are shown demonstrating HiAER-Spike's capabilities for event-driven vision on benchmark CIFAR-10, DVS event-based gesture, MNIST, and Pong tasks.
在这项工作中,我们介绍了HiAER-Spike,这是一个模块化、可重构的事件驱动神经形态计算平台,旨在执行高达1.6亿个神经元和400亿个突触的大规模脉冲神经网络——这大约是老鼠大脑神经元数量的两倍,并且运行速度超过实时。该系统在加州大学圣地亚哥分校超级计算机中心组装而成,它包含了一个专门为大规模并行处理而优化、采用层次化地址事件路由(HiAER)的软硬件堆栈,在提高内存效率的同时支持网络存储和执行。这种架构能够高效地应对稀疏连接和稀疏活动,适用于边缘计算和云计算中的稳健且低延迟的事件驱动推理。我们还提供了一个Python编程接口给HiAER-Spike,该接口不依赖于硬件细节,并能简化用户在配置和执行一般脉冲神经网络时的操作复杂性,仅需遵守有限的拓扑约束。通过一个网页门户,这个系统被广泛地社区成员所轻松访问并使用。 接下来,我们概述了软硬件堆栈,解释了背后的设计原则,并展示了该系统的部分能力,同时邀请更广泛的神经形态计算社区提供反馈意见。文中还提供了实例展示HiAER-Spike在事件驱动视觉(基准CIFAR-10)、DVS基于事件的手势识别、MNIST任务和Pong游戏等领域的应用能力。
https://arxiv.org/abs/2602.18072
This paper presents a methodological analysis of the gesture-recognition approach proposed by Liu and Szirányi, with a particular focus on the validity of their evaluation protocol. We show that the reported near-perfect accuracy metrics result from a frame-level random train-test split that inevitably mixes samples from the same subjects across both sets, causing severe data leakage. By examining the published confusion matrix, learning curves, and dataset construction, we demonstrate that the evaluation does not measure generalization to unseen individuals. Our findings underscore the importance of subject-independent data partitioning in vision-based gesture-recognition research, especially for applications - such as UAV-human interaction - that require reliable recognition of gestures performed by previously unseen people.
本文对刘和Szirányi提出的姿态识别方法进行了方法论分析,特别关注其评估协议的有效性。我们展示了所报告的近乎完美的准确率指标是由帧级别的随机训练-测试拆分导致的结果,这种拆分不可避免地将同一主体的数据混入了两个数据集之中,从而造成了严重的信息泄露问题。通过审查已发布的混淆矩阵、学习曲线以及数据集构建情况,我们证明该评估无法衡量对未见过个体的泛化能力。我们的研究结果强调,在基于视觉的姿态识别研究中进行独立于主体的数据拆分的重要性,尤其是在如无人机与人交互这样的应用领域,这些应用需要可靠地识别人群中新出现的手势。
https://arxiv.org/abs/2602.17854
The ever increasing intensity and number of disasters make even more difficult the work of First Responders (FRs). Artificial intelligence and robotics solutions could facilitate their operations, compensating these difficulties. To this end, we propose a dataset for gesture-based UGV control by FRs, introducing a set of 12 commands, drawing inspiration from existing gestures used by FRs and tactical hand signals and refined after incorporating feedback from experienced FRs. Then we proceed with the data collection itself, resulting in 3312 RGBD pairs captured from 2 viewpoints and 7 distances. To the best of our knowledge, this is the first dataset especially intended for gesture-based UGV guidance by FRs. Finally we define evaluation protocols for our RGBD dataset, termed FR-GESTURE, and we perform baseline experiments, which are put forward for improvement. We have made data publicly available to promote future research on the domain: this https URL.
https://arxiv.org/abs/2602.17573
Quantifying biometric characteristics within hand gestures involve derivation of fitness scores from a gesture and identity aware feature space. However, evaluating the quality of these scores remains an open question. Existing biometric capacity estimation literature relies upon error rates. But these rates do not indicate goodness of scores. Thus, in this manuscript we present an exhaustive set of evaluation measures. We firstly identify ranking order and relevance of output scores as the primary basis for evaluation. In particular, we consider both rank deviation as well as rewards for: (i) higher scores of high ranked gestures and (ii) lower scores of low ranked gestures. We also compensate for correspondence between trends of output and ground truth scores. Finally, we account for disentanglement between identity features of gestures as a discounting factor. Integrating these elements with adequate weighting, we formulate advanced acceptance score as a holistic evaluation measure. To assess effectivity of the proposed we perform in-depth experimentation over three datasets with five state-of-the-art (SOTA) models. Results show that the optimal score selected with our measure is more appropriate than existing other measures. Also, our proposed measure depicts correlation with existing measures. This further validates its reliability. We have made our \href{this https URL}{code} public.
https://arxiv.org/abs/2602.15535
Embodied social agents have recently advanced in generating synchronized speech and gestures. However, most interactive systems remain fundamentally reactive, responding only to current sensory inputs within a short temporal window. Proactive social behavior, in contrast, requires deliberation over accumulated context and intent inference, which conflicts with the strict latency budget of real-time interaction. We present \emph{ProAct}, a dual-system framework that reconciles this time-scale conflict by decoupling a low-latency \emph{Behavioral System} for streaming multimodal interaction from a slower \emph{Cognitive System} which performs long-horizon social reasoning and produces high-level proactive intentions. To translate deliberative intentions into continuous non-verbal behaviors without disrupting fluency, we introduce a streaming flow-matching model conditioned on intentions via ControlNet. This mechanism supports asynchronous intention injection, enabling seamless transitions between reactive and proactive gestures within a single motion stream. We deploy ProAct on a physical humanoid robot and evaluate both motion quality and interactive effectiveness. In real-world interaction user studies, participants and observers consistently prefer ProAct over reactive variants in perceived proactivity, social presence, and overall engagement, demonstrating the benefits of dual-system proactive control for embodied social interaction.
https://arxiv.org/abs/2602.14048
Wearable e-textile interfaces require gesture recognition capabilities but face severe constraints in power consumption, computational capacity, and form factor that make traditional deep learning impractical. While lightweight architectures like MobileNet improve efficiency, they still demand thousands of parameters, limiting deployment on textile-integrated platforms. We introduce a convexified attention mechanism for wearable applications that dynamically weights features while preserving convexity through nonexpansive simplex projection and convex loss functions. Unlike conventional attention mechanisms using non-convex softmax operations, our approach employs Euclidean projection onto the probability simplex combined with multi-class hinge loss, ensuring global convergence guarantees. Implemented on a textile-based capacitive sensor with four connection points, our approach achieves 100.00\% accuracy on tap gestures and 100.00\% on swipe gestures -- consistent across 10-fold cross-validation and held-out test evaluation -- while requiring only 120--360 parameters, a 97\% reduction compared to conventional approaches. With sub-millisecond inference times (290--296$\mu$s) and minimal storage requirements ($<$7KB), our method enables gesture interfaces directly within e-textiles without external processing. Our evaluation, conducted in controlled laboratory conditions with a single-user dataset, demonstrates feasibility for basic gesture interactions. Real-world deployment would require validation across multiple users, environmental conditions, and more complex gesture vocabularies. These results demonstrate how convex optimization can enable efficient on-device machine learning for textile interfaces.
https://arxiv.org/abs/2602.13030
The fields of human-robot interaction (HRI) and embodied conversational agents (ECAs) have long studied how empathy could be implemented in machines. One of the major drivers has been the goal of giving multimodal social and emotional intelligence to these artificially intelligent agents, which interact with people through facial expressions, body, gesture, and speech. What empathic behaviors and models have these fields implemented by mimicking human and animal behavior? In what ways have they explored creating machine-specific analogies? This chapter aims to review the knowledge from these studies, towards applying the lessons learned to today's ubiquitous, language-based agents such as ChatGPT.
人机交互(HRI)与具身智能体(ECAs)领域长期以来一直研究如何在机器中实现共情。其主要目标之一是为这些通过面部表情、身体动作、手势和语音与人交互的人工智能体赋予多模态社交与情感智能。这些领域通过模仿人类与动物行为,实现了哪些共情行为与模型?它们又以何种方式探索了创造机器特有的类比?本章旨在回顾这些研究的成果,从而将经验应用于当今无处不在的语言模型(如ChatGPT)。
https://arxiv.org/abs/2603.20200
Background: Robot-assisted minimally invasive surgery (RMIS) research increasingly relies on multimodal data, yet access to proprietary robot telemetry remains a major barrier. We introduce MiDAS, an open-source, platform-agnostic system enabling time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms. Methods: MiDAS integrates electromagnetic and RGB-D hand tracking, foot pedal sensing, and surgical video capturing without requiring proprietary robot interfaces. We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of peg transfer and hernia repair suturing tasks performed by surgical residents. Correlation analysis and downstream gesture recognition experiments were conducted. Results: External hand and foot sensing closely approximated internal robot kinematics and non-invasive motion signals achieved gesture recognition performance comparable to proprietary telemetry. Conclusion: MiDAS enables reproducible multimodal RMIS data collection and is released with annotated datasets, including the first multimodal dataset capturing hernia repair suturing on high-fidelity simulation models.
https://arxiv.org/abs/2602.12407