Reliable evaluation is essential in machine learning research, yet methodological flaws-particularly data leakage-continue to undermine the validity of reported results. In this work, we investigate whether large language models (LLMs) can act as independent analytical agents capable of identifying such issues in published studies. As a case study, we analyze a gesture-recognition paper reporting near-perfect accuracy on a small, human-centered dataset. We first show that the evaluation protocol is consistent with subject-level data leakage due to non-independent training and test splits. We then assess whether this flaw can be detected independently by six state-of-the-art LLMs, each analyzing the original paper without prior context using an identical prompt. All models consistently identify the evaluation as flawed and attribute the reported performance to non-independent data partitioning, supported by indicators such as overlapping learning curves, minimal generalization gap, and near-perfect classification results. These findings suggest that LLMs can detect common methodological issues based solely on published artifacts. While not definitive, their consistent agreement highlights their potential as complementary tools for improving reproducibility and supporting scientific auditing.
https://arxiv.org/abs/2604.14161
Stand-up comedy, and humor in general, are often studied through their verbal content. Yet live performance relies just as much on embodied presence and audience feedback. We introduce TIC-TALK, a multimodal resource with 5,400+ temporally aligned topic segments capturing language, gesture, and audience response across 90 professionally filmed stand-up comedy specials (2015-2024). The pipeline combines BERTopic for 60 s thematic segmentation with dense sentence embeddings, Whisper-AT for 0.8 s laughter detection, a fine-tuned YOLOv8-cls shot classifier, and YOLOv8s-pose for raw keypoint extraction at 1 fps. Raw 17-joint skeletal coordinates are retained without prior clustering, enabling the computation of continuous kinematic signals-arm spread, kinetic energy, and trunk lean-that serve as proxies for performance dynamics. All streams are aligned by hierarchical temporal containment without resampling, and each topic segment stores its sentence-BERT embedding for downstream similarity and clustering tasks. As a concrete use case, we study laughter dynamics across 24 thematic topics: kinetic energy negatively predicts audience laughter rate (r = -0.75, N = 24), consistent with a stillness-before-punchline pattern; personal and bodily content elicits more laughter than geopolitical themes; and shot close-up proportion correlates positively with laughter (r = +0.28), consistent with reactive montage.
https://arxiv.org/abs/2603.21803
Human communication seamlessly integrates speech and bodily motion, where hand gestures naturally complement vocal prosody to express intent, emotion, and emphasis. While recent text-to-speech (TTS) systems have begun incorporating multimodal cues such as facial expressions or lip movements, the role of hand gestures in shaping prosody remains largely underexplored. We propose a novel multimodal TTS framework, Gesture2Speech, that leverages visual gesture cues to modulate prosody in synthesized speech. Motivated by the observation that confident and expressive speakers coordinate gestures with vocal prosody, we introduce a multimodal Mixture-of-Experts (MoE) architecture that dynamically fuses linguistic content and gesture features within a dedicated style extraction module. The fused representation conditions an LLM-based speech decoder, enabling prosodic modulation that is temporally aligned with hand movements. We further design a gesture-speech alignment loss that explicitly models their temporal correspondence to ensure fine-grained synchrony between gestures and prosodic contours. Evaluations on the PATS dataset show that Gesture2Speech outperforms state-of-the-art baselines in both speech naturalness and gesture-speech synchrony. To the best of our knowledge, this is the first work to utilize hand gesture cues for prosody control in neural speech synthesis. Demo samples are available at this https URL
人类交流无缝融合了言语与肢体动作,其中手势自然地与语音韵律相辅相成,以表达意图、情感和强调。尽管近期文本转语音(TTS)系统已开始整合面部表情或唇部动作等多模态线索,但手势在塑造韵律方面的作用仍 largely 未被探索。我们提出了一种新颖的多模态 TTS 框架 Gesture2Speech,该框架利用视觉手势线索来调节合成语音的韵律。受自信且富有表现力的演讲者会协调手势与语音韵律这一观察的启发,我们引入了一种多模态混合专家(MoE)架构,可在专用的风格提取模块内动态融合语言内容与手势特征。融合后的表征条件着一个基于大语言模型的语音解码器,从而实现与手部运动时间对齐的韵律调节。我们还设计了一种手势-语音对齐损失,显式建模它们的时间对应关系,以确保手势与韵律轮廓之间的细粒度同步。在 PATS 数据集上的评估表明,Gestuer2Speech 在语音自然度和手势-语音同步性两方面均优于最新基线方法。据我们所知,这是首个利用手部手势线索进行神经语音合成中韵律控制的工作。演示样本可在此 https URL 获取。
https://arxiv.org/abs/2603.19831
Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.
在扩展现实(XR)中,空中手势常导致疲劳和精度不足。基于表面的交互虽能提升准确性与舒适度,但当前第一人称视觉方法受限于手部追踪困难与不可靠的表面平面估计。我们提出SurfaceXR,一种通过头显手部追踪与智能手表惯性测量单元数据融合的方案,可在日常表面实现稳健输入。我们的核心见解在于这些模态具有互补性:手部追踪提供三维位置数据,而惯性测量单元能捕捉高频运动。一项涵盖21名参与者的研究验证了SurfaceXR在触摸追踪与八类手势识别中的有效性,相较于单模态方法展现出显著改进。
https://arxiv.org/abs/2603.19529
Community literacy programs supporting young newcomer children in Canada face limited staffing and scarce one-to-one time, which constrains personalized English and cultural learning support. This paper reports on a co-design study with United for Literacy tutors that informed Maple, a table-top, peer-like Socially Assistive Robot (SAR) designed as a practice partner within tutor-mediated sessions. From shadowing and co-design interviews, we derived newcomer-specific requirements and added them in an integrated prototype that uses short story-based activities, multi-modal scaffolding (speech, facial feedback, gesture), and embedded quizzes that support attention while producing tutor-actionable formative signals. We contribute system design implications for tutor-in-the-loop SARs supporting language socialization in community settings and outline directions for child-centered evaluation in authentic programs.
加拿大社区扫盲项目中支持新移民儿童的课程面临人员配备有限、一对一时间稀缺的问题,这限制了针对性的英语和文化学习支持。本文报告了一项与“联合扫盲组织”导师开展的协同设计研究,该研究为“枫叶”提供了设计依据——这是一款桌面式、拟人化的社会辅助机器人(SAR),旨在导师介导的会话中作为练习伙伴。通过观察与协同设计访谈,我们提取了新移民儿童的具体需求,并将其整合到一个集成原型中。该原型采用基于短篇故事的活动、多模态支架(语音、面部反馈、手势)以及嵌入式测验,既能支持儿童注意力,又能生成可供导师采取行动的形成性信号。我们提出了关于导师在环社会辅助机器人支持社区场景中语言社会化的系统设计启示,并概述了在真实项目中开展以儿童为中心评估的方向。
https://arxiv.org/abs/2603.18804
This article suggests a reasoning-guided vision-language-motion diffusion framework (RG-VLMD) for generating instruction-aware co-speech gestures for humanoid robots in educational scenarios. The system integrates multi-modal affective estimation, pedagogical reasoning, and teaching-act-conditioned motion synthesis to enable adaptive and semantically consistent robot behavior. A gated mixture-of-experts model predicts Valence/Arousal from input text, visual, and acoustic features, which then mapped to discrete teaching-act categories through an affect-driven this http URL signals condition a diffusion-based motion generator using clip-level intent and frame-level instructional schedules via additive latent restriction with auxiliary action-group supervision. Compared to a baseline diffusion model, our proposed method produces more structured and distinctive motion patterns, as verified by motion statics and pairwise distance analysis. Generated motion sequences remain physically plausible and can be retargeted to a NAO robot for real-time execution. The results reveal that reasoning-guided instructional conditioning improves gesture controllability and pedagogical expressiveness in educational human-robot interaction.
本文提出了一种推理引导的视觉-语言-运动扩散框架(RG-VLMD),用于为教育场景中的人形机器人生成指令感知的共讲手势。该系统整合了多模态情感估计、教学推理和教学行为条件运动合成,以实现自适应且语义一致的机器人行为。一个门控专家混合模型从输入的文本、视觉和声学特征中预测效价/唤醒度,随后通过情感驱动信号将其映射到离散的教学行为类别。这些信号通过添加式潜在约束和辅助动作组监督,以片段级意图和帧级教学调度来条件化一个基于扩散的运动生成器。与基线扩散模型相比,本文方法产生了更具结构性和差异性的运动模式,这通过运动统计量和成对距离分析得到了验证。生成的运动序列保持物理合理性,并可重新定位到NAO机器人上实现实时执行。结果表明,推理引导的教学条件化提高了教育人机交互中的手势可控性和教学表现力。
https://arxiv.org/abs/2603.18771
Estimating 3D hand pose from monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint token Transformer guided by gesture embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.
https://arxiv.org/abs/2603.17396
Event-based vision sensors provide significant advantages for high-speed perception, including microsecond temporal resolution, high dynamic range, and low power consumption. When combined with Spiking Neural Networks (SNNs), they can be deployed on neuromorphic hardware, enabling energy-efficient applications on embedded systems. However, this potential is severely limited by the scarcity of large-scale labeled datasets required to effectively train such models. In this work, we introduce SpikeCLR, a contrastive self-supervised learning framework that enables SNNs to learn robust visual representations from unlabeled event data. We adapt prior frame-based methods to the spiking domain using surrogate gradient training and introduce a suite of event-specific augmentations that leverage spatial, temporal, and polarity transformations. Through extensive experiments on CIFAR10-DVS, N-Caltech101, N-MNIST, and DVS-Gesture benchmarks, we demonstrate that self-supervised pretraining with subsequent fine-tuning outperforms supervised learning in low-data regimes, achieving consistent gains in few-shot and semi-supervised settings. Our ablation studies reveal that combining spatial and temporal augmentations is critical for learning effective spatio-temporal invariances in event data. We further show that learned representations transfer across datasets, contributing to efforts for powerful event-based models in label-scarce settings.
https://arxiv.org/abs/2603.16338
Micro-gesture recognition (MGR) is challenging due to subtle inter-class variations. Existing methods rely on category-level supervision, which is insufficient for capturing subtle and localized motion differences. Thus, this paper proposes a Fine-Grained Semantic Guidance Learning (FG-SGL) framework that jointly integrates fine-grained and category-level semantics to guide vision--language models in perceiving local MG motions. FG-SA adopts fine-grained semantic cues to guide the learning of local motion features, while CP-A enhances the separability of MG features through category-level semantic guidance. To support fine-grained semantic guidance, this work constructs a fine-grained textual dataset with human annotations that describes the dynamic process of MGs in four refined semantic dimensions. Furthermore, a Multi-Level Contrastive Optimization strategy is designed to jointly optimize both modules in a coarse-to-fine pattern. Experiments show that FG-SGL achieves competitive performance, validating the effectiveness of fine-grained semantic guidance for MGR.
https://arxiv.org/abs/2603.16269
Dexterous grasping in multi-object scene constitutes a fundamental challenge in robotic manipulation. Current mainstream grasping datasets predominantly focus on single-object scenarios and predefined grasp configurations, often neglecting environmental interference and the modeling of dexterous pre-grasp gesture, thereby limiting their generalizability in real-world applications. To address this, we propose DGS-Net, an end-to-end grasp prediction network capable of learning dense grasp configurations from single-view point clouds in multi-object scene. Furthermore, we propose a two-stage grasp data generation strategy that progresses from dense single-object grasp synthesis to dense scene-level grasp generation. Our dataset comprises 307 objects, 240 multi-object scenes, and over 350k validated grasps. By explicitly modeling grasp offsets and pre-grasp configurations, the dataset provides more robust and accurate supervision for dexterous grasp learning. Experimental results show that DGS-Net achieves grasp success rates of 88.63\% in simulation and 78.98\% on a real robotic platform, while exhibiting lower penetration with a mean penetration depth of 0.375 mm and penetration volume of 559.45 mm^3, outperforming existing methods and demonstrating strong effectiveness and generalization capability. Our dataset is available at this https URL.
https://arxiv.org/abs/2603.15410
This paper presents a comprehensive pipeline for recognizing objects targeted by human pointing gestures using RGB images. As human-robot interaction moves toward more intuitive interfaces, the ability to identify targets of non-verbal communication becomes crucial. Our proposed system integrates several existing state-of-the-art methods, including object detection, body pose estimation, monocular depth estimation, and vision-language models. We evaluate the impact of 3D spatial information reconstructed from a single image and the utility of image captioning models in correcting classification errors. Experimental results on a custom dataset show that incorporating depth information significantly improves target identification, especially in complex scenes with overlapping objects. The modularity of the approach allows for deployment in environments where specialized depth sensors are unavailable.
https://arxiv.org/abs/2603.15403
Realizing on-device ML-based gesture detection under tight real-time performance, energy and memory constraints is challenging, especially when considering mobile devices with varying battery-power levels. Existing EdgeAI deployments typically rely on a single fixed detector, limiting optimization opportunities. We present Scale-Gest, a novel run-time adaptive gesture detection framework that expands the detector space into a dense family of tiny-YOLO architectures. We introduce multiple novel device-calibrated ACE (Accuracy-Complexity-Energy) profiles by analyzing different model-resolution-stride operating points. A lightweight run-time controller selects an appropriate ACE mode under user-defined and battery constraints, while a motion-aware hand-gesture-tracking ROI gate crops the input for reduced complexity detection. To evaluate performance of our system in real-world car driving scenarios, we introduce a temporally-annotated Driver Simulated Gesture (DSG-18) dataset. Scale-Gest maintains event-level F1 while significantly reducing energy and latency compared to single-detector approaches. On a battery-powered laptop running gesture streams, our ACE controller reduces per-frame energy by 4x (from 6.9 mJ to 1.6 mJ) while maintaining high gesture-detection performance (event-level F1 = 0.8-0.9) and low mean latency (6 ms).
https://arxiv.org/abs/2605.12506
Spike sparsity is widely believed to enable efficient spiking neural network (SNN) inference on GPU hardware. We demonstrate this is an illusion: five distinct sparse computation strategies on Apple M3 Max all fail to outperform dense convolution, because SIMD architectures cannot exploit the fine-grained, unstructured sparsity of i.i.d. binary spikes. Instead, we propose Temporal Aggregated Convolution (TAC), which exploits convolution linearity to pre-aggregate $K$ spike frames before a single convolution call, reducing $T$ calls to $T/K$. On rate-coded data, TAC achieves 13.8times speedup with +1.6% accuracy on MNIST and +5.4% on Fashion-MNIST -- a simultaneous improvement in both speed and accuracy. However, on event-based data where the temporal dimension carries genuine motion information, TAC's temporal collapse is harmful. We therefore introduce TAC-TP (Temporal Preservation), which shares each group's convolution output across K independent LIF steps, preserving full temporal resolution for downstream layers. On DVS128-Gesture, TAC-TP achieves 95.1% accuracy (vs. 96.3% baseline) with 50% fewer convolution calls, while standard TAC drops to 91.3%. Our key finding is that the optimal temporal aggregation strategy is data-dependent: collapse the temporal dimension for rate-coded data (noise reduction) but preserve it for event data (information retention). Speedup is hardware-agnostic: TAC achieves 11.0times on NVIDIA V100, confirming the mechanism transfers across GPU architectures. All operators in the mlx-snn library are open source.
https://arxiv.org/abs/2603.13810
Understanding and answering questions based on a user's pointing gesture is essential for next-generation egocentric AI assistants. However, current Multimodal Large Language Models (MLLMs) struggle with such tasks due to the lack of gesture-rich data and their limited ability to infer fine-grained pointing intent from egocentric video. To address this, we introduce EgoPointVQA, a dataset and benchmark for gesture-grounded egocentric question answering, comprising 4000 synthetic and 400 real-world videos across multiple deictic reasoning tasks. Built upon it, we further propose Hand Intent Tokens (HINT), which encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input to provide explicit spatial and temporal context for interpreting pointing intent. We show that our model outperforms others in different backbones and model sizes. In particular, HINT-14B achieves 68.1% accuracy, on average over 6 tasks, surpassing the state-of-the-art, InternVL3-14B, by 6.6%. To further facilitate the open research, we will release the code, model, and dataset. Project page: this https URL
https://arxiv.org/abs/2603.12533
Long-range Human-Robot Interaction (HRI) remains underexplored. Within it, Command Source Identification (CSI) - determining who issued a command - is especially challenging due to multi-user and distance-induced sensor ambiguity. We introduce HiSync, an optical-inertial fusion framework that treats hand motion as binding cues by aligning robot-mounted camera optical flow with hand-worn IMU signals. We first elicit a user-defined (N=12) gesture set and collect a multimodal command gesture dataset (N=38) in long-range multi-user HRI scenarios. Next, HiSync extracts frequency-domain hand motion features from both camera and IMU data, and a learned CSINet denoises IMU readings, temporally aligns modalities, and performs distance-aware multi-window fusion to compute cross-modal similarity of subtle, natural gestures, enabling robust CSI. In three-person scenes up to 34m, HiSync achieves 92.32% CSI accuracy, outperforming the prior SOTA by 48.44%. HiSync is also validated on real-robot deployment. By making CSI reliable and natural, HiSync provides a practical primitive and design guidance for public-space HRI.
长距离人机交互(HRI)领域尚待探索。其中,命令来源识别(CSI)——即确定哪位用户发出的指令——由于多用户环境和因距离导致的传感器模糊性而尤为具有挑战性。我们提出了一种光学-惯性融合框架HiSync,该框架通过将机器人安装摄像头捕捉到的手部运动光流与佩戴在手上的IMU信号对齐来处理手势绑定线索。 首先,我们从一组12名用户中定义并征集了一系列自定义手势,并收集了长距离多用户HRI场景下的跨模态命令手势数据集(N=38)。接下来,HiSync框架会分别从摄像头和IMU数据中提取频域下手部运动特征;通过CSINet算法对IMU读数进行去噪处理并完成不同传感器的时间同步工作,并采用具有距离感知能力的多窗口融合技术计算细微自然手势间的跨模态相似度,从而实现可靠的CSI。 在三人场景(最远34米)中,HiSync框架实现了92.32%的CSI准确性,比之前的最佳解决方案高出48.44%。此外,HiSync还在实际机器人部署中进行了验证。通过确保CSI既可靠又自然,HiSync为公共场所的人机交互提供了一种实用的基础技术,并对未来的系统设计提出了指导性建议。
https://arxiv.org/abs/2603.11809
Robots that interact with humans must adapt to individual users' preferences to operate effectively in human-centered environments. An intuitive and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, e.g., trajectories, gestures, or voices. Existing techniques primarily focus on generating queries that optimize preference learning outcomes, such as sample efficiency or final preference estimation accuracy. However, the focus on outcome overlooks key user expectations in the process of providing these rankings, which can negatively impact users' adoption of robotic systems. This work proposes the Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG) algorithm. CMA-ES-IG explicitly incorporates user experience considerations into the preference learning process by suggesting perceptually distinct and informative trajectories for users to rank. We demonstrate these benefits through both simulated studies and real-robot experiments. CMA-ES-IG, compared to state-of-the-art alternatives, (1) scales more effectively to higher-dimensional preference spaces, (2) maintains computational tractability for high-dimensional problems, (3) is robust to noisy or inconsistent user feedback, and (4) is preferred by non-expert users in identifying their preferred robot behaviors. This project's code is available at this http URL
与人类互动的机器人必须适应个别用户的偏好,才能在以人为中心的环境中有效运行。学习非专家用户偏好的直观且有效的技术之一是通过机器人的行为排名来进行,例如路径、手势或语音等。现有技术主要关注于生成优化偏好学习结果的查询,比如样本效率或者最终的偏好估计精度。然而,只关注结果忽视了用户提供这些排序过程中的重要期望,这可能会影响用户对机器人系统的采用。 本文提出了一种算法——协方差矩阵适应进化策略与信息增益(Covariance Matrix Adaptation Evolution Strategies with Information Gain, CMA-ES-IG)。CMA-ES-IG通过建议用户可以感知并具有信息量的路径来明确地将用户体验考虑纳入偏好学习过程中,使用户能够对其进行排名。我们通过模拟研究和真实机器人的实验展示了这种方法的好处。 与最先进的替代方案相比,CMA-ES-IG(1)更有效地扩展到更高的维度偏好数量空间;(2)对于高维问题保持计算可行性;(3)对嘈杂或不一致的用户反馈具有鲁棒性;(4)在识别非专家用户的优选机器人行为方面更受用户青睐。该项目的代码可在提供的网址中获取。 原文链接:请将“this http URL”替换为实际的有效URL,以访问项目代码。
https://arxiv.org/abs/2603.09011
Programming robots by demonstration (PbD) is an intuitive concept, but scaling it to real-world variability remains a challenge for most current teaching frameworks. Conditional task graphs are very expressive and can be defined incrementally, which fits very well with the PbD idea. However, acting using conditional task graphs requires reliable perception-grounded online branch selection. In this paper, we present See & Switch, an interactive teaching-and-execution framework that represents tasks as user-extendable graphs of skill parts connected via decision states (DS), enabling conditional branching during replay. Unlike prior approaches that rely on manual branching or low-dimensional signals (e.g., proprioception), our vision-based Switcher uses eye-in-hand images (high-dimensional) to select among competing successor skill parts and to detect out-of-distribution contexts that require new demonstrations. We integrate kinesthetic teaching, joystick control, and hand gestures via an input-modality-abstraction layer and demonstrate that our proposed method is teaching modality-independent, enabling efficient in-situ recovery demonstrations. The system is validated in experiments on three challenging dexterous manipulation tasks. We evaluate our method under diverse conditions and furthermore conduct user studies with 8 participants. We show that the proposed method reliably performs branch selection and anomaly detection for novice users, achieving 90.7 % and 87.9 % accuracy, respectively, across 576 real-robot rollouts. We provide all code and data required to reproduce our experiments at this http URL.
通过演示编程机器人(Programming by Demonstration,PbD)是一个直观的概念,但在现实中扩大其应用范围以适应各种变化仍然对大多数当前的教学框架构成了挑战。条件任务图非常具有表现力,并且可以逐步定义,这与PbD的理念十分契合。然而,在使用条件任务图时,需要可靠的基于感知的在线分支选择来执行相应的操作。 本文介绍了一种称为“See & Switch”的互动教学和执行框架,该框架将任务表示为通过决策状态(Decision States, DS)连接起来的技能部分图形,使用户能够在重播过程中实现条件分支。与以前依赖手动分支或低维信号(如本体感受)的方法不同,我们的基于视觉的“Switcher”使用手眼相机图像(高维度信息),来在竞争性的后续技能部分之间选择,并检测需要新演示的异常情况。 我们通过输入模式抽象层集成了助动式教学、操纵杆控制和手势操作。这种方法独立于教学模态,使用户能够有效地进行现场恢复性示范。该系统已在三项具有挑战性的灵巧操控任务中进行了验证实验,涵盖了多样化的条件,并对8名参与者进行了用户研究。 我们展示了所提出的方法在执行分支选择和异常检测方面对于初学者用户的可靠性,在576个真实机器人演示中分别达到了90.7%的分支选择准确率和87.9%的异常检测准确率。所有代码和数据均可用,以重现我们的实验成果(参见文中提供的网址)。
https://arxiv.org/abs/2603.08057
In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me \textit{that}''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing \emph{stroke}. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio--visual alignment} required by deictic interaction. To bridge this gap, we introduce \textbf{Egocentric Co-Speech Grounding (EcoG)}, where grounding is executable only if an agent jointly predicts \textit{What}, \textit{Where}, and \textit{When}. To operationalize this, we present \textbf{EcoG-Bench}, an evaluation-only bilingual (EN/ZH) diagnostic benchmark of \textbf{811} egocentric clips with dense spatial annotations and millisecond-level stroke supervision. It is organized under a \textbf{Progressive Cognitive Evaluation} protocol. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (\textbf{96.9\%} strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: \textbf{17.0\%}). Moreover, in a diagnostic ablation, replacing the native video--audio interface with timestamped frame samples and externally verified ASR (with word-level timing) substantially improves the same model (\textbf{17.0\%}$\to$\textbf{42.9\%}). Overall, EcoG-Bench provides a strict, executable testbed for event-level speech--gesture binding, and suggests that multimodal interfaces may bottleneck the observability of temporal alignment cues, independently of model reasoning.
在情境合作中,说话者常常使用故意欠指代的指示性命令(例如,“把那个递给我”),其具体所指的是通过讲话与短暂的手势配合——即指示动作“笔画”来确定。然而,许多基于体感的评估标准允许只依赖语言的捷径,使得多模态大型语言模型(MLLMs)在不需要学习指示性互动所需的音视频对齐的情况下也能表现良好。为了弥补这一差距,我们引入了**第一人称视角伴随言语指代(Egocentric Co-Speech Grounding, EcoG)**,在这种机制下,只有当代理同时预测“什么”、“在哪里”和“何时”的时候,才允许执行指向操作。为实现这一点,我们提出了**EcoG-Bench**,这是一个针对**811个第一人称视角片段**的诊断评估基准,这些片段具有密集的空间注释以及毫秒级笔画监督,并且是双语(英语/中文)模式下进行的。该基准遵循**渐进认知评估**协议。 对最新多模态大型语言模型的评测揭示了执行能力上的显著差距:虽然人类在EcoG-Bench上接近满分地表现优异(严格第一人称视角准确率为96.9%),但在原生视频-音频设置中的最佳性能依然很低(通义千问3Pro: 17.0%)。此外,在一次诊断消融实验中,用带有时间戳的帧样本和外部验证过的自动语音识别(ASR,具有单词级定时)来代替原生音视频接口,极大地提升了同一模型的表现(从**17.0%**提升到**42.9%**)。 总体而言,EcoG-Bench为事件级别的言语-手势绑定提供了一个严格的可执行测试环境,并暗示多模态界面可能独立于模型推理能力之外限制了时间对齐线索的可观测性。
https://arxiv.org/abs/2603.07966
Micro-gestures are subtle and transient movements triggered by unconscious neural and emotional activities, holding great potential for human-computer interaction and clinical monitoring. However, their low amplitude, short duration, and strong inter-subject variability make existing deep models prone to degradation under low-sample, noisy, and cross-subject conditions. This paper presents an active inference-based framework for micro-gesture recognition, featuring Expected Free Energy (EFE)-guided temporal sampling and uncertainty-aware adaptive learning. The model actively selects the most discriminative temporal segments under EFE guidance, enabling dynamic observation and information gain maximization. Meanwhile, sample weighting driven by predictive uncertainty mitigates the effects of label noise and distribution shift. Experiments on the SMG dataset demonstrate the effectiveness of the proposed method, achieving consistent improvements across multiple mainstream backbones. Ablation studies confirm that both the EFE-guided observation and the adaptive learning mechanism are crucial to the performance gains. This work offers an interpretable and scalable paradigm for temporal behavior modeling under low-resource and noisy conditions, with broad applicability to wearable sensing, HCI, and clinical emotion monitoring.
微手势是由于无意识的神经和情绪活动触发的细微且短暂的动作,它们在人机交互和临床监测方面具有巨大的潜力。然而,这些动作幅度小、持续时间短以及个体间的高度差异性使得现有的深度学习模型在样本量少、数据噪声大及跨不同用户的情况下容易退化。本文提出了一种基于主动推理框架的微手势识别方法,该方法结合了预期自由能(EFE)引导的时间采样和预测不确定性驱动的学习机制。通过EFE指导,模型能够主动选择最具判别力的时间片段,实现动态观察与信息增益的最大化。同时,由预测不确定性的样本权重调节减轻了标签噪声和分布变化的影响。 在SMG数据集上的实验表明,该方法在多个主流骨干网络上均取得了持续的性能提升。消融研究表明,EFE引导观测和自适应学习机制对于性能改进都是至关重要的因素。这项工作提供了一种可解释且可扩展的时间行为建模范式,在资源有限及噪声较大的条件下具有广泛的应用潜力,包括可穿戴传感、人机交互(HCI)以及临床情绪监测等领域。
https://arxiv.org/abs/2603.07559
We present LiveSense - a cross-platform that transforms a commercial off-the-shelf (COTS) Wi-Fi Network Interface Card (NIC) on a laptop into a centimeter-level Range-Doppler sensor while preserving simultaneous communication capability. The laptops are equipped with COTS Intel AX211 (Wi-Fi 6E) or Intel BE201 (Wi-Fi 7) NICs. LiveSense can (i) Extract fully-synchronized channel state information (CSI) at >= 40 Hz, (ii) Perform time-phase alignment and self-interference cancellation on-device, and (iii) Provide a real-time stream of range, Doppler, subcarrier magnitude/phase and annotated video frames to a Python/Qt Graphical User Interface (GUI). The demo will showcase the ability to detect (i) Distance and radial velocity of attendees within a few meters of the device, (ii) Micro-motion (respiration), and (iii) Hand-gesture ranging. To the best of our knowledge, this is the first-ever demo to obtain accurate range information of targets from commercial Wi-Fi, despite the limited 160 MHz bandwidth.
我们介绍 LiveSense——一个跨平台解决方案,它将笔记本电脑上的商用现货(COTS)Wi-Fi网络接口卡(NIC)转变为具备厘米级精度的测距-多普勒传感器,并同时保持通信功能。LiveSense 配备了 COTS Intel AX211 (Wi-Fi 6E) 或 Intel BE201 (Wi-Fi 7) NIC。 LiveSense 可以实现以下功能: (i) 提取全同步信道状态信息(CSI),频率高达 >= 40 Hz; (ii) 在设备上执行时间相位对齐和自干扰消除; (iii) 向 Python/Qt 图形用户界面 (GUI) 实时提供范围、多普勒频移、子载波幅度/相位以及标注的视频帧流。 演示将展示以下能力: (i) 检测距离及装置周围几米范围内参会者的径向速度; (ii) 微小运动(如呼吸)检测; (iii) 手势测距。 据我们所知,这是首次从商用 Wi-Fi 中获得准确目标范围信息的演示,尽管其带宽仅限于 160 MHz。
https://arxiv.org/abs/2603.06545