Abstract
Real-time understanding of continuous video streams is essential for intelligent agents operating in high-stakes environments, including autonomous vehicles, surveillance drones, and disaster response robots. Yet, most existing video understanding and highlight detection methods assume access to the entire video during inference, making them unsuitable for online or streaming scenarios. In particular, current models optimize for offline summarization, failing to support step-by-step reasoning needed for real-time decision-making. We introduce Aha, an autoregressive highlight detection framework that predicts the relevance of each video frame against a task described in natural language. Without accessing future video frames, Aha utilizes a multimodal vision-language model and lightweight, decoupled heads trained on a large, curated dataset of human-centric video labels. To enable scalability, we introduce the Dynamic SinkCache mechanism that achieves constant memory usage across infinite-length streams without degrading performance on standard benchmarks. This encourages the hidden representation to capture high-level task objectives, enabling effective frame-level rankings for informativeness, relevance, and uncertainty with respect to the natural language task. Aha achieves state-of-the-art (SOTA) performance on highlight detection benchmarks, surpassing even prior offline, full-context approaches and video-language models by +5.9% on TVSum and +8.3% on this http URL in mAP (mean Average Precision). We explore Aha's potential for real-world robotics applications given a task-oriented natural language input and a continuous, robot-centric video. Both experiments demonstrate Aha's potential effectiveness as a real-time reasoning module for downstream planning and long-horizon understanding.
Abstract (translated)
实时理解连续视频流对于在高风险环境中运行的智能代理至关重要,包括自动驾驶汽车、监控无人机和灾害响应机器人。然而,大多数现有的视频理解和亮点检测方法都假设在整个推理过程中可以访问整个视频,这使得它们不适合在线或流媒体场景。特别是,当前模型优化离线总结,无法支持实时决策所需的逐步推理过程。我们引入了Aha框架,这是一个自回归的亮点检测框架,该框架预测每个视频帧相对于描述任务的自然语言的相关性。在不访问未来视频帧的情况下,Aha利用多模态视觉-语言模型和轻量级、解耦头进行训练,并使用一个大型、经过策划的人类为中心的视频标签数据集。 为了实现可扩展性,我们引入了动态SinkCache机制,在无限长度的数据流中实现了恒定的内存使用率,而不会在标准基准测试中降低性能。这鼓励隐藏表示捕捉高层次的任务目标,从而能够有效地根据自然语言任务对帧级的相关性和不确定性进行排名。Aha在亮点检测基准上取得了最先进的(SOTA)性能,超过了之前离线和全上下文的方法,并且比视频-语言模型在TVSum上的mAP提高了+5.9%,在this http URL 上提高了+8.3%。 我们探索了Aha作为实时推理模块的潜力,在给定任务导向的自然语言输入以及连续、以机器人为中心的视频的情况下,应用于现实世界的机器人应用。这两个实验都展示了Aha作为下游规划和长期理解的有效性。
URL
https://arxiv.org/abs/2509.16421