Abstract
Drones operating in human-occupied spaces suffer from insufficient communication mechanisms that create uncertainty about their intentions. We present HoverAI, an embodied aerial agent that integrates drone mobility, infrastructure-independent visual projection, and real-time conversational AI into a unified platform. Equipped with a MEMS laser projector, onboard semi-rigid screen, and RGB camera, HoverAI perceives users through vision and voice, responding via lip-synced avatars that adapt appearance to user demographics. The system employs a multimodal pipeline combining VAD, ASR (Whisper), LLM-based intent classification, RAG for dialogue, face analysis for personalization, and voice synthesis (XTTS v2). Evaluation demonstrates high accuracy in command recognition (F1: 0.90), demographic estimation (gender F1: 0.89, age MAE: 5.14 years), and speech transcription (WER: 0.181). By uniting aerial robotics with adaptive conversational AI and self-contained visual output, HoverAI introduces a new class of spatially-aware, socially responsive embodied agents for applications in guidance, assistance, and human-centered interaction.
Abstract (translated)
在有人类活动的空间中运行的无人机由于通信机制不足，导致人们对它们的意图存在不确定性。我们提出了HoverAI，这是一种集成了无人机移动性、基础设施独立的视觉投影和实时对话人工智能于一体的综合平台的实体空中代理。通过配备MEMS激光投影仪、机载半刚性屏幕以及RGB相机，HoverAI能够通过视觉和语音感知用户，并利用与其人口统计特征相适应的唇同步化身做出回应。 该系统采用了一种多模态管道，结合了语音活动检测（VAD）、自动语音识别（使用Whisper模型）、基于大型语言模型的意图分类、RAG对话系统、面部分析以实现个性化以及语音合成（XTTS v2）等技术。评估结果显示，在命令识别中表现出高准确性（F1: 0.90），人口统计估计准确率在性别上为0.89，年龄误差平均绝对值仅为5.14岁，并且在语音转录中的错误率为0.181。 通过将空中机器人技术与适应性对话人工智能及自包含视觉输出相结合，HoverAI引入了一种新型的具有空间感知和社会响应能力的实体代理。这种新技术适用于指导、协助和以人为中心的人机互动应用中。
URL
https://arxiv.org/abs/2601.13801
