Abstract
This paper introduces VLN-Pilot, a novel framework in which a large Vision-and-Language Model (VLLM) assumes the role of a human pilot for indoor drone navigation. By leveraging the multimodal reasoning abilities of VLLMs, VLN-Pilot interprets free-form natural language instructions and grounds them in visual observations to plan and execute drone trajectories in GPS-denied indoor environments. Unlike traditional rule-based or geometric path-planning approaches, our framework integrates language-driven semantic understanding with visual perception, enabling context-aware, high-level flight behaviors with minimal task-specific engineering. VLN-Pilot supports fully autonomous instruction-following for drones by reasoning about spatial relationships, obstacle avoidance, and dynamic reactivity to unforeseen events. We validate our framework on a custom photorealistic indoor simulation benchmark and demonstrate the ability of the VLLM-driven agent to achieve high success rates on complex instruction-following tasks, including long-horizon navigation with multiple semantic targets. Experimental results highlight the promise of replacing remote drone pilots with a language-guided autonomous agent, opening avenues for scalable, human-friendly control of indoor UAVs in tasks such as inspection, search-and-rescue, and facility monitoring. Our results suggest that VLLM-based pilots may dramatically reduce operator workload while improving safety and mission flexibility in constrained indoor environments.
Abstract (translated)
本文介绍了VLN-Pilot，这是一个新颖的框架，在该框架中，一个大型视觉与语言模型（VLLM）扮演室内无人机导航的人类飞行员角色。通过利用VLLMs的多模态推理能力，VLN-Pilot可以解释自由形式的自然语言指令，并将其与视觉观察相结合以在没有GPS的室内环境中规划和执行无人机轨迹。不同于传统的基于规则或几何路径规划的方法，我们的框架将语言驱动的语义理解与视觉感知相结合，能够实现上下文感知、高层级飞行行为，同时减少了特定任务所需的工程量。 VLN-Pilot支持无人机完全自主地遵循指令，通过推理空间关系、障碍物规避以及对未预见事件的动态反应来完成任务。我们在一个定制的真实感室内模拟基准上验证了我们的框架，并展示了VLLM驱动代理在复杂指令执行任务中实现高成功率的能力，包括具有多个语义目标的长时导航。 实验结果强调了用语言引导的自主代理替代远程无人机飞行员的可能性，为在检查、搜索救援和设施监控等任务中控制室内UAV提供了一种可扩展且人性化的途径。我们的研究结果表明，基于VLLM的飞行员可能大幅减少操作员的工作量，并提高受限室内环境中的安全性和任务灵活性。
URL
https://arxiv.org/abs/2602.05552
