Abstract
Autonomous 3D scanning of open-world target structures via drones remains challenging despite broad applications. Existing paradigms rely on restrictive assumptions or effortful human priors, limiting practicality, efficiency, and adaptability. Recent foundation models (FMs) offer great potential to bridge this gap. This paper investigates a critical research problem: What system architecture can effectively integrate FM knowledge for this task? We answer it with FlyCo, a principled FM-empowered perception-prediction-planning loop enabling fully autonomous, prompt-driven 3D target scanning in diverse unknown open-world environments. FlyCo directly translates low-effort human prompts (text, visual annotations) into precise adaptive scanning flights via three coordinated stages: (1) perception fuses streaming sensor data with vision-language FMs for robust target grounding and tracking; (2) prediction distills FM knowledge and combines multi-modal cues to infer the partially observed target's complete geometry; (3) planning leverages predictive foresight to generate efficient and safe paths with comprehensive target coverage. Building on this, we further design key components to boost open-world target grounding efficiency and robustness, enhance prediction quality in terms of shape accuracy, zero-shot generalization, and temporal stability, and balance long-horizon flight efficiency with real-time computability and online collision avoidance. Extensive challenging real-world and simulation experiments show FlyCo delivers precise scene understanding, high efficiency, and real-time safety, outperforming existing paradigms with lower human effort and verifying the proposed architecture's practicality. Comprehensive ablations validate each component's contribution. FlyCo also serves as a flexible, extensible blueprint, readily leveraging future FM and robotics advances. Code will be released.
Abstract (translated)
自主无人机三维扫描开放环境中的目标结构仍然面临诸多挑战，尽管其应用范围广泛。现有的方法依赖于严格的假设或需要大量的先验人工干预，这限制了其实用性、效率和适应能力。最近的基础模型（Foundation Models, FMs）为解决这些问题提供了巨大的潜力。本文探讨了一个关键的研究问题：何种系统架构能够有效地将基础模型知识整合到该任务中？我们通过提出FlyCo来回答这个问题——这是一个基于原则的自主感知-预测-规划循环框架，它利用了强大的基础模型能力，能够在多样且未知的开放环境中进行完全自主、指令驱动的目标三维扫描。 FlyCo直接将低投入的人类指令（如文本和视觉注释）转化为精确而适应性强的无人机飞行任务，通过三个协调阶段实现这一目标： 1. **感知**：该阶段融合了实时传感器数据与视觉-语言基础模型的知识，以确保对目标对象的稳定识别和跟踪。 2. **预测**：此步骤提取并利用基础模型知识，并结合多模态线索来推断部分可见目标的整体几何结构。 3. **规划**：这一环节利用预测性的洞察力生成高效且安全的飞行路径，确保全面覆盖目标物体。 基于上述架构，我们进一步设计了关键组件以提升开放环境下的目标定位效率和鲁棒性，并增强预测质量，特别是在形状准确性、零样本泛化能力和时间稳定性方面。此外，还优化了长时段任务的执行效率与实时计算能力以及在线避障之间的平衡。一系列广泛的现实世界和仿真实验显示，FlyCo在场景理解精度、运行效率及即时安全性等方面超越现有方法，并且需要较低的人工干预量，验证了所提出架构的实际可行性。 详细消融研究进一步证实了每个组件的贡献。此外，FlyCo还作为一个灵活、可扩展的设计蓝图存在，能够轻松利用未来的基础模型和机器人技术进步。代码将在适当时候发布。
URL
https://arxiv.org/abs/2601.07558
