Abstract
We propose SketchPlan, a diffusion-based planner that interprets 2D hand-drawn sketches over depth images to generate 3D flight paths for drone navigation. SketchPlan comprises two components: a SketchAdapter that learns to map the human sketches to projected 2D paths, and DiffPath, a diffusion model that infers 3D trajectories from 2D projections and a first person view depth image. Our model achieves zero-shot sim-to-real transfer, generating accurate and safe flight paths in previously unseen real-world environments. To train the model, we build a synthetic dataset of 32k flight paths using a diverse set of photorealistic 3D Gaussian Splatting scenes. We automatically label the data by computing 2D projections of the 3D flight paths onto the camera plane, and use this to train the DiffPath diffusion model. However, since real human 2D sketches differ significantly from ideal 2D projections, we additionally label 872 of the 3D flight paths with real human sketches and use this to train the SketchAdapter to infer the 2D projection from the human sketch. We demonstrate SketchPlan's effectiveness in both simulated and real-world experiments, and show through ablations that training on a mix of human labeled and auto-labeled data together with a modular design significantly boosts its capabilities to correctly interpret human intent and infer 3D paths. In real-world drone tests, SketchPlan achieved 100\% success in low/medium clutter and 40\% in unseen high-clutter environments, outperforming key ablations by 20-60\% in task completion.
Abstract (translated)
我们提出了SketchPlan,这是一种基于扩散模型的规划器,它能够解析深度图像上的二维手绘草图以生成无人机导航的三维飞行路径。SketchPlan 包含两个组成部分:一个名为 SketchAdapter 的组件,用于学习将人类绘制的草图映射到投影的二维路径;另一个是 DiffPath,这是一个从二维投影和第一人称视角深度图像推断三维轨迹的扩散模型。 我们的模型实现了零样本仿真到现实的迁移能力,在以前未见过的真实世界环境中生成准确且安全的飞行路径。为了训练该模型,我们构建了一个包含32,000条合成飞行路径的数据集,这些数据是基于多样化的逼真三维高斯点云场景创建的。通过计算三维飞行路径在相机平面上的二维投影来自动标注数据,并利用这些标注来训练 DiffPath 扩散模型。 然而,由于实际的人类绘制的二维草图与理想的二维投影存在显著差异,我们还对其中872条三维飞行路径使用真实人类草图进行人工标注,并用此来训练 SketchAdapter 以从人类草图中推断出二维投影。我们在模拟和现实世界实验中展示了 SketchPlan 的有效性,并通过消融研究证明,在混合了人类标签数据和自动标签数据的培训下,以及采用模块化设计的方式能够显著提升其准确解读人类意图并推断三维路径的能力。 在真实世界的无人机测试中,SketchPlan 在低/中等复杂度环境中实现了100%的成功率,并且在未见过的高复杂度环境下达到了40%的成功率。与关键消融版本相比,在任务完成方面,SketchPlan 的表现要高出20%-60%。
URL
https://arxiv.org/abs/2510.03545