Abstract
Autonomous drone racing (ADR) systems have recently achieved champion-level performance, yet remain highly specific to drone racing. While end-to-end vision-based methods promise broader applicability, no system to date simultaneously achieves full sim-to-real transfer, onboard execution, and champion-level performance. In this work, we present SkyDreamer, to the best of our knowledge, the first end-to-end vision-based ADR policy that maps directly from pixel-level representations to motor commands. SkyDreamer builds on informed Dreamer, a model-based reinforcement learning approach where the world model decodes to privileged information only available during training. By extending this concept to end-to-end vision-based ADR, the world model effectively functions as an implicit state and parameter estimator, greatly improving interpretability. SkyDreamer runs fully onboard without external aid, resolves visual ambiguities by tracking progress using the state decoded from the world model's hidden state, and requires no extrinsic camera calibration, enabling rapid deployment across different drones without retraining. Real-world experiments show that SkyDreamer achieves robust, high-speed flight, executing tight maneuvers such as an inverted loop, a split-S and a ladder, reaching speeds of up to 21 m/s and accelerations of up to 6 g. It further demonstrates a non-trivial visual sim-to-real transfer by operating on poor-quality segmentation masks, and exhibits robustness to battery depletion by accurately estimating the maximum attainable motor RPM and adjusting its flight path in real-time. These results highlight SkyDreamer's adaptability to important aspects of the reality gap, bringing robustness while still achieving extremely high-speed, agile flight.
Abstract (translated)
最近,自主无人机竞速(ADR)系统已经达到了冠军级别的表现,但这些系统仍高度特定于无人机竞速领域。虽然端到端视觉方法承诺更广泛的适用性,但迄今为止还没有一个系统能够同时实现从仿真环境到真实环境的完全迁移、在机上执行以及达到冠军级别性能。在这项工作中,我们介绍了SkyDreamer,在我们的知识范围内,它是第一个直接将像素级表示映射为电机指令的基于端到端视觉的ADR策略。SkyDreamer建立在知情梦想家的基础上,这是一种模型强化学习方法,其中世界模型解码成仅在训练期间可用的特权信息。通过将这一概念扩展到基于端到端视觉的ADR中,世界模型实际上充当了隐式状态和参数估计器,大大提高了可解释性。 SkyDreamer完全自主运行,无需外部支持,在飞行过程中利用从世界模型隐藏状态解码出的状态来解决视觉模糊问题,并且不需要外在相机校准,这使得它能够在不同无人机上快速部署而无需重新训练。真实世界的实验表明,SkyDreamer实现了稳健的高速飞行,执行诸如倒立环、分叉S和梯形等紧密机动动作,最高速度可达21米/秒,最大加速度可达6g。此外,通过在质量较差的分割掩码上操作并实时准确估计可达到的最大电机转速,并调整其飞行路径,SkyDreamer还展示了非平凡的视觉仿真到现实转移能力以及对电池耗尽的鲁棒性。 这些结果突显了SkyDreamer适应真实性和模拟环境之间重要差距方面的能力,在保持极高速度和敏捷飞行的同时实现稳健性能。
URL
https://arxiv.org/abs/2510.14783