Abstract
Preference-based reinforcement learning (PbRL) enables agents to learn control policies without requiring manually designed reward functions, making it well-suited for tasks where objectives are difficult to formalize or inherently subjective. Acrobatic flight poses a particularly challenging problem due to its complex dynamics, rapid movements, and the importance of precise execution. In this work, we explore the use of PbRL for agile drone control, focusing on the execution of dynamic maneuvers such as powerloops. Building on Preference-based Proximal Policy Optimization (Preference PPO), we propose Reward Ensemble under Confidence (REC), an extension to the reward learning objective that improves preference modeling and learning stability. Our method achieves 88.4% of the shaped reward performance, compared to 55.2% with standard Preference PPO. We train policies in simulation and successfully transfer them to real-world drones, demonstrating multiple acrobatic maneuvers where human preferences emphasize stylistic qualities of motion. Furthermore, we demonstrate the applicability of our probabilistic reward model in a representative MuJoCo environment for continuous control. Finally, we highlight the limitations of manually designed rewards, observing only 60.7% agreement with human preferences. These results underscore the effectiveness of PbRL in capturing complex, human-centered objectives across both physical and simulated domains.
Abstract (translated)
基于偏好的强化学习(PbRL)使智能体能够在无需手动设计奖励函数的情况下学习控制策略,因此非常适合目标难以形式化或本质上具有主观性的任务。特技飞行因其复杂的动力学特性、快速的运动以及执行精确度的重要性而构成一个特别具挑战性的问题。在这项工作中,我们探讨了将PbRL用于敏捷无人机控制的可能性,并专注于动态机动动作(如螺旋筋斗)的执行。在基于偏好的近端策略优化(Preference PPO)的基础上,我们提出了置信度下的奖励集成(REC),这是一种对奖励学习目标进行扩展的方法,能够改进偏好建模和学习稳定性。我们的方法实现了88.4%的塑形奖励性能,而标准的Preference PPO仅实现55.2%。 我们在模拟环境中训练策略,并成功将其转移到真实的无人机上,展示了多个强调运动风格质量的人类偏好的特技动作。此外,我们还在一个具有代表性的MuJoCo环境中演示了我们的概率性奖励模型在连续控制任务中的适用性。最后,我们指出了手动设计奖励的局限性,观察到仅达到60.7%与人类偏好一致的程度。 这些结果强调了PbRL在捕捉复杂且以人为中心的目标方面的有效性,无论是在物理环境还是模拟环境中都是如此。
URL
https://arxiv.org/abs/2508.18817