Abstract
Ensuring both optimality and safety is critical for the real-world deployment of agents, but becomes particularly challenging when the system dynamics are unknown. To address this problem, we introduce a notion of maximum safe dynamics learning via sufficient exploration in the space of safe policies. We propose a $\textit{pessimistically}$ safe framework that $\textit{optimistically}$ explores informative states and, despite not reaching them due to model uncertainty, ensures continuous online learning of dynamics. The framework achieves first-of-its-kind results: learning the dynamics model sufficiently $-$ up to an arbitrary small tolerance (subject to noise) $-$ in a finite time, while ensuring provably safe operation throughout with high probability and without requiring resets. Building on this, we propose an algorithm to maximize rewards while learning the dynamics $\textit{only to the extent needed}$ to achieve close-to-optimal performance. Unlike typical reinforcement learning (RL) methods, our approach operates online in a non-episodic setting and ensures safety throughout the learning process. We demonstrate the effectiveness of our approach in challenging domains such as autonomous car racing and drone navigation under aerodynamic effects $-$ scenarios where safety is critical and accurate modeling is difficult.
Abstract (translated)
确保代理在现实世界中的部署既具有最优性又安全至关重要,但在系统动力学未知的情况下,这一问题变得尤为棘手。为了解决这个问题,我们引入了一种通过充分探索安全策略空间来学习最大安全动态的概念。我们提出了一种“悲观”地保证安全但“乐观”地探索信息丰富状态的框架,在由于模型不确定性未能达到这些状态的情况下,仍能确保持续在线学习系统动力学。该框架实现了前所未有的成果:在有限时间内以任意小的容忍度(受噪声影响)充分学习动态模型,同时在整个过程中几乎可以肯定地保证安全操作,并且无需重置。在此基础上,我们提出了一种算法,在仅需达到接近最优性能所需的程度的情况下进行动态学习,从而最大化奖励。与典型的强化学习(RL)方法不同,我们的方法在非集中式设置中在线运行,并在整个学习过程中确保安全性。 我们在具有挑战性的领域如自动驾驶汽车赛车和受空气动力学影响的无人机导航中展示了我们方法的有效性——这些场景对安全性和精确建模提出了关键要求且难以实现。
URL
https://arxiv.org/abs/2509.16650