Abstract
Reinforcement learning (RL) algorithms can find an optimal policy for a single agent to accomplish a particular task. However, many real-world problems require multiple agents to collaborate in order to achieve a common goal. For example, a robot executing a task in a warehouse may require the assistance of a drone to retrieve items from high shelves. In Decentralized Multi-Agent RL (DMARL), agents learn independently and then combine their policies at execution time, but often must satisfy constraints on compatibility of local policies to ensure that they can achieve the global task when combined. In this paper, we study how providing high-level symbolic knowledge to agents can help address unique challenges of this setting, such as privacy constraints, communication limitations, and performance concerns. In particular, we extend the formal tools used to check the compatibility of local policies with the team task, making decentralized training with theoretical guarantees usable in more scenarios. Furthermore, we empirically demonstrate that symbolic knowledge about the temporal evolution of events in the environment can significantly expedite the learning process in DMARL.
Abstract (translated)
强化学习(RL)算法能够为单一智能体找到完成特定任务的最优策略。然而,许多现实世界的问题需要多个智能体协作以达成共同目标。例如,在仓库中执行任务的机器人可能需要无人机的帮助来从高架上取回物品。在分散式多代理强化学习(DMARL)中,各个智能体独立学习并在执行时结合各自的策略,但通常必须满足关于局部策略兼容性的约束条件,以确保它们可以实现整体目标。 在这篇论文中,我们研究了向各智能体提供高层次的符号知识如何能帮助解决这一设置中的独特挑战,例如隐私限制、通信限制和性能问题。具体而言,我们将用于检查局部策略与团队任务兼容性的形式化工具进行了扩展,使得带有理论保证的分散式训练在更多场景中得以应用。 此外,我们还通过实验证明了关于环境中事件时间演变的知识可以显著加快DMARL中的学习过程。
URL
https://arxiv.org/abs/2506.07829