Abstract
Multi-Agent Reinforcement Learning (MARL) is commonly deployed in settings where agents are trained via self-play with homogeneous teammates, often using parameter sharing and a single policy architecture. This opens the question: to what extent do self-play PPO agents learn general coordination strategies grounded in the underlying game, compared to overfitting to their training partners' behaviors? This paper investigates the question using the Heterogeneous Multi-Agent Challenge (HeMAC) environment, which features distinct Observer and Drone agents with complementary capabilities. We introduce Rotating Policy Training (RPT), an approach that rotates heterogeneous teammate policies of different learning algorithms during training, to expose the agent to a broader range of partner strategies. When playing alongside a withheld teammate policy (DDQN), we find that RPT achieves similar performance to a standard self-play baseline, IPPO, where all agents were trained sharing a single PPO policy. This result indicates that in this heterogeneous multi-agent setting, the IPPO baseline generalizes to novel teammate algorithms despite not experiencing teammate diversity during training. This shows that a simple IPPO baseline may possess the level of generalization to novel teammates that a diverse training regimen was designed to achieve.
Abstract (translated)
多智能体强化学习(MARL)通常在同质队友之间通过自博弈训练代理的环境中部署,常常使用参数共享和单一策略架构。这引发了一个问题:自我对局中的PPO智能体所学到的一般协调策略,在多大程度上基于底层游戏,而非过度拟合于其训练伙伴的行为?本文利用异构多智能体挑战(HeMAC)环境来研究这个问题,该环境中具有不同的观察者和无人机代理,它们具备互补的能力。我们引入了旋转策略培训(RPT),一种在训练期间交换不同学习算法的异构队友策略的方法,以使智能体接触更广泛的合作策略范围。当与一个被保留下来的队友策略(DDQN)一起游戏时,我们发现RPT达到了类似性能的标准自博弈基线IPPO的水平,在该基线下所有代理都是通过共享单一PPO策略进行训练的。这一结果表明,在这种异构多智能体设置中,即使没有经历队友多样性的训练,基于IPPO的基准也能推广到新颖的算法伙伴上。这显示了一个简单的IPPO基线可能具有实现多样化训练方案所设计的新颖队友泛化水平的能力。
URL
https://arxiv.org/abs/2512.08877