【作者】 邓高峰； 王博； 何晓； 姚小兰； 费庆；

【Author】 Gaofeng Deng;Bo Wang;Xiao He;Xiaolan Yao;Qing Fei;School of Automation, Beijing Institute of Technology;Systems Engineering Research Institute, China State Shipbuilding Corporation;

【机构】 北京理工大学自动化学院； 中国船舶集团有限公司系统工程研究院；

【摘要】 多智能体强化学习系统通过群体协同在实现复杂任务时面临训练周期长、收敛速度慢问题,本文旨在通过强化学习模型融合方法实现多智能体系统多任务并行训练并加快其收敛速度。针对目前多智能体强化学习普遍存在的采样效率低、训练稳定性差及多任务强化学习存在的可扩展性差、灾难性遗忘、分心困境等问题,本文提出多任务强化学习模型融合方法框架,构建新型融合模型网络,通过平均权重融合、基于平均奖励的加权融合及基于梯度下降的加权融合等方式,实现不同强化学习训练基础模型的有效融合。最后通过设计面向集群跟踪的多智能体强化学习训练验证所提出的方法,结果表明本文所提方法能够实现不同模型间有效特征融合,同时适用于面向相同任务和不同任务的基础模型,且具有一定的鲁棒性。面向多智能体系统的多任务强化学习模型融合方法能够在保证模型性能的同时降低数据依赖,对提高多智能体强化学习的样本效率和收敛速度具有重要意义。更多还原

【Abstract】 Multi-agent reinforcement learning systems face problems of long training cycles and slow convergence speed when implementing complex tasks through group collaboration. This article aims to achieve multi-task parallel training of multi-agent systems and accelerate their convergence speed through reinforcement learning model fusion method. In response to the common problems of low sampling efficiency, poor training stability, poor scalability, catastrophic forgetting, and distraction dilemma in multi-agent reinforcement learning, this paper proposes a framework for the fusion method of multi-task reinforcement learning models, and constructs a new fusion model network. Through methods such as average weight fusion, weighted fusion based on average reward, and weighted fusion based on gradient descent, effectively integrate different reinforcement learning training basic models. Finally, the proposed method is validated through the design of multi-agent reinforcement learning training for cluster tracking. The results show that the proposed method can achieve effective feature fusion between different models, and is suitable for basic models facing the same and different tasks, with a certain degree of robustness. The multi-task reinforcement learning model fusion method for multi-agent systems can ensure model performance while reducing data dependence, which is of great significance for improving the sample efficiency and convergence speed of multi-agent reinforcement learning.更多还原