【作者】 仲宇；

【导师】 顾国昌；

【作者基本信息】 哈尔滨工程大学 ， 计算机应用技术， 2003， 博士

【摘要】 强化学习是近年发展起来的一种新的人工智能方法。由于人类对脑的真实结构了解甚少，至今无法清楚地解释智能的本质，只能依靠对人类智能系统的观察和猜测来定义人工智能系统的结构和外部特性，这种传统的基于逻辑规则的人工智能方法有诸多缺陷，所以人们转向了让人工智能系统通过自我学习来逐渐地获得智能的研究方向。强化学习就是这样一种机器学习方法，它模拟动物学习过程，通过试探的方法修正从状态到动作的映射策略，最终学会在各种环境状态下采取最好的反应行为，从而提高智能系统的自适应性和鲁棒性。 机器人是对人类个体的仿生，而多机器人系统是对人类社会的仿生。由于学习、交流和协作是人类的本质特征，所以对多机器人系统中的分布式强化学习问题的研究具有重要意义。现有的分布式强化学习方法还存在着结构信度分配困难、学习速度慢等缺陷，这些缺陷大大限制了分布式强化学习方法的应用范围。本文对分布式强化学习理论进行了系统地研究，并对其存在的部分问题提出了初步的解决办法。 系统地介绍了强化学习的基本原理、体系结构和主要算法，研究了各种强化学习算法的特点和适用范围。 研究了分布式强化学习系统的体系结构和主要算法，并根据体系结构的差异将其分为四类。介绍了这四类分布式强化学习系统的体系结构，形式化定义了它们的结构框架，描述了它们的主要组成部分的相互关系，讨论了它们的特点和适用范围等。 研究了分布式强化学习系统的结构信度分配方法。针对目前仅有的两种结构信度分配方法显失公平、不能真正体现智能体贡献大小的现象，提出了一种通过比较智能体动作的变化来分配信度的非线形规划方案，并通过对计算过程的分解获得了可行的结构信度分配算法，然后根据强化学习是一种增量式学习的特点将此算法修订为增量式的结构信度分配算法。 研究了独立强化学习系统内的信息共享方式，规范化了几种主要信息共享方式的算法流程，并比较了这几种主要信息共享方式的特点和适用范围。提出了基于有色轨道理论和基于状态分量重要性的两种状态空间划分方法，这两种方法可以压缩独立强化学习系统的学习空间，加快学习速度。 针对群体强化学习系统的学习空间随着智能体个数的增加而指数级膨胀的问题，提出了一种基于预测的群体强化学习算法。这种方法可以削减学习哈尔滨工程大学博士学位论文单元的冗余状态信息，降低学习空间的组合强度，加快群体强化学习算法的学习速度。 以多水下机器人的编队任务为例探讨了分布式强化学习在多机器人系统中的应用模式。仿真实验系统实现了多机器人的编队学习功能和避碰学习功能，实验结果表明机器人能够形成规整的队形，并能在通过障碍物或队形被打乱后迅速重新回到编队位置。更多还原

【Abstract】 Reinforcement Learning(RL) is a newly arising Artificial Intelligence(AI) means. Under the condition that the secret of brain remains concealed and the essence of intelligence remains unexplainable, traditional logic-based AI means have many shortages in defining the structures and characteristics of AI systems by observation and hypotheses. Therefore, people turn to the research field that AI systems learn to achieve intelligence gradually, RL is such a kind of machine learning means. RL devises the map policy from states to actions by "trial-error" principle and learns to react under all states, so that the adaptability and robustness of AI systems could be improved.A robot simulates a human being, while a multi-robot system simulates the human society. As learning, communication and collaboration are essential characteristics of human beings, so it is of great importance to perform researches on Distributed Reinforcement Learning(DRL) in multi-robot systems. Nevertheless, existing DRL algorithms suffer from the hardness of Structural Credit Assignment(SCA), the slowness of learning rate, and other problems, thus their application fields are strongly restricted. In this thesis, in-depth researches on DRL theory are made and the primary solutions of some existing problems are presented.The basic idea, architecture and main algorithms of RL are introduced systematically, and the specialties and application fields of RL algorithms are discussed.The architectures and main algorithms of DRL are researched, and the architectures of DRL are classified into four frameworks. The four frameworks are introduced and formalized, their specialties, their application fields and the relationship between their main components are researched.The SCA algorithms in DRL systems are researched. Considering the instance that the only two existing SCA means can not embody the contribution of agents, a non-linear programming SCA scheme based on comparison between agents’ actions is presented. After that, a feasible SCA algorithm is acquired by decompose the calculation course, which is subsequently devised into an incremental SCA algorithm to meet the fact that RL is a kind of incremental learning algorithm.The information sharing means in Reinforcement Learning Individually(RLI) systems are discussed, and the specialties and application fields of some main information sharing means are compared. A colored-orbit state segmentation algorithm and an element-importance state segmentation algorithm are presented, which can compress the scale of state space and accelerate the convergence of RLI algorithms.Considering the instance that the learning space of a Reinforcement Learning in Groups(RLG) system grows exponentially to the numbers of agents, a prediction-based RLG algorithm is presented. The new algorithm can cut down the redundant state information, so that the composition intensity of learning space is decreased and the convergence of the learning course is accelerated.Taking formation problem of multiple underwater vehicles as an example, the application model of DRL in multi-robot systems is illustrated. Collision-avoidance learning and formation learning are implemented in the simulation, and the simulation shows that the vehicles can learn to form a regular shape and reform the shape rapidly when they overpass obstacles or the shape is disturbed.更多还原