【作者】 邓凯；

【导师】 班晓军；

【作者基本信息】 哈尔滨工业大学 ， 控制科学与工程， 2018， 硕士

【摘要】 由于旋翼式飞行器具有机动性好、成本低和机械结构简单等特点,在近些年已经成为消费市场的新宠,无论是在民用还是军用领域,它都扮演着重要的角色。对于高校或一些科研机构来说,制作一台实际的旋翼式飞行器是不太现实的,尤其是对于控制学科。所以,为了对飞行姿态的控制方法进行试验验证,很多高校自主设计研发了一些模拟飞行姿态的实验平台,其中二自由度的飞行姿态模拟器最为常见。本文以实验室自主研发的一套二自由度飞行姿态模拟器系统作为研究对象,基于强化学习(Reinforcement Learning)算法对其进行控制算法设计。强化学习方法的优势为在于它不需要直接建立被控对象的动力学模型,仅仅依据系统的输入输出数据即可完成对控制率参数的更新,并且通过迭代能够使整个闭环系统工作在某种指标意义下的最优状态。首先,本文研究了无模型(Model-free)的强化学习算法。利用DQN(Deep Q Network)算法对飞行姿态模拟系统进行了控制算法设计,首先利用值函数对当前时刻的状态做一个评估,再通过贪婪策略来进行动作的选取,数值仿真结果表明,经过一定时间的学习之后,能达到良好的控制效果。但是,该算法只适用于离散的情况,所以针对实际的连续控制系统,又采用Policy-Gradient算法进行算法设计,建立了一个策略网络,用环境给出的奖励信息来评价由神经网络算出的梯度信息,从而进行训练。此算法由于是回合更新的机制,具有数据利用率不高的缺点,需要耗费大量时间去进行训练。为克服以上两种方法的缺点,采用DDPG(Deep Deterministic Policy Gradient)算法设计了控制算法。此方法融合了DQN算法和Policy-Gradient算法的优点,建立了策略网络和值网络,用值网络输出的Q值来评判策略网络输出的梯度,从而很大程度上提高了学习的效率,并且达到了更好的控制效果。进而,对于飞行姿态模拟器这类实际的机械系统,本文研究了具有更好效果的基于模型(Model-based)的强化学习算法。采用PILCO(probabilistic inference for learning control)算法来学习出径向基网络控制器的参数,先用高斯过程拟合出系统的概率模型,然后再根据这个概率学模型来进行状态的预测,由预测的状态算出预测的奖励信息,然后根据这个奖励信息来进行控制器参数的优化。最后,本文提出一种基于PILCO算法框架的整定PID控制器参数的方法,很好的解决了PID参数调节的难题。本文将此方法进行了数值仿真和实际试验的实现,达到了理论与实际结合的目的。更多还原

【Abstract】 Due to rotorcraft’s good maneuverability,low cost,and simple mechanical structure,rotorcraft has been popular in the consumer market in recent years.It plays an important role in both civil and military applications.For a university or some scientific research institutions,it is not realistic to manufacture an actual rotorcraft,especially within the control field.Therefore,in order to verify the flight attitude control method,many universities have independently designed and developed some experimental platforms for simulating flight attitudes,of which the two-degree-of-freedom flight attitude simulator is the most common.In this paper,the research object is a two-degree-of-freedom flight attitude simulator system developed by the laboratory,and the controller is designed based on Reinforcement Learning algorithm.Since the dynamic model of the controlled object is not required,the advantages of the reinforcement learning method are the updating of the controller’s parameters is only based on the input and output data of the system,and iteration makes the entire closed-loop system working in an optimal state on some conditions.First,this paper studies the Model-free reinforcement learning algorithm.The DQN(Deep Q Network)algorithm is used to design the controller of the flight attitude simulator.Firstly,a value function is used to evaluate the status of the current moment,and then the action is selected through a greedy strategy.Numerical simulation results show that after a certain time of learning,a good control performance can be achieved.However,this algorithm is only applicable to discrete cases.Therefore,for the actual continuous control system,the Policy-Gradient algorithm is used to design and a strategy network is established.In order to conduct training,the reward information given by the environment is used as the error of the neural network to transmit backwards.Because this algorithm is a round-up updating mechanism,it has the disadvantage of low data utilization rate and it takes a lot of time to train.In order to overcome the shortcomings of the above two methods,a controller of the system was designed using the DDPG(Deep Deterministic Policy Gradient)algorithm.This method combines the advantages of the DQN algorithm and the Policy-Gradient algorithm,by establishing a strategy network and a value network,and it uses the Q value output from the value network to judge the gradient of the strategy network output,so that the efficiency of learning is greatly improved,and a better control effect is achieved.Furthermore,considering practical mechanical systems such as flight attitude simulators,this paper studies model-based reinforcement learning algorithms with better performance.Using PILCO(probabilistic inference for learning control)algorithm to learn the parameters of the radial-based network controller,the Gaussian process is used to fit the system’s probability model,and then the state prediction is performed according to this probability model.The state calculates predicted reward information and then optimizes the controller parameters based on this reward information.Finally,based on PILCO algorithm framework,a method of adjusting PID controller’s parameters is proposed,which solves the problem of PID parameter adjustment.In this paper,the numerical simulation and actual experiment of this method are realized,which achieves the purpose of combining theory and practice.更多还原