节点文献
基于强化学习的分布式光伏产消用户端对端电能交易决策研究
Research on Peer to Peer Electricity Transaction Decision of Prosumers with Distributed Photovoltaic Energy Based on Reinforcement Learning
【作者】 刘博；
【导师】 王丹；
【作者基本信息】 天津大学 ， 电气工程， 2020， 硕士
【摘要】 近年来,光伏、储能等分布式电源在用户侧广泛使用,传统被动的能源消费用户正转变为“产消用户”。端对端(Peer to Peer,P2P)电能交易,是解决光伏补贴问题,促进光伏消纳利用,降低用户用电成本并提高售电收益的有效途径。在市场参与度低、缺少第三方组织管理、信息不完整、优化问题难以分析的小规模分布式光伏用户群实现P2P交易决策方案的制定具有重要研究意义。强化学习是一种促进智能体在与环境交互过程中通过学习策略达成回报最大化的人工智能方法,在不进行优化计算和不充分了解市场机制的情况下,该方法非常适合处理小规模用户电能交易行为。因此本文基于上述背景,研究了基于强化学习的分布式光伏产消用户P2P电能交易决策,主要工作如下:(1)研究了分布式光伏产消用户的P2P电能交易模型。考虑用户侧资源特点,建立了P2P电能交易主体模型和用户侧设备资源模型;考虑小规模电能交易市场特点,建立了基于供需比的交易电价制定模型;考虑交易过程的物理网络约束,建立了P2P电能交易物理约束模型。(2)提出了单个配有储能产消用户静态场景下P2P电能交易的马尔可夫决策过程(Markov Decision Process,MDP)模型和离线强化学习求解算法。建立了包含状态空间、动作空间、奖励函数和状态转移率的MDP模型;对储能连续空间进行均匀离散采样,建立了适用于工程应用的离散MDP状态和动作空间;提出基于时序差分离线控制的UDBA-Q强化学习改进算法。(3)提出了多个配有储能产消用户动态场景下P2P电能交易的改进MDP模型和在线强化学习求解算法。针对多个配有储能产消用户场景,建立动态空间的MDP改进模型;提出基于时序差分在线控制的UDBA-SARSA改进强化学习方法;分析对比了两种算法的选取和应用特点。(4)基于区块链即服务平台,设计了针对用户侧本地运行、电能交易决策和用户结算信息的信息记录与保护流程与方案。并通过信息记录保护实例,实现了基于区块链技术的电能交易信息记录保护管理。
【Abstract】 In recent years,photovoltaic,energy storage and other distributed generation are widely used in the user side,and the traditional passive energy consumers are changing into “prosumers”.Peer to peer(P2P)electricity transaction is an effective way to solve the problems of photovoltaic subsidy,promoting photovoltaic energy consumption and utilization,reduce electricity consumption cost and increase electricity sales income.It is of great research significance to realize the decision making of P2 P electricity transaction in the small-scale distributed photovoltaic users group with low market participation,lack of third-party organization management,incomplete information,and difficult to analyze optimization problems.Reinforcement learning(RL)is an artificial intelligence method to maximize the return of intelligent agents through learning strategies in the process of interaction with the environment.Without the optimal calculation and the full understanding of market mechanism,this method is very suitable for prosumers dealing with the energy transaction behavior of small-scale users.Therefore,based on the above background,this thesis studies the P2 P electricity transaction decision of prosumers with distributed photovoltaic energy based on RL.The main work of this thesis is as following:(1)The P2 P electricity transaction model of prosumers with distributed photovoltaic energy is studied.Considering the characteristics of user side resources,a P2 P electricity transaction model of participants and a user side equipments model are established.Considering the characteristics of small-scale electricity transaction market,a transaction price model based on supply and demand ratio is established.Considering the physical network constraints of transaction process,a physical constraint model of P2 P electricity transaction is established.(2)A Markov decision process(MDP)model and an off-policy RL algorithm for P2 P electricity transaction in a static scenario with single energy storage prosumers are proposed.The MDP model including state space,action space,reward function and state transition rate is established.The discrete MDP state and action space for engineering applications are established by uniformly discrete sampling in the continuous energy storage space.An improved RL algorithm of UDBA-SARSA based on temporal difference off-policy control is proposed.(3)An improved MDP model and an on-policy RL algorithm for P2 P electricity transaction in dynamic scenario with multiple energy storage generation prosumers is proposed.The improved MDP model of dynamic space is established for scenario with multiple prosumers with energy storage.An improved RL method of UDBA-SARSA based on temporal difference on-policy control is proposed.The selection and application characteristics of the two algorithms are analyzed and compared.(4)Based on the blockchain as a service platform,the recording and protection process and scheme for the local operation,electricity transation decision-making and settlement information of user side are designed.Through the example of information record protection,the protection and management of electric electricity transaction information record based on blockchain technology is realized.