朱杰杰,皮志勇,陈代才,谭洪
ZHU Jiejie,PI Zhiyong,CHEN Daicai,TAN Hong

• 1:三峡大学电气与新能源学院
• 2:国网湖北省电力有限公司荆门供电公司
• 3:国网湖北省电力有限公司利川供电公司

摘要(Abstract)：

针对电-热综合能源系统中可再生能源输出的不确定性和间歇性问题，提出一种电-热综合能源系统能量管理的强化学习方法，以电-热综合能源系统运行成本最低为目标，实现综合能源系统的能量管理。首先，建立电-热综合能源系统能量管理模型；然后，将含可再生能源的电-热综合能源系统能量管理过程转化为马尔可夫决策过程，并采用融合NoisyNet（噪声网络）和自注意力机制的DQN-CE（深度Q网络-交叉熵）算法对智能体进行交互学习训练。最后，通过算例分析表明，所提方法训练的智能体能够实时响应可再生能源的不确定性，并能在线管理包含可再生能源在内的电-热综合能源系统的能量。
To address the uncertainty and intermittency of renewable energy output in integrated electricity-thermal energy systems, a reinforcement learning method for energy management is proposed, aiming to minimize the operating costs of the system. First, an energy management model for the integrated electricity-thermal energy system is established. Next, the energy management process of the system, which includes renewable energy, is transformed into a Markov decision process(MDP). The DQN-CE(Deep Q-Network with cross-entropy) algorithm, integrating NoisyNet and a self-attention mechanism, is then used to train the agent through interactive learning. Finally, case study analysis shows that the agent trained using the proposed method can respond in real time to the uncertainties of renewable energy and manage the energy of the integrated electricity-thermal energy system with renewable sources online.

关键词(KeyWords)： 噪声网络;深度Q网络;自注意力机制;交叉熵损失函数
NoisyNet;Deep Q-network;self-attention mechanism;cross-entropy loss function

Abstract:

Keywords:

基金项目(Foundation): 湖北省自然科学基金创新发展联合基金项目（2024AFD350）

作者(Author): 朱杰杰,皮志勇,陈代才,谭洪
ZHU Jiejie,PI Zhiyong,CHEN Daicai,TAN Hong

DOI: 10.19585/j.zjdl.202501005

参考文献(References)：

• [1] WANG R,CHENG S,ZUO X W,et al.Optimal management of multi stakeholder integrated energy system considering dual incentive demand response and carbon trading mechanism[J].International Journal of Energy Research,2022,46(5):6246-6263.
• [2]赵佩尧，李正烁，高晗，等.电-气-热综合能源系统协同调度优化研究综述[J].山东电力技术，2024,51(4):1-11.ZHAO Peiyao,LI Zhengshuo,GAO Han,et al.Review on collaborative scheduling optimization of electricity-gasheat integrated energy system[J]. Shandong Electric Power,2024,51(4):1-11.
• [3]周专，苗帅，边家瑜，等.基于系统动力学氢需求预测与综合能源系统优化配置研究[J].电力电容器与无功补偿，2023,44(6):12-22.ZHOU Zhuan,MIAO Shuai,BIAN Jiayu,et al.Hydrogen demand prediction based on system dynamics and research on optimal configuration of integrated energy system[J].Power Capacitor&Reactive Power Compensation,2023,44(6):12-22.
• [4] SONG D R,MENG W Q,DONG M,et al.A critical survey of integrated energy system:summaries,methodologies and analysis[J].Energy Conversion and Management,2022,266:115863.
• [5] LYU X M,LIU T Q,LIU X,et al. Low-carbon robust economic dispatch of park-level integrated energy system considering price-based demand response and vehicle-togrid[J].Energy,2023,263:125739.
• [6] NIKAS A,GAMBHIR A,TRUTNEVYTE E,et al.Perspective of comprehensive and comprehensible multimodel energy and climate science in Europe[J].Energy,2021,215:119153.
• [7]员盼锋，徐舒涵，丹慧杰，等.基于火电厂源侧的综合能源系统集成及优化配置研究[J].动力工程学报，2024,44(4):650-657.YUN Panfeng,XU Shuhan,DAN Huijie,et al.Integration and optimal allocation of integrated energy system based on source side of thermal power plant[J].Journal of Chinese Society of Power Engineering,2024,44(4):650-657.
• [8]彭春华，郑聪，陈婧，等.基于置信间隙决策的综合能源系统鲁棒优化调度[J].中国电机工程学报，2021,41(16):5593-5603.PENG Chunhua,ZHENG Cong,CHEN Jing,et al. Robust optimal dispatching of integrated energy system based on confidence gap decision[J].Proceedings of the CSEE,2021,41(16):5593-5603.
• [9]张志一，窦震海，于润泽，等.考虑电-热等效虚拟储能的综合能源系统低碳经济调度[J].电力建设，2024,45(3):16-26.ZHANG Zhiyi,DOU Zhenhai,YU Runze,et al. Lowcarbon economic dispatch of integrated energy system considering electric-thermal equivalent virtual energy storage[J].Electric Power Construction,2024,45(3):16-26.
• [10]王佳蕊，孙勇，胡枭，等.基于MICP的多能耦合综合能源系统可再生能源消纳能力研究[J].电力建设，2023,44(8):157-170.WANG Jiarui,SUN Yong,HU Xiao,et al.Research on renewable energy absorption capacity of multi-energy coupling integrated energy systems based on MICP[J].Electric Power Construction,2023,44(8):157-170.
• [11]柳松林，成贵学，赵晋斌，等.基于双层优化的电-热综合能源系统调峰运行方法[J].智慧电力，2023,51(5):65-72.LIU Songlin,CHENG Guixue,ZHAO Jinbin,et al.Peakshaving operation method of electric-thermal integrated energy system based on double-layer optimization[J].Smart Power,2023,51(5):65-72.
• [12] QADRDAN M,WU J Z,JENKINS N,et al. Operating strategies for a GB integrated gas and electricity network considering the uncertainty in wind power forecasts[J].IEEE Transactions on Sustainable Energy,2014,5(1):128-138.
• [13]高晓松，李更丰，肖遥，等.基于分布鲁棒优化的电-气-热综合能源系统日前经济调度[J].电网技术，2020,44(6):2245-2254.GAO Xiaosong,LI Gengfeng,XIAO Yao,et al. Dayahead economic scheduling of electric-gas-thermal integrated energy system based on the optimization of dividing rod[J].Power grid technology,2020,44(6):2245-2254.
• [14] WANG Q J,XIAO Y F,TAN H,et al.Day-Ahead scheduling of rural integrated energy systems based on distributionally robust optimization theory[J]. Applied Thermal Engineering,2024,246:123001.
• [15]任萱，李桐歌，马骏毅，等.考虑荷电状态的光伏微电网混合储能容量优化配置[J].电测与仪表，2024,61(2):150-156.REN Xuan,LI Tongge,MA Junyi,et al.Optimal configuration of hybrid energy storage capacity of photovoltaic microgrid considering the state of charge[J].Electrical measurement and instrumentation,2024,61(2):150-156.
• [16]张晗，杨继斌，张继业，等.基于多种群萤火虫算法的车载燃料电池直流微电网能量管理优化[J].中国电机工程学报，2021,41(3):833-845.ZHANG Han,YANG Jibin,ZHANG Jiye,et al.Multiplepopulation firefly algorithm-based energy management strategy for vehicle-mounted fuel cell DC microgrid[J].Proceedings of the CSEE,2021,41(3):833-845.
• [17]冯昌森，张瑜，文福拴，等.基于深度期望Q网络算法的微电网能量管理策略）[J].电力系统自动化，2022,46(3):14-22.FENG Changsen,ZHANG Yu,WEN Fusuan,et al.Energy management strategy of microgrid based on deep expectation Q network algorithm[J].Power system automation,2022,46(3):14-22.
• [18]李扬，马文捷，卜凡金，等.多智能体深度强化学习驱动的跨园区能源交互优化调度[J].电力建设，2024,45(5):59-70.LI Yang,MA Wenjie,BU Fanjin,et al. Deep reinforcement learning-driven cross-community energy interaction optimal scheduling[J].Electric Power Construction,2024,45(5):59-70.
• [19]黎海涛，申保晨，杨艳红，等.基于改进竞争深度Q网络算法的微电网能量管理与优化策略[J].电力系统自动化，2022,46(7):42-49.LI Haitao,SHEN Baochen,YANG Yanhong,et al. Energy management and optimization strategy for microgrid based on improved dueling deep Q network algorithm[J].Automation of Electric Power Systems,2022,46(7):42-49.
• [20]雷嘉明，姜爱华，吴新飞，等.计及源荷不确定性的综合能源系统近端策略优化调度[J].电力科学与技术学报，2023,38(5):1-11.LEI Jiaming,JIANG Aihua,WU Xinfei,et al. Proximal policy optimization dispatch of integrated energy system considering source-load uncertainty[J].Journal of Electric Power Science and Technology,2023,38(5):1-11.
• [21]刘倩，王馨，朱刘柱，等.基于深度强化学习的园区级综合能源系统双层规划策略[J/OL].武汉大学学报（工学版），1-13[2024-05-10].http：//kns.cnki.net/kcms/detail/42.1675.T.20240326.1452.002.html.LIU Qian,WANG Xin,ZHU Liuzhu,et al. A two-layer planning strategy for integrated energy systems at the park level based on deep reinforcement learning[J/OL].Journal of Wuhan University(Engineering),1-13[2024-05-10].http：//kns. cnki. net/kcms/detail/42.1675. T. 20240326.1452.002.html.
• [22]王新迎，赵琦，赵黎媛，等.基于深度Q学习的电热综合能源系统能量管理[J].电力建设，2021,42(3):10-18.WANG Xinying,ZHAO Qi,ZHAO Liyuan,et al.Energ-y management of electrothermal integrated energy system based on deep Q learning[J].Electric power construction,2021,42(3):10-18.
• [23]刘俊峰，陈剑龙，王晓生，等.基于深度强化学习的微能源网能量管理与优化策略研究[J].电网技术，2020,44(10):3794-3803.LIU Junfeng,CHEN Jianlong,WANG Xiaosheng,et al.Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology,2020,44(10):3794-3803.