曾磊,丁泉,陈孝煜,岳娴雅
ZENG Lei,DING Quan,CHEN Xiaoyu,YUE Xianya

• 1:国电南京自动化股份有限公司
• 2:东南大学电气工程学院

摘要(Abstract)：

针对当前微电网中的源荷不确定性及调度策略灵活性不足的问题，提出基于深度强化学习的微电网运行优化方法。首先，构建包含光伏、储能和发电设备的微电网模型及其约束条件。其次，以降低系统运行成本和不平衡度为考量，构建多目标优化框架，同时考虑光伏发电、负荷需求和电价等不确定性因素，采用TD3（双延迟深度确定性策略梯度）算法，基于数据驱动的方式获得微电网调度策略。然后，通过在奖励函数中添加高比例错误动作惩罚项将各设备的出力约束在合理范围内，降低强化学习方法缺乏安全约束保障的风险。最后，仿真结果表明，与DDPG（深度确定策略梯度）算法相比，所提方法在经济性和稳定性方面表现更优，其经济成本更接近于理想状态下的确定性优化方法。
To address the challenges of source-load uncertainty and insufficient scheduling flexibility in microgrids, an optimization method for microgrid operation based on deep reinforcement learning(DRL) is proposed. First, a microgrid model comprising photovoltaic(PV), energy storage, and generation equipment is constructed, along with its constraint conditions. Second, a multi-objective optimization framework is established to minimize operating costs and imbalance of the system, considering uncertainties such as PV generation, load demand, and electricity prices. The twin delayed deep deterministic policy gradient(TD3) algorithm is employed to derive microgrid scheduling strategies in a data-driven manner. Third, a penalty term for high-proportion erroneous actions is incorporated into the reward function to constrain the output of each device within a reasonable range, mitigating the risk of insufficient safety guarantees inherent in reinforcement learning methods. Finally, simulation results demonstrate that, compared to the deep deterministic policy gradient(DDPG) algorithm, the proposed method achieves superior economic efficiency and stability, with economic costs closer to those of ideal deterministic optimization methods.

关键词(KeyWords)： 深度强化学习;微电网;优化调度;双延迟深度确定性策略梯度;成本优化
DRL;microgrid;optimal scheduling;TD3;cost optimization

Abstract:

Keywords:

基金项目(Foundation): 上海市科学技术委员会科研计划项目（19DZ1205704）

作者(Author): 曾磊,丁泉,陈孝煜,岳娴雅
ZENG Lei,DING Quan,CHEN Xiaoyu,YUE Xianya

DOI: 10.19585/j.zjdl.202506003

参考文献(References)：

• [1] CHEN H P,GAO L,ZHANG Z.Multi-objective optimal scheduling of a microgrid with uncertainties of renewable power generation considering user satisfaction[J].International Journal of Electrical Power&Energy Systems,2021,131:107142.
• [2]马燕峰，谢家荣，赵书强，等.考虑园区综合能源系统接入的主动配电网多目标优化调度[J].电力系统自动化，2022,46(13):53-61.MAYanfeng,XIEJiarong,ZHAOShuqiang,et al. Multiobjective optimal dispatching for active distribution network considering park-level integrated energy system[J].Automation of Electric Power Systems,2022,46(13):53-61.
• [3]李子凯，杨波，周忠堂，等.基于强化学习算法的微电网优化策略[J].山东电力技术，2024,51(6):27-35.LI Zikai,YANG Bo,ZHOU Zhongtang,et al. Optimization strategy for microgrid based on reinforcement learning algorithm[J].Shandong Electric Power,2024,51(6):27-35.
• [4]马立红，梁亚峰，程西，等.计及构网型储能稳定拓展的微电网群优化运行[J].电力工程技术，2024,43(6):214-222.MA Lihong,LIANG Yafeng,CHENG Xi,et al.Optimal operation of microgrids considering stabilized expansion of grid-forming energy storage[J].Electric Power Engineering Technology,2024,43(6):214-222.
• [5]魏震波，张芷琪，李银江，等.多主体合作模式下微电网规划运行一体化模型[J].电力建设，2024,45(10):47-58.WEI Zhenbo,ZHANG Zhiqi,LI Yinjiang,et al.An integrated model of microgrid planning and operation under A multi-subject cooperation model[J]. Electric Power Construction,2024,45(10):47-58.
• [6]李嘉伟，巨云涛，张璐，等.基于分布鲁棒模型预测控制的微电网多时间尺度优化调度[J].电力工程技术，2024,43(4):45-55.LI Jiawei,JU Yuntao,ZHANG Lu,et al.Multi-time scale optimal scheduling of microgrid based on distributed robust model predictive control[J].Electric Power Engineering Technology,2024,43(4):45-55.
• [7]樊晓伟，王瑞妙，杨海峰，等.计及源荷不确定的综合能源微电网集群优化运行[J].电力建设，2024,45(8):128-139.FAN Xiaowei,WANG Ruimiao,YANG Haifeng,et al.Optimization operation of integrated energy microgrid cluster considering source-load uncertainty[J].Electric Power Construction,2024,45(8):128-139.
• [8]王涛，钟浩，李世春，等.基于主从博弈的多微电网储能容量优化配置[J].智慧电力，2023,51(1):9-15.WANG Tao,ZHONG Hao,LI Shichun,et al.Optimal allocation of energy storage capacity in multi-microgrid based on master-slave game[J]. Smart Power,2023,51(1):9-15.
• [9]刘任，刘洋，许立雄，等.计及分布式需求响应的多微电网系统协同优化策略[J].电力建设，2023,44(5):72-83.LIU Ren,LIU Yang,XU Lixiong,et al. Multi-microgrid system collaborative optimization strategy considering distributed demand response[J]. Electric Power Construction,2023,44(5):72-83.
• [10]向真，李振聪，谈赢杰，等.计及能量共享的多微电网与配电系统两层协同优化调度方法[J].智慧电力，2024,52(8):42-49.XIANG Zhen,LI Zhencong,TAN Yingjie,et al.Bi-level collaborative optimal scheduling method for multiple microgrids and distribution system considering energy sharing[J].Smart Power,2024,52(8):42-49.
• [11]孙昕，刘景延，赵冬雪，等.基于可再生能源不确定性的多能源微网调度优化模型[J].科学技术与工程，2020,20(35):14523-14529.SUN Xin,LIU Jingyan,ZHAO Dongxue,et al.Scheduling optimization model of multi-energy micro grid based on uncertainty of renewable rnergy[J].Science Technology and Engineering,2020,20(35):14523-14529.
• [12]马恺珧，王国庆，于雷.不确定性环境下微电网优化调度综述[J].工程研究（跨学科视野中的工程），2023,15(2):93-103.MA Kaiyao,WANG Guoqing,YU Lei. Review on optimal scheduling of microgrids under uncertainty[J].Journal of Engineering Studies,2023,15(2):93-103.
• [13]谭九鼎，李帅兵，李明澈，等.计及不确定性的分布式微网参与电网优化调度方法综述[J].综合智慧能源，2024,46(1):38-48.TAN Jiuding,LI Shuaibing,LI Mingche,et al.Optimized scheduling of the power grid with participation of distributed microgrids considering their uncertainties[J].Huadian Technology,2024,46(1):38-48.
• [14] LI Y,YANG Z,LI G Q,et al.Optimal scheduling of isolated microgrid with an electric vehicle battery swapping station in multi-stakeholder scenarios:a bi-level programming approach via real-time pricing[J]. Applied Energy,2018,232:54-68.
• [15]张国平，王维俊，毛龙波，等.微电网经济运行优化方法研究综述[J].电力与能源，2019,40(5):585-590.ZHANG Guoping,WANG Weijun,MAO Longbo,et al.Overview of microgrid economic operation optimization method[J].Power&Energy,2019,40(5):585-590.
• [16]吴定会，高聪，纪志成.混合粒子群算法在微电网经济优化运行的应用[J].控制理论与应用，2018,35(4):457-467.WU Dinghui,GAO Cong,JI Zhicheng.Economic optimization operation of the microgrid using the hybrid particle swarm optimization algorithm[J].Control Theory&Applications,2018,35(4):457-467.
• [17]鲁卓欣，徐潇源，严正，等.不确定性环境下数据驱动的电力系统优化调度方法综述[J].电力系统自动化，2020,44(21):172-183.LU Zhuoxin,XU Xiaoyuan,YAN Zheng,et al.Overview on data-driven optimal scheduling methods of power system in uncertain environment[J]. Automation of Electric Power Systems,2020,44(21):172-183.
• [18]赵鹏杰，吴俊勇，王燚，等.基于深度强化学习的微电网优化运行策略[J].电力自动化设备，2022,42(11):9-16.ZHAO Pengjie,WU Junyong,WANG Yi,et al.Optimal operation strategy of microgrid based on deep reinforcement learning[J].Electric Power Automation Equipment,2022,42(11):9-16.
• [19] YI Z G,LUO Y S,WESTOVER T,et al.Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system[J]. Applied Energy,2022,328:120113.
• [20] NAKABI T A,TOIVANEN P.Deep reinforcement learning for energy management in a microgrid with flexible demand[J].Sustainable Energy,Grids and Networks,2021,25:100413.
• [21]毕聪博，唐聿劼，罗永红，等.电力系统优化控制中强化学习方法应用及挑战[J].中国电机工程学报，2024,44(1):1-21.BI Congbo,TANG Yujie,LUO Yonghong,et al.Application and challenge of reinforcement learning method in optimal control of power system[J]. Proceedings of the CSEE,2024,44(1):1-21.
• [22] CHEN X,QU G N,TANG Y J,et al. Reinforcement learning for selective key applications in power systems:recent advances and future challenges[J].IEEE Transactions on Smart Grid,2022,13(4):2935-2958.
• [23]王新迎，赵琦，赵黎媛，等.基于深度Q学习的电热综合能源系统能量管理[J].电力建设，2021,42(3):10-18.WANG Xinying,ZHAO Qi,ZHAO Liyuan,et al.Energy management of integrated electricity and heat system based on deep Q-learning[J]. Electric Power Construction,2021,42(3):10-18.
• [24]龚锦霞，刘艳敏.基于深度确定策略梯度算法的主动配电网协调优化[J].电力系统自动化，2020,44(6):113-120.GONG Jinxia,LIU Yanmin. Coordinated optimization of active distribution network based on deep deterministic policy gradient algorithm[J]. Automation of Electric Power Systems,2020,44(6):113-120.
• [25]马冲冲，王一铮，王坤，等.考虑源荷不确定性下微电网能量调度的深度强化学习策略[J].高技术通讯，2023,33(1):79-87.MA Chongchong,WANG Yizheng,WANG Kun,et al.Deep reinforcement learning based dispatch strategy for microgrid energy management considering uncertainty of source and load[J]. Chinese High Technology Letters,2023,33(1):79-87.
• [26]周翔，王继业，陈盛，等.基于深度强化学习的微网优化运行综述[J].全球能源互联网，2023,6(3):240-257.ZHOU Xiang,WANG Jiye,CHEN Sheng,et al.Review of microgrid optimization operation based on deep reinforcement learning[J].Journal of Global Energy Interconnection,2023,6(3):240-257.
• [27]高冠中，杨胜春，郭晓蕊，等.深度强化学习在含分布式柔性资源的电网优化调度中的应用研究综述[J].中国电机工程学报，2024,44(16):6385-6403.GAO Guanzhong,YANG Shengchun,GUO Xiaorui,et al.A review of research on the application of deep reinforcement learning in optimization dispatch of power grids with distributed flexible resources[J]. Proceedings of the Chinese Society for Electrical Engineering,2024,44(16):6385-6403.
• [28] YANG T,ZHAO L Y,LI W,et al.Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning[J].Energy,2021,235:121377.