高远,李香帅,余光正,杨彬
GAO Yuan,LI Xiangshuai,YU Guangzheng,YANG Bin

• 1:上海电力大学电气工程学院
• 2:国网浙江省电力有限公司衢州供电公司

摘要(Abstract)：

针对高比例可再生能源并网引发的电力系统频率失稳问题，提出一种融合统一建模与改进深度强化学习的频率控制方法。首先，通过提取电动汽车及储能的共性动态特性，构建表征调频能力的统一模型，解决异构资源参数差异导致的建模难题；其次，构建AGC（自动发电控制）信号协同控制机制，实现多资源动态权重分配与状态自适应响应；然后，设计熵正则化双延迟DDPG（深度确定性策略梯度）算法，利用动态熵项增强探索能力，通过双延迟网络抑制价值函数高估；最后，通过仿真验证所提方法的有效性。仿真结果表明，所提方法在频率控制效果与收敛效率方面均显著优于传统方法，可为高比例新能源电力系统的安全稳定运行提供方案参考。
Addressing frequency instability in power systems with high-penetration renewables, this paper proposes a control method that integrates unified modeling with improved deep reinforcement learning. Firstly, a unified model characterizing frequency regulation capability is developed by extracting the common dynamics of electric vehicles and energy storage systems, thus overcoming modeling challenges posed by heterogeneous resource parameters. Secondly, a coordinated automatic generation control(AGC) signal control mechanism is designed to achieve dynamic weight allocation and state-adaptive response for multiple resources. Furthermore, an entropy-regularized twin delayed deep deterministic policy gradient(DDPG) algorithm is proposed, which enhances exploratory behavior via a dynamic entropy term and reduces value function overestimation through twin delayed networks. Simulation results demonstrate the superiority of the proposed method over conventional approaches in both frequency control performance and convergence efficiency, offering a viable solution for securing power systems with high shares of renewable energy.

关键词(KeyWords)： 熵正则化;深度确定性策略梯度算法;AGC信号协同控制;灵活性资源
entropy regularization;DDPG algorithm;coordinated AGC signal control;flexible resources

Abstract:

Keywords:

基金项目(Foundation): 国家自然科学基金(52207121)

作者(Author): 高远,李香帅,余光正,杨彬
GAO Yuan,LI Xiangshuai,YU Guangzheng,YANG Bin

DOI: 10.19585/j.zjdl.202602009

参考文献(References)：

• [1]谢小荣，贺静波，毛航银，等.“双高”电力系统稳定性的新问题及分类探讨[J].中国电机工程学报，2021,41(2):461-475.XIE Xiaorong,HE Jingbo,MAO Hangyin,et al.New issues and classification of power system stability with high shares of renewables and power electronics[J]. Proceedings of the CSEE,2021,41(2):461-475.
• [2]雷傲宇，苏婷婷，梅勇，等.基于分频器理论的新能源电网暂态频率稳定分析方法[J].电测与仪表，2024,61(4):132-140.LEI Aoyu,SU Tingting,MEI Yong,et al.A transient frequency stability analysis method for new energy grid based on frequency divider theory[J].Electrical Measurement&Instrumentation,2024,61(4):132-140.
• [3]陈景文，王媛，王福强，等.光储直流微电网多运行工况稳定性分析[J].智慧电力，2024,52(4):15-23.CHEN Jingwen,WANG Yuan,WANG Fuqiang,et al.Stability analysis of optical storage DC microgrid under multiple operating conditions[J]. Smart Power,2024,52(4):15-23.
• [4]随权，魏繁荣，林湘宁，等.一种基于可控负荷效率控制的孤岛微网新型调度策略[J].中国电机工程学报，2019,39(24):7168-7178.SUI Quan,WEI Fanrong,LIN Xiangning,et al.A novel dispatching strategy for isolated microgrid based on controllable load efficiency control[J]. Proceedings of the CSEE,2019,39(24):7168-7178.
• [5]KESAVAN P K,SUBRAMANIAM U,ALMAKHLES D J,et al.Modelling and coordinated control of grid connected photovoltaic, wind turbine driven PMSG,and energy storage device for a hybrid DC/AC microgrid[J].Protection and Control of Modern Power Systems,2024,9(1):154-167.
• [6]王枭，何怡刚，马恒瑞，等.面向电网辅助服务的虚拟储能电厂分布式优化控制方法[J].电力系统自动化，2022,46(10):181-188.WANG Xiao,HE Yigang,MA Hengrui,et al.Distributed optimization control method of virtual energy storage plants for power grid ancillary services[J].Automation of Electric Power Systems,2022,46(10):181-188.
• [7]HUI H X,DING Y,SHI Q X,et al.5G network-based Internet of Things for demand response in smart grid:a survey on application potential[J]. Applied Energy,2020,257:113972.
• [8]MOGHADAM M R V,ZHANG R,MA R T B.Distributed frequency control via randomized response of electric vehicles in power grid[J].IEEE Transactions on Sustainable Energy,2016,7(1):312-324.
• [9]杨冰，王丽芳，廖承林.大规模电动汽车充电需求及影响因素[J].电工技术学报，2013,28(2):22-27.YANG Bing,WANG Lifang,LIAO Chenglin. Research on power-charging demand of large-scale electric vehicles and its impacting factors[J].Transactions of China Electrotechnical Society,2013,28(2):22-27.
• [10]王文倬，谢丁，谢醉冰，等.考虑集群划分的分布式光伏无功电压控制策略[J].浙江电力，2024,43(7):64-75.WANG Wenzhuo,XIE Ding,XIE Zuibing,et al.A reactive voltage control strategy for distributed PV based on cluster segmentation[J].Zhejiang Electric Power,2024,43(7):64-75.
• [11]吕林，许威，向月，等.基于马尔科夫链充电负荷预测的多区域充电桩优化配置研究[J].工程科学与技术，2017,49(3):170-178.LYU Lin,XU Wei,XIANG Yue,et al.Optimal allocation of charging piles in multi-areas considering charging load forecasting based on Markov chain[J]. Advanced Engineering Sciences,2017,49(3):170-178.
• [12]姚若钰，王馨杉.电动汽车充电站参与电网辅助调频的控制方法[J].电力科学与技术学报，2025,40(1):19-28.YAO Ruoyu,WANG Xinshan.Control method of electric vehicle charging station participating in auxiliary frequency regulation for power grid[J].Journal of Electric Power Science and Technology,2025,40(1):19-28.
• [13]WANG S Y,WU W C. Aggregate flexibility of virtual power plants with temporal coupling constraints[J].IEEE Transactions on Smart Grid,2021,12(6):5043-5051.
• [14]FENG C,CHEN Q X,WANG Y,et al.Provision of contingency frequency services for virtual power plants with aggregated models[J].IEEE Transactions on Smart Grid,2023,14(4):2798-2811.
• [15]WEN Y L,HU Z C,LIU L K. Aggregate temporally coupled power flexibility of DERs considering distribution system security constraints[J]. IEEE Transactions on Power Systems,2023,38(4):3884-3896.
• [16]秦珺晗.面向主动配电网的柔性负荷聚合建模及其调控策略研究[D].北京：华北电力大学，2021.QIN Junhan.Study on the flexible load aggregation modeling and control strategy for active distribution network[D].Beijing:North China Electric Power University,2021.
• [17]LIU D,YANG Q F,CHEN Y,et al.Optimal parameters and placement of hybrid energy storage systems for frequency stability improvement[J]. Protection and Control of Modern Power Systems,2025,10(2):40-53.
• [18]张江丰，葛一统，谭珺敉，等.风电场联合分布式储能的协同调频策略[J].浙江电力，2024,43(9):67-77.ZHANG Jiangfeng,GE Yitong,TAN Junmi,et al.A coordinated frequency regulation strategy integrating wind farms with distributed energy storage[J].Zhejiang Electric Power,2024,43(9):67-77.
• [19]LIU W Y,TAN J B,CUI J W.Design of a PID control scheme for pound-drever-hall frequency-tracking system based on system identification and fuzzy control[J].IEEE Sensors Journal,2024,24(23):39252-39259.
• [20]WANG W K,YIN H,CHEN C,et al.Frequency disturbance event detection based on synchrophasors and deep learning[J].IEEE Transactions on Smart Grid,2020,11(4):3593-3605.
• [21]CAO D,ZHAO J B,HU W H,et al. Attention enabled multi-agent DRL for decentralized volt-VAR control of active distribution system using PV inverters and SVCs[J].IEEE Transactions on Sustainable Energy,2021,12(3):1582-1592.
• [22]LIU H Y,ZHANG C,CHAI Q M,et al.Robust regional coordination of inverter-based volt/var control via multiagent deep reinforcement learning[J].IEEE Transactions on Smart Grid,2021,12(6):5420-5433.
• [23]ZHANG J,PENG F F,WANG L L,et al. A load frequency control strategy based on double deep Q-network and upper confidence bound algorithm of multi-area interconnected power systems[J]. Computers and Electrical Engineering,2024,120:109778.
• [24]梁煜东，陈峦，张国洲，等.基于深度强化学习的多能互补发电系统负荷频率控制策略[J].电工技术学报，2022,37(7):1768-1779.LIANG Yudong,CHEN Luan,ZHANG Guozhou,et al.Load frequency control strategy of hybrid power generation system:a deep reinforcement learning:based approach[J].Transactions of China Electrotechnical Society,2022,37(7):1768-1779.
• [25]谢黎龙，李勇汇，范培潇，等.基于深度强化学习的孤立多微电网系统频率和电压综合控制[J].电力自动化设备，2024,44(6):118-126.XIE Lilong,LI Yonghui,FAN Peixiao,et al. Deep reinforcement learning-based integrated frequency and voltage control for isolated multi-microgrid system[J]. Electric Power Automation Equipment,2024,44(6):118-126.
• [26]LI J W,YU T,ZHANG X S,et al.Efficient experience replay based deep deterministic policy gradient for AGC dispatch in integrated energy system[J]. Applied Energy,2021,285:116386.
• [27]范培潇，柯松，杨军，等.基于改进多智能体深度确定性策略梯度的多微网负荷频率协同控制策略[J].电网技术，2022,46(9):3504-3515.FAN Peixiao,KE Song,YANG Jun,et al.Load frequency coordinated control strategy of multi-microgrid based on improved MA-DDPG[J]. Power System Technology,2022,46(9):3504-3515.
• [28]MOEINI A,KAMWA I,BRUNELLE P,et al.Open data IEEE test systems implemented in sim power systems for education and research in power grid dynamics and control[C]//2015 50th International Universities Power Engineering Conference(UPEC).September 1-4,2015,Stoke on Trent,UK.IEEE,2015:1-6.