陈祖歌,毛冬,饶涵宇,张辰,章九鼎,刘泽瀚
CHEN Zuge,MAO Dong,RAO Hanyu,ZHANG Chen,ZHANG Jiuding,LIU Zehan

• 1:国网浙江省电力有限公司信息通信分公司
• 2:北京航空航天大学杭州创新研究院

摘要(Abstract)：

哈希算法目前是实现云计算数据存储的主流算法，但是该算法存在一些不足，如数据处理效率低、容错率低等。针对这些问题，将HDFS(Hadoop分布式文件系统)与哈希算法相结合来优化物联网中云计算的数据存储。首先，采用HDFS优化结构化数据的存储架构；然后，对文件类型数据的存储设计结构进行优化；最后，通过实验对所提算法进行验证。实验结果表明，所提出的优化算法相较于原始算法数据处理效率平均提升24.53%，容错率平均提升25.56%，充分说明了所提云计算数据存储优化算法的优越性。
Hash algorithm is currently the mainstream algorithm of data storage for cloud computing, but it has some shortcomings, such as low data processing efficiency and low fault tolerance. To solve these problems, this paper combines HDFS(Hadoop distributed file system) and Hash algorithm to optimize the data storage of cloud computing in the Internet of Things. Firstly, HDFS is used to optimize the storage architecture of structured data. Then, the storage design structure of file type data is optimized. Finally, the experiments are carried out to verify the proposed algorithm. The experimental results show that compared with the original algorithm, the data processing efficiency of the proposed optimization algorithm is improved by 24.53% on average, and the fault tolerance rate is improved by 25.56% on average, which fully shows that the proposed data storage optimization algorithm for cloud computing is more superior.

关键词(KeyWords)： 能源大数据;云计算;数据存储;哈希算法;物联网
energy big data;cloud computing;data storage;Hash algorithm;Internet of Things

Abstract:

Keywords:

基金项目(Foundation): 国网浙江省电力有限公司科技项目（B311XT220006）

作者(Author): 陈祖歌,毛冬,饶涵宇,张辰,章九鼎,刘泽瀚
CHEN Zuge,MAO Dong,RAO Hanyu,ZHANG Chen,ZHANG Jiuding,LIU Zehan

DOI: 10.19585/j.zjdl.202308003

参考文献(References)：

• [1]尹钟舒，洛向刚，杨成，等.物联网（IoT）：国内现状和国家标准综述[J].网络安全技术与应用，2022(9):108-111.YIN Zhongshu,LUO Xianggang,YANG Cheng,et al.Internet of Things(IoT):overview of domestic status and national standards[J]. Network Security Technology&Application,2022(9):108-111.
• [2]王波，王红霞，姚良忠，等.电力系统多模态数据融合模式及关键技术问题[J].电力系统自动化，2022,46(19):188-199.WANG Bo,WANG Hongxia,YAO Liangzhong,et al.Multi-modal data fusion mode for power system and its key technical issues[J].Automation of Electric Power Systems,2022,46(19):188-199.
• [3]李大伟，霍瑛，等.基于侧链技术的电力物联网跨域认证研究[J].电力工程技术，2020,39(6):8-12.LI Dawei,HUO Ying. Cross domain authentication of power IoT based on side chain[J]. Electric Power Engineering Technology,2020,39(6):8-12.
• [4]王海，曾飞，杨雄等.基于区块链的配电物联网数据安全防护方法[J].电力工程技术，2021,40(5):47-53.WANG Hai,ZENG Fei,YANG Xiong.Blockchain-based data security protection for distribution Internet of Things[J].Electric Power Engineering Technology,2021,40(5):47-53.
• [5]曾飞，杨雄，苏伟，等.基于区块链与数据湖的电力数据存储与共享方法[J].电力工程技术，2022,41(3):48-54.ZENG Fei,YANG Xiong,SU Wei,et al.Power data storage and sharing method based on blockchain and data lake[J].Electric Power Engineering Technology,2022,41(3):48-54.
• [6]林志贵，张晓慧，刘英平，等.基于存储阈值的蛇形时隙数据存储算法[J].计算机工程，2016,42(12):32-38.LIN Zhigui,ZHANG Xiaohui,LIU Yingping,et al.Snakelike slot time data storage algorithm based on storage threshold[J].Computer Engineering,2016,42(12):32-38.
• [7]钱伟强.基于海量存储云调度机制的云网络数据存储算法[J].国外电子测量技术，2017,36(3):27-30.QIAN Weiqiang. Research on cloud data storage algorithm based on mass storage cloud scheduling[J].Foreign Electronic Measurement Technology,2017,36(3):27-30.
• [8]朱维.物联网数据通信安全机制的研究[D].北京：华北电力大学，2015.ZHU Wei.Research on the security mechanism of Internet of Things data communication[D].Beijing:North China Electric Power University,2015.
• [9]陈明，张凯，王熙，等.云计算在电力信息化建设中的应用与发展研究[J].科技与创新，2022(20):176-178.CHEN Ming,ZHANG Kai,WANG Xi,et al.Research on application and development of cloud computing in electric power informatization construction[J].Science and Technology&Innovation,2022(20):176-178.
• [10]袁爱平，邓河，陈为满.HDFS数据副本优化放置策略研究[J].电脑与信息技术，2022,30(5):49-51.YUAN Aiping,DENG He,CHEN Weiman.Research on optimal placement strategy of HDFS data copy[J].Computer and Information Technology,2022,30(5):49-51.
• [11] SA R N. Cloud computing resource scheduling scheme based on ant colony particle swarm optimization algorithm[J].Journal of Jilin University(Science Edition),2017,55(6):1518-1522.
• [12]陈国瑞，袁旭华.基于HDFS开源架构的异常数据实时检测算法[J].计算机仿真，2021,38(8):445-449.CHEN Guorui,YUAN Xuhua. Real-time detection algorithm of abnormal data based on HDFS open source architecture[J].Computer Simulation,2021,38(8):445-449.
• [13] DIAH J M,AHMAD J,MUKRI M.The methodology on statistical analysis of data transformation for model development[J].International Journal of Statistics and Applications,2013,2(6):94-100.
• [14] ZHOU M,LI J Q,YE J. Design of big data compatible storage system based on cloud computing environment[C]//2020 39th Chinese Control Conference(CCC),July 27-29,2020,Shenyang,China:3158-3161.
• [15] ALI-ELDIN A,TORDSSON J,ELMROTH E.An adaptive hybrid elasticity controller for cloud infrastructures[C]//2012 IEEE Network Operations and Management Symposium,April 16-20,2012,Maui,USA:204-212.
• [16]石方夏，高屹.Hadoop大数据技术应用分析[J].现代电子技术，2021,44(19):153-157.SHI Fangxia,GAO Yi. Applied analysis of Hadoop big data technology[J].Modern Electronics Technique,2021,44(19):153-157.
• [17] YANG C G,WANG Z R,HE W,et al.Development of a fast transmission method for 3D point cloud[J].Multimedia Tools and Applications,2018,77(19):25369-25387.
• [18] MORENO C,LI M.A progressive transmission technique for the streaming of point cloud data using the Kinect[C]//2018 International Conference on Computing,Networking and Communications(ICNC),March 5-8,2018,Maui,USA:593-598.
• [19]常会丽，周金莲.基于MVC架构的多维数据压缩存储方法研究[J].计算机仿真，2022,39(4):279-283.CHANG Huili,ZHOU Jinlian. Research on multidimensional data compression storage method based on MVC architecture[J]. Computer Simulation,2022,39(4):279-283.
• [20]曹熙.基于一致性哈希算法的电力企业分布式数据存储研究[J].长江信息通信，2022,35(6):147-149.CAO Xi. Research on distributed data storage of electric power enterprises based on consistent hash algorithm[J].Changjiang Information&Communications,2022,35(6):147-149.
• [21] ZHANG Hongwei,BAILIAN L. Features of 3D laser scanning technology and it’s application prospects[J].Bulletin of Surveying and Mapping,2012,16(S1):320-322.
• [22] SHI G,DANG X,GAO X. Research on adaptive point cloud simplification and compression technology based on curvature estimation of energy function[J]. Revista de la Facultad de Ingeniería UCV,2017,32:336-343.
• [23]潘灏，丘刚玮，黄欣.一种小文件存储优化方法在云存储平台中的应用[J].轻工科技，2019,35(9):83-85.PAN Hao,QIU Gangwei,HUANG Xin.Application of a small file storage optimization method in cloud storage platform[J]. Light Industry Science and Technology,2019,35(9):83-85.