全部 标题 作者
关键词 摘要

OALib Journal期刊
ISSN: 2333-9721
费用:99美元

查看量下载量

相关文章

更多...

云平台下输变电设备状态监测大数据存储优化与并行处理

DOI: 10.13334/j.0258-8013.pcsee.2015.02.001, PP. 255-267

Keywords: 大数据,输变电设备,状态监测,一致哈希,云计算

Full-Text   Cite this paper   Add to My Lib

Abstract:

结合大数据技术提升输变电设备状态评价的广度和深度,并解决实际应用问题成为目前电力行业新的挑战。针对输变电设备状态监测大数据可靠存储和快速访问两方面大数据处理核心问题,基于开源的Hadoop云计算实验平台进行了数据分布策略、数据块尺寸调优、集群网络拓扑规划三方面的存储优化研究和大数据并行分析的研究。提出计及数据相关性的多副本一致哈希数据存储算法,能将具有相关性的数据在集群中聚集,提升数据处理执行效率。基于数据相关性多副本一致哈希数据分布,应用MapReduce并行编程模型设计实现了多数据源并行连接查询算法和多通道数据融合并行特征提取算法。将两种算法在实验室搭建的集群上测试运行,结果表明,多数据源并行连接查询的执行时间仅为标准Hadoop方案的32%,多通道数据融合并行特征提取算法执行时间仅为标准Hadoop方案的35%。

References

[1]  宋亚奇,周国亮,朱永利.智能电网大数据处理技术现状与挑战[J].电网技术,2013,37(4):927-935.Song Yaqi,Zhou Guoliang,Zhu Yongli.Present status and challenges of big data processing in smart grid[J].Power System Technology,2013,37(4):927-935(in Chinese).
[2]  Tom White.Hadoop权威指南[M].2版.曾大聃,周傲英,译.北京:清华大学出版社,2011:260-262.Tom White.Hadoop:the definitive guide[M].2nd Edition.Beijing:O'Reilly Media,2011:260-262.
[3]  Dean J,Ghemawat S.MapReduce:simplified data processing on large clusters[J].Communications of the ACM,2008,51(1):107-113.
[4]  Cooper B F,Baldeschwieler E,Fonseca R,et al.Building a cloud for yahoo[J].IEEE Data Eng.Bull.,2009,32(1):36-43.
[5]  Lars George.HBase:The definitive guide[M].Beijing:O'Reilly Media,2011:324-327.
[6]  宫学庆,金澈清,王晓玲,等.数据密集型科学与工程:需求和挑战[J].计算机学报,2012,35(8):1563-1578.Gong Xueqing,Jin Cheqing,Wang Xiaoling,et al.Data-intensive science and engineering:requirements and challenges[J].Chinese Journal of Computers,2012,35(8):1563-1578(in Chinese).
[7]  Rao J,Zhang C,Megiddo N,et al.Automating physical database design in a parallel database[C]//Proceedings of the 2002 ACM SIGMOD international conference on Management of data.New York:ACM,2002:558-569.
[8]  王德文,宋亚奇,朱永利.基于云计算的智能电网信息平台[J].电力系统自动化,2010,34(22):7-12.Wang Dewen,Song Yaqi,Zhu Yongli.Information platform of smart grid based on cloud computing[J].Automation of Electric Power Systems,2010,34(22):7-12(in Chinese).
[9]  朱征,顾中坚,吴金龙,等.云计算在电力系统数据灾备业务中的应用研究[J].电网技术,2012,36(9):43-50.Zhu Zheng,Gu Zhongjian,Wu Jinlong,et al.Application of cloud computing in electric power system data recovery[J].Power System Technology,2012,36(9):43-50(in Chinese).
[10]  沐连顺,崔立忠,安宁.电力系统云计算中心的研究与实践[J].电网技术.2011,35(6):170-175.Mu Lianshun,Cui Lizhong,An Ning.Research and practice of cloud computing center for power system[J].Power System Technology,2011,35(6):170-175(in Chinese).
[11]  田芳,黄彦浩,史东宇,等.电力系统仿真分析技术的发展趋势[J].中国电机工程学报,2014,34(13):2151-2163.TIAN Fang, HUANG Yanhao, SHI Dongyu, etc.Developing Trend of Power System Simulation and Analysis Technology[J].Proceedings of the CSEE,2014,34(13):2151-2163(in Chinese).
[12]  齐林海,艾明浩.一种基于云计算的电压暂降并行计算方法[J].中国电机工程学报,2014,34(31):5493-5499.Qi Linhai,Ai Minghao.A voltage sag parallel calculation method based on cloud computing[J].Proceedings of the CSEE,2014,34(31):5493-5499(in Chinese).
[13]  屈志坚,郭亮,刘明光,等.智能配电网量测信息变断面柔性压缩新算法[J].中国电机工程学报,2013,33(19):191-199.Qu Zhijian,Guo Liang,Liu Mingguang,et al.New variable section flexible compression algorithm for measurement information in intelligent distribution network[J].Proceedings of the CSEE,2013,33(19):191-199(in Chinese).
[14]  曲朝阳,朱莉,张士林.基于Hadoop的广域测量系统数据处理[J].电力系统自动化,2014,37(4):92-97.Qu Zhaoyang,Zhu Li,Zhang Shilin.Data processing of hadoop-based wide area measurement system[J].Automation of Electric Power Systems,2014,37(4):92-97(in Chinese).
[15]  王保义,赵硕,张少敏.基于云计算和极限学习机的分布式电力负荷预测算法[J].电网技术,2014,38(2):526-531.Wang Baoyi,Zhao Shuo,Zhang Shaomin.A distributed load forecasting algorithm based on cloud computing and extreme learning machine[J].Power System Technology,2014,38(2):526-531(in Chinese).
[16]  张逸,杨洪耕,叶茂清.基于分布式文件系统的海量电能质量监测数据管理方案[J].电力系统自动化,2014,38(2):102-108.Zhang Yi,Yang Honggeng,Ye Maoqing.A data management scheme for massive power quality monitoring data based on distributed file system[J].Automation of Electric Power Systems,2014,38(2):102-108(in Chinese).
[17]  曲广龙,杨洪耕,张逸.采用Map-Reduce模型的海量电能质量数据交换格式文件快速解析方案[J].电网技术,2014,38(6):1705-1711.Qu Guanglong,Yang Honggeng,Zhang Yi.A fast parallel parsing scheme for massive PQDIF files with map-reduce model[J].Power System Technology,2014,38(6):1705-1711(in Chinese).
[18]  刘树仁,宋亚奇,朱永利,等.基于Hadoop的智能电网状态监测数据存储研究[J].计算机科学,2013,40(1):81-84.Liu Shuren,Song Yaqi,Zhu Yongli,et al.Research on data storage for smart grid condition monitoring using Hadoop[J].Computer Science,2013,40(1):81-84(in Chinese).
[19]  宋亚奇,刘树仁,朱永利,等.电力设备状态高速采样数据的云存储技术研究[J].电力自动化设备,2013,33(10):150-156.Song Yaqi,Liu Shuren,Zhu Yongli,et al.Research on cloud storage of power equipment high-speed sampled data[J].Electric Power Automation Equipment,2013,33(10):150-156(in Chinese).
[20]  Christophe Bisciglia.The smart grid:Hadoop at the tennessee valley authority(TVA)[EB/OL].http://www. cloudera.com/blog/2009/06/smart-grid-hadoop-tennessee-valley-authority-tva/.
[21]  Rusitschka S,Eger K,Gerdes C.Smart grid data cloud:a model for utilizing cloud computing in the smart grid domain[C]// IEEE International Conference on Smart Grid Communications.Gaithersburg:IEEE,2010:483-488.
[22]  Michael G N.Benchmarking and stress testing an hadoop cluster with terasort,TestDFSIO&Co.[EB/OL].(2011-04-09).http://www.michael-noll.com/blog/2011/ 04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/.
[23]  Kawasoe S,Igarashi Y,Shibayama K,et al.Examples of distributed information platforms constructed by power utilities in Japan[C]//CIGRE 2012.Paris:CIGRE,2012:D2_108_2012.
[24]  Kenneth P B,Lakshmi G,Robbert van Renesse.Running smart grid control software on cloud computing architectures[C]//Workshop on Computational Needs for the Next Generation Electric Grid.Ithaca:Cornell University,2011:1-28.
[25]  胡丽聪,徐雅静,徐惠民,等.基于动态反馈的一致性哈希负载均衡算法[J].微电子学与计算机,2012,29(1):177-180.Hu Licong,Xu Yajing,Xu Huimin,et al.A consistent Hash load balancing algorithm based on dynamic feedback[J].Microelectronics & Computer,2012,29(1):177-180(in Chinese).
[26]  赵彦荣,王伟平,孟丹,等.基于Hadoop的高效连接查询处理算法CHMJ[J].软件学报,2012,23(8):2032-2041.Zhao Yanrong,Wang Weiping,Meng Dan,et al.Efficient join query processing algorithm CHMJ based on Hadoop[J].Journal of Software,2012,23(8):2032-2041(in Chinese).
[27]  Zhang Jing,Wu Gongqing,Hu Xuegang,et al.A distributed cache for Hadoop distributed file system in real-time cloud services[C]//2012 ACM/IEEE 13th International Conference on Grid Computing.Beijing,China:IEEE,2012:12-21.
[28]  Shang Pengju,Xiao Qiangju,Wang Jun.DRAW:a new data-grouping-aware data placement scheme for data intensive applications with interest locality[J].IEEE Transactions on Magnetics,2013,49(6):2514-2520.
[29]  郭力争,赵曙光,姜长远.云计算环境下基于关联量的数据部署与任务调度[J].计算机工程与科学,2013(8):1-7.Guo Lizheng,Zhao Shuguang,Jiang Changyuan.Data placement and task scheduling based on associated amount in cloud computing[J].Computer Engineering & Science,2013(8):1-7(in Chinese).
[30]  童明.基于HDFS 的分布式存储研究与应用[D].武汉:华中科技大学,2012.Tong Ming.Research and application of distributed storage based on HDFS[D].Wuhan:Huazhong University of Science and Technology,2012.
[31]  李鹏,刘澄玉,李丽萍,等.多尺度多变量模糊熵分析[J].物理学报,2013,62(12):120512-120512.Li Peng,Liu Chengyu,Li Liping,et al.Multiscale multivariate fuzzy entropy analysis[J].Acta Physica Sinica,2013,62(12):120512-120512(in Chinese).
[32]  Ahmed M U,Mandic D P.Multivariate multiscale entropy analysis[J].IEEE Signal Processing Letters,2012,19(2):91-94.
[33]  Morabito F C,Labate D,La Foresta F,et al.Multivariate multi-scale permutation entropy for complexity analysis of Alzheimer’s disease EEG[J].Entropy,2012,14(7):1186-1202.
[34]  Cao L,Mees A,Judd K.Dynamics from multivariate time series[J].Physica D:Nonlinear Phenomena,1998,121(1):75-88.

Full-Text

Contact Us

service@oalib.com

QQ:3279437679

WhatsApp +8615387084133