基于多级列式索引的海量数据高效查询设计.pdf

2016 年 软 件 2016, Vol. 37, No. 3 第 37 卷 第 3 期 COMPUTER ENGINEERING SOFTWARE 国际 IT 传媒品牌 基于多级列式索引的海量数据高效查询设计 杨淙钧，艾中良，刘忠麟，李常宝 (华北计算技术研究所，北京 100083) 摘 要: 伴随海量数据存储、处理技术的发展，数据中心中积累了大量的格式化历史数据，此类数据呈现出数据 规模庞大、被查询频次低和查询内容规律不确定等特点，而当前以文件为操作对象的系统在查询此类数据时主要采用 分布式计算引擎对数据进行全局遍历，存在处理时间长、系统资源消耗高等问题。因此，本文提出了一种基于列式多 级索引的海量数据高效查询方法，使得查询过程中只有承载相关数据的节点参与计算，大幅降低了系统资源消耗。实 验表明，本文方法在用于大规模历史数据内容查询时，相对于较主流的文件系统查询技术有明显的效率提升。 关键词: 海量数据；文件系统；索引机制 中图分类号: TP314 文献标识码: A DOI： 10.3969/j.issn.1003-6970.2016.03.020 本文著录格式：杨淙钧，艾中良，刘忠麟，等. 基于多级列式索引的海量数据高效查询设计[J]. 软件，2016，37（3）： 7983 The Design of High Performance Massive Data Query System Based on K/V Index YANG Cong-jun, AI Zhong-liang, LIU Zhong-lin, LI Chang-bao (North China Institute of Computing Technology, Beijing 100083, China) 【Abstract 】: With mass data storage and processing technology development, data center has accumulated a large amount of historical formatted data, historical data has following characteristics: large scale, query frequency is low and the target of query is irregularly, and the current system which include query service and target file usually use distrib- uted computation engine to search global data, they usually take long time and system resources consumption is high. Therefore, this paper puts forward a method of efficient query based on the column-multilevel index, which greatly re- duces the consumption of system resources. Experimental results show that this method is effective in improving the efficiency of query technology for large scale historical data content. 【Key words 】: Mass Data; File System; Index Mechanism 0 引言 统，通过建立合理的索引、数据模型为分布式文件