大数据管理与数据思维(全)孟小峰 教授,人民大学.pdf

大数据管理与数据思维 Big Data and Data Thinking 孟小峰 中国人民大学信息学院 • 数据海洋 海量异构 281EB in 2009 仍以指数级增长 内容多样 半结构 非结构 动态增长 Data Explosion Big Data MB = 106 bytes a typical book in text format GB = 109 bytes 12 a one hour video is about 1GB; TB = 10 bytes PB = 1015 bytes data produced by a biology data produced in astronomy data in one night; experiment in one day US Library of Congress has 1000 TB data; search log of Bing is 20 TB per day Google processes 20 PB data per day; VLDB vs. Big Data Very Large Database （VLDB ） 数据工程 Data Engineering  MB, 结构数据  以数据为对象解决其存储 和管理问题 Big Data 数据思维 PB，非结构数据 Data Thinking  以数据为资源解决诸领域 问题 More data vs. better algorithms The Netflix challenge ($1M awarded Sep. 22, 09)  A: sophisticated algorithm  B: simple algorithm + additional data (IMDB)  B gets much better results Google PageRank:  Rank pages by additional data: link + anchortext Overture and Google AdWords:  Overture ranks advertisers on their bids  Google ranks by bids * CTR More data beats better algorithms The Big Data Makes the Scientific Method Obsolete by Chris Anderson, in “All models are wrong, but some are useful.” George