12企业应用实例.pptx

企业应用实例主要内容Hadoop在企业的应用FaceBookYahoo!淘宝百度腾讯Hadoop在FaceBook的应用Facebook对HDFS及HBase的使用数据仓储实时随机读写实时连续读写Web工具Workflow 语言Job Scheduling ToolFacebook 消息HiveShell…Hive…监控工具PUMA…HBaseMapReduce …ScribePeregrineHDFSNameNode Scalability – FederationNN3 DN3 NN1DN1NN2 DN2HBase的层次存储 应用程序缓存HBase HBlock缓存应用程序管理 HBase管理操作系统的HDFS文件缓存 操作系统管理的 HDFS数据文件 在磁盘上的HDFS文件Hadoop在Yahoo!的应用Hadoop在雅虎MOBILE ANALYTICS PERSONALIZATION AD SELECTION DATA PIPELINES AD TARGETING WEB ANALYTICS MACHINE LEARNING SPAM FILTERING LOGIC REGRESSIONUSER INTEREST PREDICTION CONTENT OPTIMIZATIONTIME SERIESCONTENT CLUSTERING TRAFFIC VALIDATIONHadoop在雅虎 Search Assist Problem Related concepts appear close together in text corpus to assist users with search term Solution Hadoop helps Yahoo! process 1B web pages of about 10K bytes each (10 TB of inputdata) to create the outputlist of related words Behavioral TargetingProblemQuickly make complex decisions to serve the right ads to theright customer by targetingbillions of impressions per dayacross one of the largest ad networks in the worldSolutionHadoop helps Yahoo! process declared data and recentactivity to segment users and determine the right ads to servein millisecondsHadoop在雅虎 Mail Anti-SpamProblemYahoo! Mail delivers 5.6 billion email a day across 300 millionmailboxes. Users want to see emails from friendsand familyin the inboxes, from the people who matter the most… not MembershipAnti-AbuseProblemMembership processes 2.22M new registrations(127 Mlogins) every day! Abuse taints metrics, and parsing outabusive vs. legitimate user is an ongoing challengefrom spammers and phishersAccountRecoveryRegistrationLoginRegistration Success Rates Today1 Global Reg Success Rates WebAvg total regs 2.22 M/ dayAverage Daily Successful Regs (1M/ day)Web Registration For Yahoo! MailAccountRecoveryRegistrationLogin1Avg good regs1Avg total sessions0.88 M/ day2.82 M/ day107,845(11%)11,611(1%) Avg totalregs/ day Avg good regs/ dayU.S.576K226KNon- U.S. 442K 194K% of All Regs 46% 48%Avg g