好用的Nutch系统.ppt

Google Cluster Computing Faculty Training Workshop Module III: Nutch Meta-details Built to encourage public search work Open-source, w/pluggable modules Cheap to run, both machines admins Goal: Search more pages, with better quality, than any other engine Pretty good ranking Has done ~ 200M pages, more possible Hadoop is a spinoff Outline Nutch design Link database, fetcher, indexer, etc… Hadoop support Distributed filesystem, job control Moving Parts Acquisition cycle WebDB Fetcher Index generation Indexing Link analysis (maybe) Serving results WebDB Contains info on all pages, links URL, last download, # failures, link score, content hash, ref counting Source hash, target URL Must always be consistent Designed to minimize disk seeks 19ms seek time x 200m new pages/mo = ~44 days of disk seeks! Single-disk WebDB was huge headache Fetcher Fetcher is very stupid. Not a “crawler” Pre-MapRed: divide “to-fetch list” into k pieces, one for each fetcher machine URLs for one domain go to same list, otherwise random “Politeness” w/o inter-fetcher protocols Can observe robots.txt similarly Better DNS, robots caching Easy parallelism Two outputs: pages, WebDB edits WebDB/Fetcher Updates Indexing Iterate through all k page sets in parallel, constructing inverted index Creates a “searchable document” of: URL text Content text Incoming anchor text Other content types might have a different document fields Eg, email has sender/receiver Any searchable field end-user will want Uses Lucene text indexer Link analysis A page’s relevance depends on both intrinsic and extrinsic factors Intrinsic: page title, URL, text Extrinsic: anchor text, link graph PageRank is most famous of many Others include: HITS OPIC Simple incoming link count Link analysis is sexy, but importance generally overstated Link analysis (2) Nutch performs analysis in WebDB Emit a score for each known page At index time, incorporate score into inverted index Extremely time-consuming In our case, disk-consuming