chapter02-part03-search-engine_有哪些信誉好的足球投注网站引擎.ppt

Mining the Web Chakrabarti and Ramakrishnan Search Engine 朱廷劭（Zhu, Tingshao）Ph.D Examples of search engines Conventional (library catalog). Search by keyword, title, author, etc. Text-based (Lexis-Nexis, Google, Yahoo!).Search by keywords. Limited search using queries in natural language. Multimedia (QBIC, WebSeek, SaFe)Search by visual appearance (shapes, colors,… ). Question answering systems (Ask, NSIR, Answerbus)Search in (restricted) natural language Clustering systems (Vivisimo, Clusty) Research systems (Lemur, Nutch) What does it take to build a search engine? Decide what to index Collect it Index it (efficiently) Keep the index up to date Provide user-friendly query facilities What else? Understand the structure of the web for efficient crawling Understand user information needs Preprocess text and other unstructured data Cluster data Classify data Evaluate performance How Search Engines Work Gather the contents of all web pages (using a program called a crawler or spider) Organize the contents of the pages in a way that allows efficient retrieval (indexing) Take in a query, determine which pages match, and show the results (ranking and display of results) Standard Web Search Engine Architecture Standard Web Search Engine Architecture Standard Web Search Engine Architecture Spiders (crawlers) How to find web pages to visit and copy? Can start with a list of domain names, visit the home pages there. Look at the hyperlink on the home page, and follow those links to more pages. Use HTTP commands to GET the pages Keep a list of urls visited, and those still to be visited. Each time the program loads in a new HTML page, add the links in that page to the list to be crawled. Four Laws of Crawling A Crawler must show identification A Crawler must obey the robots exclusion standard /wc/norobots.html A Crawler must not hog resources A Crawler must report errors Example robots.txt file /robots.txt (just the first few lines) Lots of tricky aspects Servers are ofte