HowSearchEnginesWorkGeneralSearchStrategies.ppt

How Search Engines Work General Search Strategies Dr. Dania Bilal IS 587 SIS Fall 2007 Fun Quiz Take the search engine quiz located at /library/quizzes/search_engine_quiz/blsearchenginequiz.htm Record the no. of incorrect answers Share the results of the quiz with a classmate. How Search Engines Work? They collect information from selected web sites The employ special software robots, called spiders, to crawl web pages Spiders build lists of the words found in Web sites. When a spider is building its lists, the spider is Web crawling. Spiders store the lists in the engine’s database The engine’s indexing software builds an index of words Information is matched against query input and retrieved (processing algorithm) How Spiders and Crawlers Work? They begin with popular and heavily used web servers. They begin with a popular site, collect the words on its pages and follow every link found within the site. Spiders travel across pages and the most widely used portions of the Web How Spiders and Crawlers Work? A dedicated server of URLs is built by a search engine company (e.g., Google) so that spiders collect information quickly More than one spider is used to craw web pages at a time Google uses 3-4 spiders and collect over 100 pages per second How Spiders and Crawlers Work? When no dedicated URL server is used, search engine company relies on ISP for the domain names (translated into addresses) to use for crawling the web Delay in gathering information Delay in updating information Lack of control over URL addresses Google Spider and How it Works A spider looks at the html or xml or other coding used to build a web page and collects information from the meta-tags It indexes words within the actual text of a page It indicates where the words were found (URL, title, headings, etc.) It disregards initial articles It disregards pages that should not be crawled or indexed Google Spider and How it Works It uses Robot-Exclusion Protocol in disregarding pages