WebtoDeepWeb-HobartandWilliamSmithColleges.ppt

Mining the Michael Hunter Reference Librarian Hobart and William Smith Colleges For Western New York Library Resources Council Member Libraries’ Staff Sponsored by the Western New York Library Resources Council For today . . . From Web to Deep Web Search Services: Genres and Differences The Topography of the Internet Mining the Deep Web: Techniques and Tips Hands-on Session Evaluating Deep Web Resources Using Proprietary Software Web to Deep Web 1991 – Gopher Menu-based text only You had to KNOW the sites 1992 – Veronica Menus of menus Difficult to access Web to Deep Web 1991 - Hyper-Text Markup Language Linkage capability leads you to related information elsewhere “Classic” Web Site Relatively stable content of static, separate documents or files Typically no larger than 1,000 documents navigated via static directory structures Web to Deep Web 1994 – Lycos launched First crawler-based search engine with database of 54,000 html documents (CMU) Growth of html documents unprecedented and unanticipated 2000 (April) “The Web is doubling in size every 8 months” (FAST) Web to Deep Web 1996 – Three phenomena pivotal for the development of the Deep Web: HTML-based database technology introduced Bluestone’s Sapphire/Web, Oracle Commercialization of the Web Growth of home PC-users and e-commerce Web Servers adapted to embrace “dynamic” serving of data Microsoft’s ASP, Unix PHP and others Web to Deep Web 1998 – Deep Web comes of Age Larger sites redesigned with a database orientation rather than static directory structure U.S Bureau of the Census Securities and Exchange Commission Patent and Trademark Office Search Services:Genres and Differences Exclusively crawler-created Search engines Meta search engines Human created and/or influenced Directories Specialized search engines Subject metasites Deep Web gateway sites Search Services:Exclusively Crawler Created Database compiled through automated, link-dependent crawling and site submission Unable to access Dyna