heritrix框架下网络爬虫应用毕业设计.doc

Heritrix框架下网络爬虫应用 THE WEB CRAWLER APPLICATION BASE ON HERITRIX FRAMEWORK 专 业：软件工程 姓 名： 指导教师姓名： 申请学位级别：学 士 论文提交日期：2013年06月20日 学位授予单位：天津科技大学 摘 要 互联网是一个庞大的非结构化的数据库，目前网络中的资源非常丰富本文首先介绍了关键词： ABSTRACT The Internet is a vast unstructured database, the network is very rich in resources, but through the method of artificial browsing is very difficult to achieve the security of information browsing and consolidation, a lot of useful information is lost, the data effectively retrieval and organization has shown great application prospects.To meet the diverse needs and to further improve the relevance of search request, the theme of search engine starts to become the hot spot of research[5]. This paper firstly introduces the development status of web crawler, based on the analysis of the system architecture of Heritrixweb crawler recursive and Archiving Mechanism and the realization technology[3]; Combined with the characteristics of information network, based on analysis of the crawler system structure and working principle of the network, analyzes the Heritrix crawler to grab search strategy; At the same time, in view of the present stage Web information extraction technology, through the distribution characteristics of topic search page analysis , on the web page information extraction and webpage analysis technology,it can make web crawler flexible and efficient search related theme information[1]. Among them, the web crawler part adopts the open source, Heritrix web crawler written in JAVA; Webpage content extraction by traversing the Htmldocument tree and obtain the corresponding webpage content[4].Sign webpage and Url theme correlation on the topic search criterion, the application of the Web document clustering algorithm to analysis of Web document, and through Java implementation a network Reptile procedures, analyzes and summarizes the operating results, the topic web crawl