Focused Crawling Using Context Graphs 翻译.doc

Focused Crawling Using Context Graphs M. Diligenti?, F. M. Coetzee, S. Lawrence, C. L. Giles and M. Gori Abstract Maintaining currency of search engine indices by exhaustive crawling is rapidly becoming impossi- ble due to the increasing size and dynamic content of the web. Focused crawlers aim to search only the subset of the web related to a specific cate- gory, and offer a potential solution to the currency problem. The major problem in focused crawl- ing is performing appropriate credit assignment to different documents along a crawl path, such that short-term gains are not pursued at the ex- pense of less-obvious crawl paths that ultimately yield larger sets of valuable pages. To address this problem we present a focused crawling algo- rithm that builds a model for the context within which topically relevant pages occur on the web. This context model can capture typical link hierar- chies within which valuable pages occur, as well as model content on documents that frequently co- occur with relevant pages. Our algorithm further leverages the existing capability of large search engines to provide partial reverse crawling capa- bilities. Our algorithm shows significant perfor- mance improvements in crawling efficiency over standard focused crawling. 1 介绍 公共可变址互联网规模已可超过十亿没有显示出增长趋于平缓的迹象。动态内容也随着时，如新闻，财务数据，娱乐和时间表，通过网络被广泛散发。因此，试图爬行目前的指数，有哪些信誉好的足球投注网站引擎日益受到挑战。即使使用，如AltaVista，据说每天抓取一千万网页，抓取可能需数周。消耗大量的存储和带宽资源，其中一些有哪些信誉好的足球投注网站引擎的控制。 主题爬虫只有哪些信誉好的足球投注网站和检索涉及相关的具体万维网子集。理想主题爬虫检索相关的网页，同时网络不相关文件。主题爬虫允许提供问题的潜在解决。主题爬虫也很适合有效地生成利基门户网站有哪些信誉好的足球投注网站引擎和用户群体维护指数，其中有限的带宽和存储空间规范。最后，由于有良好的主题爬虫使用有限的资源，用户已经在使用个人电脑为基础的实现。最后简单的主题爬虫可能会成为执行相关材料的综合有哪些信誉好的足球投注网站用户的选择。 虽然前途，主题抓取技术仍处于起步阶段。主题的主要开放问题是，沿着爬行路线分配本文的如下：第2提供更详细概述第3节描述体系结构和我们的实施。第4节显示一些测试算法与现有主题爬的比较，第5我们通过讨论扩展和影响。 Figure 1: a) A standard crawler b) A focused crawler 主题爬虫高效地爬取关于特定主题文件并且指导基于网页内容和链接结构的有哪些信誉好的足球投注网站。图1形象描述了彻底的广度优先爬虫和典型的主题爬虫的区别。主题爬虫实现了一个策略，这个策略把它下载的每个链接都按值关联起来。链接按值排序并插入队列。一个最好的第一个有哪些信誉好的足球投注网站通过从队列头部爆出下一个页面进行分析而执行的。这种策略保证了爬虫优先选择最可能爬行路径。 最简单的主题爬虫使用相关种类的固定模式，通常作为一个分类编码，以评估当前文档链接的为主题的相关文件（以下简称孩子）。