【摘要】 提出了一种可定制聚焦网络爬虫技术.该技术采用简单的主题描述方法提高网络爬虫的可定制性,利用基于站点页面链接结构的链接导航技术实现对主题信息的高效抓取,通过配置文件实施定制,从而构建一个资源消耗小、数据采集准确性高、可控性强的轻量级聚焦网络爬虫,以满足P2P搜索的需求.文中进一步提出一种增量更新和批量更新相结合的网络爬虫数据更新机制,这种混合更新机制降低了增量更新的实现复杂性,相比批量更新具有更小的资源消耗,实验表明采用这种机制能达到较高的数据新鲜度和召回率.更多还原

【Abstract】 A customized focused crawler is proposed to meet the higher demands of peer-to-peer(P2P) Web search.The crawler employs a simple topic description method for convenient customization and uses link navigation based on link structure of Web sites to achieve efficient crawling.The peer can easily customize a lightweight focused crawler with less resource consumption,higher data crawling precision and better manageability by setting a configuration file.Furthermore,a novel data collection updating mechanism for the crawler is presented.This mechanism is a hybrid of incremental update and batch update,which can reduce the complexity of incremental update and have lower overhead compared with batch update.Experimental results indicate that this hybrid updating mechanism can obtain high freshness and recall.更多还原