节点文献
新型分布式Web Spider的设计
Design of new distributed web spider
【摘要】 针对日益突出的网页访问问题,设计了一种新型分布式WebSpider。该分布式WebSpider采用中央控制节点来协调各个webspider的行为,利用宽度优先搜索来获得高质量的网页,通过对DNS缓存来提高访问webserver的速度,以增加并行线程数量的方式来增加网页下载速度,并能动态地加入webspider节点和子中央控制节点,具有很强的灵活性和扩张能力。实验结果表明该分布式WebSpider作为搜索引擎的前端能够快速有效地下载网页,具有较好的性能。
【Abstract】 For the increasingly prominent web access problems,a New Distributed Web Spider(NDWS) is proposed.NDWS uses central control node to coordinate actions of all web spiders,employs breadth-first search to obtain high-quality web pages,caches DNS to improve speed of access to web server,increases number of concurrent threads to increase download speed of web pages.NDWS also can dynamically add web spider node and sub-central-control-node so that NDWS has strong flexibility and expansion capability.Experimental results show that as a front-end of search engine,NDWS can quickly and efficiently download web pages,and has better performance.
【Key words】 central control node; breadth-first search; thread; search engine;
- 【文献出处】 计算机工程与应用 ,Computer Engineering and Applications , 编辑部邮箱 ,2011年16期
- 【分类号】TP393.092
- 【被引频次】5
- 【下载频次】43