【摘要】 通用搜索引擎存在不能有针对性地满足用户查询需求和搜索关键词难以准确描述的问题。从数据挖掘和机器学习的角度出发,提出一种基于网络爬虫开源框架Heritrix的可配置主题的聚焦爬虫方法,从指定的站源,根据不同的爬取策略,启动多线程爬取,按照预先设置的关键字和栏目信息进行分类搜索,把最符合条件和要求的信息爬取下来供选择、判断、分析和利用。这种方法可在一定程度上解决搜索引擎查询信息的需求问题,提升用户体验,提高检索效率。更多还原

【Abstract】 During the time of development of the Internet, massive information was generated in the cyber-world and has become an important asset. Meanwhile users’ requirement on information search has become higher and higher. How to search key information quickly and effectively is one of the most difficult problems to solve. Basically, the search engine satisfies needs in data searching. However, needs of users only focusing on special themes or fields cannot be satisfied. Through searching key words only is hard to describe their needs or their problems. Thus, this study focuses on data mining and machine learning and proposes a crawler method of configurable theme focused on crawler system that is based on open-source framework of web crawler Heritrix. To a certain extent this method can solve the above mentioned problems and improve users’ perception and searching efficiency.更多还原