Research on focused crawler based on Bayes classifier
基于贝叶斯分类器的主题爬虫研究*

Focused crawler is the core technology to implement a focused search engine. This paper made a research on focused crawler based on Bayes classifier, introduced the system architecture and key implementation of a focused crawler based on Bayes classifier. The key parts of the crawler include URL queue, crawling history, page downloader and page analyzer. This paper focused on the page ranking algorithm based on Bayes classifier.Used an improved TF-IDF algorithm to extract the characteristics of the page content, and adopted Bayes classifier to compute the page rank. The experiment results show that Bayes classifier is a better choice than PageRank algorithm to implement a focused crawler, under the condition of large network of resources.