A Survey on Web Crawling
Web信息采集研究进展

As a basic component of search engine and a series of other services on Web,Web crawler is playing an important role. Roughly,a Web crawler is a program which automatically traverses the Web by downloading documents and following links from page to page. This article detailedly explains the principles and difficulties on the Web crawler, comprehensively argues several hot directions of Web crawler,and at last views the new direction of Web crawler.