Efficiency bottlenecks analysis and solution of Web crawler
网络爬虫效率瓶颈的分析与解决方案

The efficiency of a web crawler determines the quality of services a web searching system offers to its users. How to design a more efficient and faster web crawler is becoming a hot issue in the research of web crawler. In order to raise the crawling efficiency of a web crawler, the crawling strategy needs to be reformed. Besides, the design of the web crawler system has to be optimized and its structure also needs to be improved to eliminate bottlenecks. In this paper, an improved scheme of designing a general web crawler was presented through analyzing crawler's structure, application environment and user requirement, and the preferable testing result has proven better efficiency it has.