Ajax crawling algorithm based on state transition graph
一种基于状态转换图的Ajax爬行算法

Traditional Web crawler could not meet the challenges of crawling Ajax application, such as JavaScript execution, state identification and navigation, duplicate states elimination etc.By exploring such challenges,this paper introduced state transition graph, based on which an algorithm was proposed to retrieve Ajax states and the background Deep Web. In order to uplift the accuracy,reduce the unnecessary states,improved the algorithm by Ajax fingerprinting and DOM filtering. The experimental results indicate the effectivity and efficiency of this algorithm.