Data Extraction from the Web Based on Pre-Defined Schema

With the development of the Internet, the World Wide Web has become an invaluable information source for most organizations. However, most documents available from the Web are in HTML form which is originally designed for document formatting with little consideration of its contents. Effectively extracting data from such documents remains a nontrivial task. In this paper, we present a schema-guided approach to extracting data from HTML pages. Under the approach, the user defines a schema specifying what to be extracted and provides sample mappings between the schema and the HTML page. The system will induce the mapping rules and generate a wrapper that takes the HTML page as input and produces the required data in the form of XML conforming to the user-defined schema. A prototype system implementing the approach has been developed. The preliminary experiments indicate that the proposed semi-automatic approach is not only easy to use but also able to produce a wrapper that extracts required data from inputted pages with high accuracy. This research is partially supported by the National Natural Science Foundation of China under Grant No.60073014. MENG Xiaofeng is a professor of the School of Information, Renmin University of China. He received his M.S. degree from Renmin University of China and the Ph.D. degree from the Institute of Computing Technology, the Chinese Academy of Sciences. His recent research work includes Web data management and mobile data management. He is also interested in DBMS implementation and natural language interfacing. He has published more than 40 research papers in database-related international journals and conferences. Now he is a member of the Database Society of CCF, ACM SIGMOD and IEEE CS. LU Hongjun is a professor at the Department of Computer Science, Hong Kong University of Science and Technology (HKUST). He received his B.Sc. in automatic control from Tsinghua University, China, and M.Sc. and Ph.D. in computer science from the University of Wisconsin, Madison. Before joining HKUST, Dr. Lu served as a principal research scientist at the Computer Science Center of Honeywell Inc. in 1985–1987 and a professor at the National University of Singapore in 1987–2000. His recent research work includes data quality, data warehousing and data mining. He is also interested in development of Internet-based database applications and electronic business systems. He has published more than 100 research papers in databaserelated international journals and conferences. WANG Haiyan is a graduate student in the School of Information, Renmin University of China. She received her B.Sc. in computer application from Renmin University of China in 2000. Her research interests include Web data management. GU Mingzhe is a graduate student in the School of Information, Renmin University of China. She received her B.Sc. in computer application from Renmin University of China in 1999. Her research interests include Web data management.