Classification is one of the data mining problems receiving enormous attention in the database community. Although artificial neural networks (ANNs) have been successfully applied in a wide range of machine learning applications, they are however often regarded as black boxes, i.e., their predictions cannot be explained. To enhance the explanation of ANNs, a novel algorithm to extract symbolic rules from ANNs has been proposed in this paper. ANN methods have not been effectively utilized for data mining tasks because how the classifications were made is not explicitly stated as symbolic rules that are suitable for verification or interpretation by human experts. With the proposed approach, concise symbolic rules with high accuracy, that are easily explainable, can be extracted from the trained ANNs. Extracted rules are comparable with other methods in terms of number of rules, average number of conditions for a rule, and the accuracy. The effectiveness of the proposed approach is clearly demonstrated by the experimental results on a set of benchmark data mining classification problems.
References
[1]
Wang, L; Sui, TZ. Application of data mining technology based on neural network in the engineering. Proceedings of International Conference on Wireless Communications, Networking and Mobile Computing, Shanghai, China, 21–25 September 2007; pp. 5544–5547.
[2]
Nirkhi, S. Potential use of artificial neural network in data mining. Proceedings of International Conference on Computer and Automation Engineering, Singapore, 26–28 February 2010; pp. 339–343.
[3]
Andrews, R; Diederich, J; Tickle, AB. Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl. Based Syst 1995, 8, 373–389, doi:10.1016/0950-7051(96)81920-4.
[4]
Setiono, R. Extracting rules from neural networks by pruning and hidden-unit splitting. Neural Comput 1997, 9, 205–225, doi:10.1162/neco.1997.9.1.205. 9117899
[5]
Darbari, A. Rule Extraction from Trained ANN: A Survey. Technical Report; Department of Computer Science, Dresden University of Technology: Dresden, Germany, 2000.
[6]
Biryulev, C; Yakymiv, Y; Selemonavichus, A. Research of artificial neural networks usage in data mining and semantic integration. Proceedings of International Conference on Perspective Technologies and Methods in MEMS Design, Lviv, Ukraine, 20–23 April 2010; pp. 144–149.
[7]
Ni, X. Research of data mining based on neural networks. World Academy Sci Eng Tech 2008, 39, 381–384.
[8]
Vesely, A. Neural networks in data mining. Agric. Econ 2003, 49, 427–431.
[9]
Singh, Y; Chauhan, AS. Neural networks in data mining. J. Theo. and App. Inf. Tech 2009, 5, 37–42.
[10]
Baesens, B; Setiono, R; Mues, C; Vanthienen, J. Using neural network rule extraction and decision tables for credit-risk evaluation. Manage. Sci 2003, 49, 312–329, doi:10.1287/mnsc.49.3.312.12739.
[11]
Jacobsson, H. Rule extraction from recurrent neural networks: A taxonomy and review. Neural Comput 2005, 17, 1223–1263, doi:10.1162/0899766053630350.
[12]
Kahramanli, H; Allahverdi, N. Rule extraction from trained adaptive neural networks using artificial immune systems. Expert Syst. Appl 2009, 36, 1513–1522, doi:10.1016/j.eswa.2007.11.024.
[13]
Setiono, R; Baesens, B; Mues, C. A note on knowledge discovery using neural networks and its application to credit screening. Eur. J. Operation. Res 2009, 192, 326–332, doi:10.1016/j.ejor.2007.09.022.
[14]
Tickle, AB; Andrews, R; Golea, M; Diederich, J. The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks. IEEE Trans. Neural Netw 1998, 9, 1057–1067, doi:10.1109/72.728352. 18255792
[15]
Setiono, R; Leow, WK. FERNN: An algorithm for fast extraction of rules from neural networks. Appl. Intell 2000, 12, 15–25, doi:10.1023/A:1008307919726.
[16]
Setiono, R; Liu, H. Symbolic representation of neural networks. IEEE Comput 1996, 29, 71–77.
[17]
Odajima, K; Hayashi, Y; Gong, T; Setiono, R. Greedy rule generation from discrete data and its use in neural network rule extraction. Neural Netw 2008, 21, 020–1028.
[18]
Setiono, R; Liu, H; Lu, H. Effective data mining using neural networks. IEEE Trans. Knowl. Data En 1996, 8, 957–961, doi:10.1109/69.553163.
[19]
Craven, M; Shavlik, J. Using neural networks for data mining. Future Gener. Comput. Syt 1997, 13, 211–229, doi:10.1016/S0167-739X(97)00022-8.
[20]
Anbananthen, SK; Sainarayanan, G; Chekima, A; Teo, J. Data mining using pruned artificial neural network tree (ANNT). Proceedings of IEEE International Conference on Information & Communication Technologies, Damascus, Syria, 24–28 April 2006; pp. 1350–1357.
[21]
Saito, K; Nakano, R. Medical diagnosis expert system based on PDP model. Proceedings of IEEE International Conference on Neural Networks, San Diego, CA, USA, 24–27 July 1988; pp. 1255–1262.
Liu, H; Tan, ST. X2R: A fast rule generator. Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Vancouver, BC, Canada, 22–25 October 1995; pp. 1631–1635.
[26]
Liu, H. A family of efficient rule generators. In Encyclopedia of Computer Science and Technology; Marcel Dekker Inc: New York, NY, USA, 1998; Volume 39, pp. 15–28.
[27]
Setiono, R. Extracting M-of-N rules from trained neural networks. IEEE Trans. Neural Netw 2000, 11, 512–519, doi:10.1109/72.839020. 18249780
[28]
Kamruzzaman, SM; Islam, MM. An algorithm to extract rules from artificial neural networks for medical diagnosis problems. Int. J. Inf. Tech 2006, 12, 41–59.
[29]
Jin, Y; Sendhoff, B. Pareto-based multiobjective machine learning: An overview and case studies. IEEE Trans. Syst. Man Cybern. C Appl. Rev 2008, 38, 397–415.
[30]
Wang, J; Zhang, W; Qin, B; Shi, W. Research on rules extraction from neural network based on linear insertion. Proceedings of WASE International Conference on Information Engineering, Beidaihe, Hebei, China, 14–15 August 2010; pp. 33–36.
[31]
Kwok, TY; Yeung, DY. Constructive algorithms for structured learning in feedforward neural networks for regression problems. IEEE Trans. Neural Netw 1997, 8, 630–645, doi:10.1109/72.572102. 18255666
[32]
Islam, MM; Yao, X; Murase, K. A constructive algorithm for training cooperative neural network ensembles. IEEE Trans. Neural Netw 2003, 14, 820–834, doi:10.1109/TNN.2003.813832. 18238062
Ash, T. Dynamic node creation in backpropagation networks. Connect. Sci 1989, 1, 365–375, doi:10.1080/09540098908915647.
[35]
Fahlman, SE; Lebiere, C. The cascade-correlation learning architecture. Adv. Neural Inf. Proc. Syst 1990, 2, 524–532.
[36]
Setiono, R; Hui, LCK. Use of quasi-Newton method in a feedforward neural network construction algorithm. IEEE Trans. Neural Netw 1995, 6, 273–277, doi:10.1109/72.363426. 18263309
[37]
Kamruzzaman, SM; Hasan, AR; Siddiquee, AB; Mazumder, MEH. Medical diagnosis using neural network. Proceedings of International Conference on Electrical and Computer Engineering, Dhaka, Bangladesh, 28–30 December 2004; pp. 537–540.
[38]
Islam, MM; Akhand, MAH; Rahman, MA; Murase, K. Weight freezing to reduce training time in designing artificial neural networks. Proceedings of International Conference on Computer and Information Technology, Dhaka, Bangladesh, 27–28 December 2002; pp. 132–136.
[39]
Sietsma, J; Dow, RJF. Neural net pruning-why and how? Proceedings of IEEE International Conference on Neural Networks, San Diego, CA, USA, 24–27 July 1988; pp. 325–333.
[40]
Ooyen, A; Nienhuis, B. Improving the convergence of backpropagation algorithm. Neural Netw 1992, 5, 465–471, doi:10.1016/0893-6080(92)90008-7.
Jiawei, H; Kamber, M. Data Mining: Concepts and Techniques; Morgan Kaufmann: San Fransisco, CA, USA, 2001.
[43]
Kaufman, L; Rousseeuw, PJ. Finding Groups in Data: An Introduction to Cluster Analysis; John Wiley & Sons: New York, NY, USA, 2005.
[44]
Raymond, TN; Han, J. Efficient and effective clustering methods for spatial data mining. Proceedings of International Conference on Very Large Data Bases, Santiago de Chile, Chile, 12–15 September 1994; pp. 144–155.
[45]
Islam, MM; Murase, K. A new algorithm to design compact two hidden-layer artificial neural networks. Neural Netw 2001, 4, 1265–1278.
[46]
Quinlan, JR. C45: Programs for Machine Learning; Morgan Kaufmann: San Fransisco, CA, USA, 1993.
[47]
Agrawal, R; Imielinski, T; Swami, A. Database mining: A performance perspective. IEEE Trans. Knowl. Data Eng 1993, 5, 914–925, doi:10.1109/69.250074.
[48]
Yen, S-J; Chen, ALP. An efficient algorithm for deriving compact rules from databases. Proceedings of International Conference on Database Systems for Advanced Applications, Singapore, 11–13 April 1995; pp. 364–371.
[49]
Murphy, PM; Aha, DW. UCI Repository of Machine Learning Databases, (Machine-Readable Data Repository); Department of Information and Computer Science, University of California: Irvine, CA, USA, 1998. Available online: http://archive.ics.uci.edu/ml/ (accessed on 12 April 2011).
[50]
Prechelt, L. PROBEN1-A Set of Neural Network Benchmark Problems and Benchmarking Rules. Technical Report 21/94;; Fakultat fur Informatik, Universitat Karlsruhe: Karlsruhe, Germany, 1994.
[51]
Setiono, R. Techniques for extracting rules from artificial neural networks. Proceedings of International Conference on Soft Computing and Information Systems, Iizuka, Japan, 16–20 October 1998.
[52]
Breiman, L; Friedman, J; Olshen, R; Stone, C. Classification and Regression Trees; Wadsworth and Brooks: Monterey, CA, USA, 1984.
