Predicting the Benefit of Rule Extraction: A Novel Component in Data Mining

When performing data mining, the selection of data mining technique is a critical decision. Often this choice boils down to whether a transparent model is needed or not. Most research indicates that techniques producing transparent models, such as decision trees, often have an inferior accuracy compared to techniques such as neural networks. On the other hand, models created by neural networks are opaque, which must be considered a serious drawback as they are to be used for decision making. As an alternative, many researchers have tried to reduce this accuracy vs. comprehensibility trade-off by converting the opaque, high accuracy model into a transparent model – a technique termed rule extraction. In this paper, the question addressed is whether it is possible to predict, from the characteristics of a data set, if rule extraction is likely to produce an accurate model. The somewhat surprising answer, found from an empirical study conducted on several publicly available data sets, is that it is possible using only a few data set features. In addition, the study shows that the chosen representation is very important forthe success of rule extraction. The results should be seen as steps in a direction towards a more automated data mining process. The overall ambition is to reduce the need for critical decisions having to be made early in the process and in an ad-hoc fashion.