全部 标题 作者
关键词 摘要

OALib Journal期刊
ISSN: 2333-9721
费用:99美元

查看量下载量

相关文章

更多...
PLOS ONE  2009 

The Validation and Assessment of Machine Learning: A Game of Prediction from High-Dimensional Data

DOI: 10.1371/journal.pone.0006287

Full-Text   Cite this paper   Add to My Lib

Abstract:

In applied statistics, tools from machine learning are popular for analyzing complex and high-dimensional data. However, few theoretical results are available that could guide to the appropriate machine learning tool in a new application. Initial development of an overall strategy thus often implies that multiple methods are tested and compared on the same set of data. This is particularly difficult in situations that are prone to over-fitting where the number of subjects is low compared to the number of potential predictors. The article presents a game which provides some grounds for conducting a fair model comparison. Each player selects a modeling strategy for predicting individual response from potential predictors. A strictly proper scoring rule, bootstrap cross-validation, and a set of rules are used to make the results obtained with different strategies comparable. To illustrate the ideas, the game is applied to data from the Nugenob Study where the aim is to predict the fat oxidation capacity based on conventional factors and high-dimensional metabolomics data. Three players have chosen to use support vector machines, LASSO, and random forests, respectively.

References

[1]  Mjolsness E, DeCoste D (2001) Machine learning for science: State of the art and future prospects. Science 93: 2051–2055.
[2]  Bishop CM (2006) Pattern Recognition and Machine Learning (Information Science and Statistics). Springer.
[3]  Hand D (2001) Measuring diagnostic accuracy of statistical prediction rules. Statistica Neerlandica 55: 3–16.
[4]  Breiman L (2001) Random forests. Machine Learning 45: 5–32.
[5]  Claeskens G, Hjort NL (2008) Model selection and model averaging.Cambridge University Press.
[6]  Simon R, Radmacher MD, Dobbin K, McShane LM (2003) Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. J Natl Cancer Inst 95: 14–8.
[7]  Steyerberg EW (2008) Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating (Statistics for Biology and Health). Springer. 1 edition.
[8]  Savage LJ (1971) Elicitation of personal probabilities and expectations. JASA 66: 783–801.
[9]  Hilden J, Habbema JDF, Bjerregaard B (1978) The measurement of performance in probabilistic diagnosis — III. Methods based on continuous functions of the diagnostic probabilities. Methods of Information in Medicine 17: 238–246.
[10]  Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102: 359–378.
[11]  Efron B (1983) Estimating the error rate of a prediction rule: Improvement on cross-validation. Journal of the American Statistical Association 78: 316–331.
[12]  Davison AC, Hinkley DV (1997) Bootstrap methods and their application, volume 1 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge: Cambridge University Press.
[13]  Fu WJ, Carroll RJ, Wang S (2005) Estimating misclassification error with small samples via bootstrap cross-validation. Bioinformatics 21: 1979–1986.
[14]  Jiang W, Simon R (2007) A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification. Statistics in Medicine 26: 5320–34.
[15]  Gerds TA, Cai T, Schumacher M (2008) The performance of risk prediction models. Biometrical Journal 50: 457–479.
[16]  Vapnik V (1982) Estimation of dependences based on empirical data. Springer Series in Statistics. New York: Springer-Verlag. Translated from the Russian by Samuel Kotz.
[17]  Tibshirani R (1996) Regression shrinkage and selection via the LASSO. J Roy Statist Soc Ser B 58: 267–288.
[18]  Efroymson MA (1960) Multiple regression analysis. Mathematical methods for digital computers. New York: Wiley. pp. 191–203.
[19]  Becker U, Fahrmeir L (2001) Bump hunting for risk: a new data mining tool and its applications. Comput Statist 16: 373–386.
[20]  Breiman L (2001) Statistical modeling: The two cultures. (With comments and a rejoinder). Statistical Sciences 16: 199–231.
[21]  Raftery AE, Madigan D, Hoeting JA (1997) Bayesian model averaging for linear regression models. J Amer Statist Assoc 92: 179–191.
[22]  Dawid AP (1986) Probability forecasting. Encyclopedia of Statistical Sciences (9 vols. plus Supplement), Wiley:NY:UK. volume 7. Wiley:NY:UK: pp. 210–218.
[23]  Gerds TA, Schumacher M (2006) Consistent estimation of the expected Brier score in general survival models with right-censored event times. Biometrical Journal 48: 1029–1040.
[24]  Matheson J, Winkler (1976) Scoring rules for continuous probability distributions. Management Science 22: 1087–1096.
[25]  S?rensen TIA, Boutin P, Taylor M, Larsen L, Verdich C, et al. (2006) Genetic polymorphisms and weight loss in obesity: a randomised trial of hypo-energetic high- versus low-fat diets. PLoS Clinical Trials 1: e12.
[26]  Pers T, Martin F, Verdich C, Holst C, Johansen J, et al. (2008) Prediction of fat oxidation capacity using 1h-nmr and lc-ms lipid metabolomic data combined with phenotypic data. Chemometrics and Intelligent Laboratory Systems 93: 34–42.
[27]  Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. The Wadsworth Statistics/Probability Series. Belmont, California.
[28]  Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Statist 32: 407–499.
[29]  R Development Core Team (2008) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. URL . ISBN 3-900051-07-0.
[30]  Liaw A, Wiener M (2002) Classification and regression by randomforest. R News 2: 18–22.
[31]  Dimitriadou E, Hornik K, Leisch F, Meyer D, et al. (2009) e1071: Misc Functions of the Department of Statistics (e1071), TU Wien. R package version 1.5-19.
[32]  Hastie T, Efron B (2007) lars: Least Angle Regression, LASSO and Forward Stagewise. URL . R package version 0.9-7.
[33]  Zhang X, Lu X, Shi Q, Xu XQ, Leung HCE, et al. (2006) Recursive SVM feature selection and sample classification for mass-spectrometry and microarray data. BMC Bioinformatics 7: 197.
[34]  Ma S, Song X, Huang J (2007) Supervised group Lasso with applications to microarray data analysis. BMC Bioinformatics 8: 60.
[35]  Fusaro VA, Mani DR, Mesirov JP, Carr SA (2009) Prediction of high-responding peptides for targeted protein assays by mass spectrometry. Nat Biotechnol 27: 190–8.
[36]  Gerds TA, Schumacher M (2007) On Efron type measures of prediction error for survival analysis. Biometrics 63: 1283–1287.
[37]  Efron B, Tibshirani R (1997) Improvement on cross-validation: The .632+ bootstrap method. Journal of the American Statistical Association 92: 548–560.
[38]  Binder H, Schumacher M (2008) Adapting prediction error estimates for biased complexity selection in high-dimensional bootstrap samples. Statistical Applications in Genetics and Molecular Biology 7: Article 12.
[39]  Politis DN, Romano JP, Wolf M (1999) Subsampling. Springer Series in Statistics. New York: Springer.

Full-Text

Contact Us

service@oalib.com

QQ:3279437679

WhatsApp +8615387084133