The Receiver Operating Characteristic (ROC) is widely used for assessing the performance of classification algorithms. In GIScience, ROC has been applied to assess models aimed at predicting events, such as land use/cover change (LUCC), species distribution and disease risk. However, GIS software packages offer few statistical tests and guidance tools for ROC analysis and interpretation. This paper presents a suite of GIS tools designed to facilitate ROC curve analysis for GIS users by applying proper statistical tests and analysis procedures. The tools are freely available as models and submodels of Dinamica EGO freeware. The tools give the ROC curve, the area under the curve (AUC), partial AUC, lower and upper AUCs, the confidence interval of AUC, the density of event in probability bins and tests to evaluate the difference between the AUCs of two models. We present first the procedures and statistical tests implemented in Dinamica EGO, then the application of the tools to assess LUCC and species distribution models. Finally, we interpret and discuss the ROC-related statistics resulting from various case studies.
References
[1]
Swets, J.A. Signal Detection Theory and ROC Analysis in Psychology and Diagnostics, 1st ed. ed.; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 1996.
[2]
Satchell, S.; Xia, W. Analytic Models of the ROC Curve: Applications to Credit Rating Model Validation. In The Analytics of Risk Model Validation, 1st ed.; Christodoulakis, G., Satchell, S., Eds.; Elsevier: London, UK, 2008.
[3]
Sonego, P.; Kocsor, A.; Pongor, S. ROC analysis: Applications to the classification of biological sequences and 3D structures. Brief. Bioinform. 2008, 9, 198–209, doi:10.1093/bib/bbm064.
[4]
Li, R.; Guan, Q.; Merchant, J. A geospatial modeling framework for assessing biofuels-related land-use and land-cover change. Agr. Ecosyst. Environ. 2012, 161, 17–26, doi:10.1016/j.agee.2012.07.014.
[5]
Pontius, R.G., Jr.; Batchu, K. Using the relative operating characteristic to quantify certainty in prediction of location of land cover change in India. Trans. GIS 2003, 7, 467–484.
[6]
Pontius, R.G., Jr.; Parmentier, B. Recommendations for using the relative operating characteristic (ROC). Landsc. Ecol. 2013. submitted for publication.
[7]
Fawcett, T. An introduction to ROC analysis. Pattern. Recogni. Lett. 2006, 27, 861–874, doi:10.1016/j.patrec.2005.10.010.
[8]
Robin, X.; Turck, N.; Hainard, A.; Tiberti, N.; Lisacek, F.; Sanchez, J.C.; Müller, M. pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinforma. 2011, 12, doi:10.1186/1471-2105-12-77.
[9]
Soares-Filho, B.S.; Rodrigues, H.O.; Follador, M. A hybrid analytical-heuristic method for calibrating land-use change models. Environ. Model. Soft. 2013, 43, 80–87, doi:10.1016/j.envsoft.2013.01.010.
[10]
Peterson, A.T.; Pape？, M.; Soberón, J. Rethinking receiver operating characteristic analysis applications in ecological Niche modelling. Ecol. Model. 2008, 213, 63–72, doi:10.1016/j.ecolmodel.2007.11.008.
[11]
McClish, D.K. Analyzing a portion of the ROC curve. Med. Decis. Making 1989, 9, 190–195, doi:10.1177/0272989X8900900307.
[12]
Santini, S. Computing the Binomial. Coefficients, 2007. Available online: http://arantxa.ii.uam.es/~ssantini/writing/notes/s667_binomial.pdf (accessed on 21 June 2013).
[13]
Pontius, R.G., Jr.; Schneider, L.C. Land-cover change model validation by an ROC method for the Ipswich Watershed, Massachusetts, USA. Agr. Ecosyst. Environ. 2001, 85, 239–248, doi:10.1016/S0167-8809(01)00187-6.
[14]
Lobo, J.M.; Jiménez-Valverde, A.; Real, R. AUC: A Misleading measure of the performance of predictive distribution models. Glob. Ecol. Biogeogr. 2008, 17, 145–151, doi:10.1111/j.1466-8238.2007.00358.x.
[15]
Phillips, S.J.; Anderson, R.P.; Schapire, R.E. Maximum entropy modeling of species geographic distributions. Ecol. Model. 2006, 190, 231–259, doi:10.1016/j.ecolmodel.2005.03.026.
[16]
Mas, J.F.; Farfán, M.; Ghilen, C.; Lima, T.; Soares Filho, B. Una Comparación de dos Enfoques de Modelación de Nicho Ecológico. In Proceedings of Memorias de la XX Reunión SELPER, San Luis Potosí, México, 21–25 October 2013.
[17]
Soares-Filho, B.S.; Alencar, A.; Nepstad, D.; Cerqueira, G.; Vera Diaz, M.; Rivero, S.; Solorzano, L.; Voll, E. Simulating the response of land-cover changes to road paving and governance along a major Amazon highway: The Santarém-Cuiabá Corridor. Glob. Change Biol. 2004, 10, 745–764, doi:10.1111/j.1529-8817.2003.00769.x.
[18]
Mas, J.F.; Pérez-Vega, A.; Clarke, K.C. Assessing simulated land use/cover maps using similarity and fragmentation indices. Ecol. Complex 2012, 11, 38–45, doi:10.1016/j.ecocom.2012.01.004.
[19]
Pontius, R.G., Jr.; Pacheco, P. Calibration and validation of a model of forest disturbance in the western Ghats, India 1920–1990. GeoJournal 2004, 61, 325–334, doi:10.1007/s10708-004-5049-5.
[20]
R Development Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2013.
