Smarter Cities: Cleaning Electricity, Gas and Water Metered Consumption Data for Social and Urban Research

Sa？etak It is claimed that ‘Big Data’ could help cities become ‘smart’, utilise resources more efficiently, and improve inhabitants’ quality of life. Metered consumption data of electricity, gas and water are collected and stored for each household in Australia and provide a valuable source of information for researchers hoping to understand the patterns of consumption and improve resource utilisation. This work tests the assumption that these datasets are sufficiently ‘clean’ to allow interrogation and details the common errors encountered. An inspection of 1-7 years of meter readings of electricity, gas and water for ~60,000 households in Canberra, Australia as well as all datasets of the local government’s energy and water efficiency programs, found this not to be the case. Common errors found include: missing data, biases in erroneous data, errors generated by the data custodians, duplicate entries, the effect of different custodian objectives and of poor data constraints (free flowing text)