Pitfalls in benchmarking data stream classification and how to avoid them
Bifet, A., Read, J., Žliobaitė, I., Pfahringer, B., & Holmes, G. (2013). Pitfalls in Benchmarking Data Stream Classification and How to Avoid Them. In H. Blockeel et al. (Eds.): ECML PKDD 2013, Part I, LNAI 8188(pp. 465-479). Springer Berlin Heidelberg.
Permanent Research Commons link: https://hdl.handle.net/10289/8517
Data stream classification plays an important role in modern data analysis, where data arrives in a stream and needs to be mined in real time. In the data stream setting the underlying distribution from which this data comes may be changing and evolving, and so classifiers that can update themselves during operation are becoming the state-of-the-art. In this paper we show that data streams may have an important temporal component, which currently is not considered in the evaluation and benchmarking of data stream classifiers. We demonstrate how a naive classifier considering the temporal component only outperforms a lot of current state-of-the-art classifiers on real data streams that have temporal dependence, i.e. data is autocorrelated. We propose to evaluate data stream classifiers taking into account temporal dependence, and introduce a new evaluation measure, which provides a more accurate gauge of data stream classifier performance. In response to the temporal dependence issue we propose a generic wrapper for data stream classifiers, which incorporates the temporal component into the attribute space.