Performance measures for evolving predictions under delayed labelling classification
Grzenda, M., Gomes, H. M., & Bifet, A. (2020). Performance measures for evolving predictions under delayed labelling classification. In Proceedings of 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1–8). Glasgow, UK: IEEE. https://doi.org/10.1109/IJCNN48605.2020.9207256
Permanent Research Commons link: https://hdl.handle.net/10289/14000
For many streaming classification tasks, the ground truth labels become available with a non-negligible latency. Given this delayed labelling setting, after the instance data arrives and before its true label is known, the online classifier model may change. Hence, the initial prediction can be replaced with additional periodic predictions gradually produced before the true label becomes available. The quality of these predictions may largely vary. Thus, the question arises of how to summarise the performance of these models when multiple predictions for a single instance are made due to delayed labels.In this study, we aim to provide intuitive performance measures summarising the performance of multiple predictions made for individual instances before their true labels arrive. Particular attention is paid to the fact that under the delayed label setting, the emphasis placed on the quality of initial predictions can vary depending on problem needs. The intermediate performance measures we propose complement existing initial and test-then-train performance evaluation when verification latency is observed. Results provided for both real and synthetic datasets show that the new measures can be used to easily rank methods in terms of their ability to produce and refine predictions before the true labels arrive.
This is an author’s accepted version of an article published in the journal: Proceedings of 2020 International Joint Conference on Neural Networks (IJCNN). ©2020 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.