Concept drift detection in delayed and partially labeled data streams: An experimental survey

Abstract

Fully labeled data is the ideal scenario for supervised model training. However, labels might be scarce in many situations, specially when large data is produced as an open-ended stream. Moreover, in dynamic streams, it is assumed that the underlying process generating the data is non-stationary, changing its behavior over time. Therefore, concepts learned by the classification model are likely to change, and non-adaptive models will suffer performance degradation, a phenomenon known as concept drift. The challenge of recognizing and reacting to such changes is even more complex by facing streams with partial or delayed labels. This refers to realistic scenarios where the true label of an incoming point is not immediately available (delay) or might never be available (partial). In this work, we focus on investigating the performance of concept drift detectors in handling delayed and partially labeled data streams. We categorize the main methods from the literature, providing a taxonomy of concept drift detectors and an overview of the most influential approaches for evaluating these detectors under conditions of delayed or partial labeling. Finally, we conduct a series of experiments to analyze the performance of these detectors in scenarios with label scarcity. Concluding, the survey discusses its main limitations and offers insights into potential avenues for future research in the field.