Practical bias variance decomposition
Bouckaert, R. R. (2008). Practical bias variance decomposition. In W. Wobcke & M. Zhang (eds), Proceedings of the 21st Australian Joint Conference on Artificial Intelligence Auckland, New Zealand, December 1-5, 2008 (pp. 247-257). Berlin, Germany: Springer Berlin Heidelberg.
Permanent Research Commons link: http://hdl.handle.net/10289/8074
Bias variance decomposition for classifiers is a useful tool in understanding classifier behavior. Unfortunately, the literature does not provide consistent guidelines on how to apply a bias variance decomposition. This paper examines the various parameters and variants of empirical bias variance decompositions through an extensive simulation study. Based on this study, we recommend to use ten fold cross validation as sampling method and take 100 samples within each fold with a test set size of at least 2000. Only if the learning algorithm is stable, fewer samples, a smaller test set size or lower number of folds may be justified.