Show simple item record  

dc.contributor.authorBouckaert, Remco R.
dc.coverage.spatialConference held at Banff, Alberta, Canadaen_NZ
dc.identifier.citationBouckaert, R. R. (2004). Estimating replicability of classifier learning experiments. In Proceedings of the 21st International Conference on Machine Learning Banff, Canada, 2004.en_NZ
dc.description.abstractReplicability of machine learning experiments measures how likely it is that the outcome of one experiment is repeated when performed with a different randomization of the data. In this paper, we present an estimator of replicability of an experiment that is efficient. More precisely, the estimator is unbiased and has lowest variance in the class of estimators formed by a linear combination of outcomes of experiments on a given data set. We gathered empirical data for comparing experiments consisting of different sampling schemes and hypothesis tests. Both factors are shown to have an impact on replicability of experiments. The data suggests that sign tests should not be used due to low replicability. Ranked sum tests show better performance, but the combination of a sorted runs sampling scheme with a t-test gives the most desirable performance judged on Type I and II error and replicability.en_NZ
dc.relation.ispartofTwenty-first international conference on Machine learning - ICML '04
dc.subjectcomputer scienceen_NZ
dc.subjectmachine learningen_NZ
dc.titleEstimating replicability of classifier learning experimentsen_NZ
dc.typeConference Contributionen_NZ
dc.relation.isPartOfProceedings of the Twenty-first International Conference on Machine Learningen_NZ
pubs.finish-date2004en_NZ York, NYen_NZ

Files in this item


There are no files associated with this item.

This item appears in the following Collection(s)

Show simple item record