The MIREX Grand Challenge: A Framework of Holistic User-Experience Evaluation in Music Information Retrieval

Simple item page
dc.contributor.authorHu, Xiaoen_NZ
dc.contributor.authorLee, Jin Haen_NZ
dc.contributor.authorBainbridge, Daviden_NZ
dc.contributor.authorChoi, Kahyunen_NZ
dc.contributor.authorOrganisciak, Peteren_NZ
dc.contributor.authorDownie, J. Stephenen_NZ
dc.date.accessioned2019-11-21T01:49:07Z
dc.date.available2017-01-01en_NZ
dc.date.available2019-11-21T01:49:07Z
dc.date.issued2017en_NZ
dc.description.abstractMusic Information Retrieval (MIR) evaluation has traditionally focused on system‐centered approaches where components of MIR systems are evaluated against predefined data sets and golden answers (i.e., ground truth). There are two major limitations of such system‐centered evaluation approaches: (a) The evaluation focuses on subtasks in music information retrieval, but not on entire systems and (b) users and their interactions with MIR systems are largely excluded. This article describes the first implementation of a holistic user‐experience evaluation in MIR, the MIREX Grand Challenge, where complete MIR systems are evaluated, with user experience being the single overarching goal. It is the first time that complete MIR systems have been evaluated with end users in a realistic scenario. We present the design of the evaluation task, the evaluation criteria and a novel evaluation interface, and the data‐collection platform. This is followed by an analysis of the results, reflection on the experience and lessons learned, and plans for future directions.
dc.format.mimetypeapplication/pdf
dc.identifier.citationHu, X., Lee, J. H., Bainbridge, D., Choi, K., Organisciak, P., & Downie, J. S. (2017). The MIREX Grand Challenge: A Framework of Holistic User-Experience Evaluation in Music Information Retrieval. Journal of the Association for Information Science and Technology, 68(1), 97–112. https://doi.org/10.1002/asi.23618en
dc.identifier.doi10.1002/asi.23618en_NZ
dc.identifier.eissn2330-1643en_NZ
dc.identifier.issn2330-1635en_NZ
dc.identifier.urihttps://hdl.handle.net/10289/13178
dc.language.isoenen_NZ
dc.publisherWileyen_NZ
dc.relation.isPartOfJournal of the Association for Information Science and Technologyen_NZ
dc.rights© 2015 ASIS&T. This is the author's submitted version.
dc.subjectScience & Technologyen_NZ
dc.subjectTechnologyen_NZ
dc.subjectComputer Science, Information Systemsen_NZ
dc.subjectInformation Science & Library Scienceen_NZ
dc.subjectComputer Scienceen_NZ
dc.subjectCATEGORIESen_NZ
dc.subjectNUMBERen_NZ
dc.subjectIMPACTen_NZ
dc.titleThe MIREX Grand Challenge: A Framework of Holistic User-Experience Evaluation in Music Information Retrievalen_NZ
dc.typeJournal Article
pubs.begin-page97
pubs.elements-id134026
pubs.end-page112
pubs.issue1en_NZ
pubs.organisational-group/Waikato
pubs.organisational-group/Waikato/2024 PBRF
pubs.organisational-group/Waikato/DHECS
pubs.organisational-group/Waikato/DHECS/2024 PBRF - DHEC
pubs.organisational-group/Waikato/DHECS/SCMS
pubs.organisational-group/Waikato/DHECS/SCMS/2024 PBRF - SCMS
pubs.publication-statusPublisheden_NZ
pubs.user.infoBainbridge, David (davidb@waikato.ac.nz)
pubs.volume68en_NZ
uow.verification.statusverified
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
GC14UX_preprint.pdf
Size:
867.79 KB
Format:
Adobe Portable Document Format
Description:
Submitted version
Thumbnail Image
Name: GC14UX_preprint.pdf
Size: 867.79 KB
Kind: Adobe PDF
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
Research Commons Deposit Agreement 2017.pdf
Size:
188.11 KB
Format:
Adobe Portable Document Format
Description:
Name: Research Commons Deposit Agreement 2017.pdf
Size: 188.11 KB
Kind: Adobe PDF
Collections
Computing and Mathematical Sciences Papers