Research Commons
      • Browse 
        • Communities & Collections
        • Titles
        • Authors
        • By Issue Date
        • Subjects
        • Types
        • Series
      • Help 
        • About
        • Collection Policy
        • OA Mandate Guidelines
        • Guidelines FAQ
        • Contact Us
      • My Account 
        • Sign In
        • Register
      View Item 
      •   Research Commons
      • University of Waikato Research
      • Computing and Mathematical Sciences
      • Computing and Mathematical Sciences Papers
      • View Item
      •   Research Commons
      • University of Waikato Research
      • Computing and Mathematical Sciences
      • Computing and Mathematical Sciences Papers
      • View Item
      JavaScript is disabled for your browser. Some features of this site may not work without it.

      The MIREX Grand Challenge: A Framework of Holistic User-Experience Evaluation in Music Information Retrieval

      Hu, Xiao; Lee, Jin Ha; Bainbridge, David; Choi, Kahyun; Organisciak, Peter; Downie, J. Stephen
      Thumbnail
      Files
      GC14UX_preprint.pdf
      Submitted version, 867.7Kb
      DOI
       10.1002/asi.23618
      Find in your library  
      Citation
      Export citation
      Hu, X., Lee, J. H., Bainbridge, D., Choi, K., Organisciak, P., & Downie, J. S. (2017). The MIREX Grand Challenge: A Framework of Holistic User-Experience Evaluation in Music Information Retrieval. Journal of the Association for Information Science and Technology, 68(1), 97–112. https://doi.org/10.1002/asi.23618
      Permanent Research Commons link: https://hdl.handle.net/10289/13178
      Abstract
      Music Information Retrieval (MIR) evaluation has traditionally focused on system‐centered approaches where components of MIR systems are evaluated against predefined data sets and golden answers (i.e., ground truth). There are two major limitations of such system‐centered evaluation approaches: (a) The evaluation focuses on subtasks in music information retrieval, but not on entire systems and (b) users and their interactions with MIR systems are largely excluded. This article describes the first implementation of a holistic user‐experience evaluation in MIR, the MIREX Grand Challenge, where complete MIR systems are evaluated, with user experience being the single overarching goal. It is the first time that complete MIR systems have been evaluated with end users in a realistic scenario. We present the design of the evaluation task, the evaluation criteria and a novel evaluation interface, and the data‐collection platform. This is followed by an analysis of the results, reflection on the experience and lessons learned, and plans for future directions.
      Date
      2017
      Type
      Journal Article
      Publisher
      Wiley
      Rights
      © 2015 ASIS&T. This is the author's submitted version.
      Collections
      • Computing and Mathematical Sciences Papers [1454]
      Show full item record  

      Usage

      Downloads, last 12 months
      245
       
       
       

      Usage Statistics

      For this itemFor all of Research Commons

      The University of Waikato - Te Whare Wānanga o WaikatoFeedback and RequestsCopyright and Legal Statement