Research Commons
      • Browse 
        • Communities & Collections
        • Titles
        • Authors
        • By Issue Date
        • Subjects
        • Types
        • Series
      • Help 
        • About
        • Collection Policy
        • OA Mandate Guidelines
        • Guidelines FAQ
        • Contact Us
      • My Account 
        • Sign In
        • Register
      View Item 
      •   Research Commons
      • University of Waikato Research
      • Computing and Mathematical Sciences
      • Computing and Mathematical Sciences Papers
      • View Item
      •   Research Commons
      • University of Waikato Research
      • Computing and Mathematical Sciences
      • Computing and Mathematical Sciences Papers
      • View Item
      JavaScript is disabled for your browser. Some features of this site may not work without it.

      Transformers for multi-label classification of medical text: an empirical comparison

      Yogarajan, Vithya; Montiel, Jacob; Smith, Tony C.; Pfahringer, Bernhard
      Thumbnail
      Files
      AIME_2021_paper33.pdf
      Accepted version, 226.9Kb
      DOI
       10.1007/978-3-030-77211-6_12
      Find in your library  
      Citation
      Export citation
      Yogarajan, V., Montiel, J., Smith, T., Pfahringer, B. (2021). Transformers for Multi-label Classification of Medical Text: An Empirical Comparison. In: Tucker, A., Henriques Abreu, P., Cardoso, J., Pereira Rodrigues, P., Riaño, D. (eds) Artificial Intelligence in Medicine. AIME 2021. Lecture Notes in Computer Science(LNAI,volume 12721). Springer, Cham. https://doi.org/10.1007/978-3-030-77211-6_12
      Permanent Research Commons link: https://hdl.handle.net/10289/14934
      Abstract
      Recent advancements in machine learning-based multi-label medical text classification techniques have been used to help enhance healthcare and aid better patient care. This research is motivated by transformers’ success in natural language processing tasks, and the opportunity to further improve performance for medical-domain specific tasks by exploiting models pre-trained on health data. We consider transfer learning involving fine-tuning of pre-trained models for predicting medical codes, formulated as a multi-label problem. We find that domain-specific transformers outperform state-of-the-art results for multi-label problems with the number of labels ranging from 18 to 158, for a fixed sequence length. Additionally, we find that, for longer documents and/or number of labels greater than 300, traditional neural networks still have an edge over transformers. These findings are obtained by performing extensive experiments on the semi-structured eICU data and the free-form MIMIC III data, and applying various transformers including BERT, RoBERTa, and Longformer variations. The electronic health record data used in this research exhibits a high level of label imbalance. Considering individual label accuracy, we find that for eICU data medical-domain specific RoBERTa models achieve improvements for more frequent labels. For infrequent labels, in both datasets, traditional neural networks still perform better.
      Date
      2021
      Type
      Conference Contribution
      Publisher
      Springer
      Rights
      © 2021 Springer Nature Switzerland AG.This is the author's accepted version. The final publication is available at Springer via dx.doi.org/10.1007/978-3-030-77211-6_12
      Collections
      • Computing and Mathematical Sciences Papers [1455]
      Show full item record  

      Usage

      Downloads, last 12 months
      11,378
       
       
       

      Usage Statistics

      For this itemFor all of Research Commons

      The University of Waikato - Te Whare Wānanga o WaikatoFeedback and RequestsCopyright and Legal Statement