Fergus Imrie, Bogdan Cebere, Eoin F. McKinney, Michaela van der Schaar
Diagnostic and prognostic models are increasingly important in medicine and inform many clinical decisions. Recently, machine learning approaches have shown improvement over conventional modeling techniques by better capturing complex interactions between patient covariates in a data-driven manner. However, the use of machine learning introduces technical and practical challenges that have thus far restricted widespread adoption of such techniques in clinical settings. To address these challenges and empower healthcare professionals, we present an open-source machine learning framework, AutoPrognosis 2.0, to facilitate the development of diagnostic and prognostic models. AutoPrognosis leverages state-of-the-art advances in automated machine learning to develop optimized machine learning pipelines, incorporates model explainability tools, and enables deployment of clinical demonstrators, without requiring significant technical expertise. To demonstrate AutoPrognosis 2.0, we provide an illustrative application where we construct a prognostic risk score for diabetes using the UK Biobank, a prospective study of 502,467 individuals. The models produced by our automated framework achieve greater discrimination for diabetes than expert clinical risk scores. We have implemented our risk score as a web-based decision support tool, which can be publicly accessed by patients and clinicians. By open-sourcing our framework as a tool for the community, we aim to provide clinicians and other medical practitioners with an accessible resource to develop new risk scores, personalized diagnostics, and prognostics using machine learning techniques.
Machine learning (ML) systems have the potential to revolutionize medicine and become core clinical tools [1]. However, there are a diverse set of challenges that must be overcome prior to routine and widespread ML adoption [2, 3]. In particular, there are substantial technical challenges in developing, understanding, and deploying ML systems which currently render them largely inaccessible for medical practitioners [3–6].
In an attempt to address this, we previously developed AutoPrognosis, an automated machine learning (AutoML) framework that optimizes predictive pipelines [7]. AutoML aims to automate various aspects of the machine learning process. Initial AutoML approaches performed Neural Architecture Search [8] or hyperparameter optimization [9]. More recently, prior work has focused on both selecting the best algorithm and optimizing its hyperparameters from a pre-defined set, known as the combined algorithm selection and hyperparameter optimization (CASH) problem [10, 11].
AutoPrognosis 2.0 is an algorithmic framework and software package that allows healthcare professionals to leverage ML to develop diagnostic and prognostic models. Our framework employs automated machine learning [11] to tackle the challenges faced by clinical users. By automating the optimization of ML pipelines involving data processing, model development, and model training, we reduce the burden on technical experts and turn deriving ML models from an art to a science, democratizing machine learning and opening the field to non-ML domain experts, such as clinicians. We believe that AutoPrognosis 2.0 represents a step-change in algorithmic and software capabilities and can unlock the potential of ML in healthcare for clinical researchers without the requirement for extensive technical capabilities.
Through the lens of our example (diabetes risk prediction), we demonstrate how AutoPrognosis 2.0 can be used to address the challenges of diagnostic and prognostic modeling introduced in Challenges in diagnostic and prognostic modeling.
Challenge 1. Developing powerful ML pipelines.
We begin by using AutoPrognosis to derive a clinical risk score for diabetes. We evaluated the performance of the models using concordance index (C-index) to assess model discrimination, Brier score to assess calibration, and the area under the receiver-operating curve (AUROC) to assess prediction accuracy. We performed imputation five times and conducted 3-fold cross-validation for each of the imputed datasets.
Advances in ML algorithms harbor the potential to transform healthcare; however, major challenges continue to limit their adoption in medicine. In this work, we define these challenges and describe the first integrated, automated framework for diagnostic and prognostic modeling, AutoPrognosis 2.0, that is designed to overcome each obstacle in a way that is accessible to non-expert users, democratizing model construction, understanding, debugging, and sharing.
While AutoPrognosis seeks to address many of the algorithmic challenges of applying machine learning to clinical settings, there remains significant responsibility with the healthcare expert using AutoPrognosis to ensure appropriate study design and data curation. In particular, inappropriate use can result in inaccurate or biased results. For example, if the data used is not representative of the patient population of interest, then the model may not be applicable or accurate in real-world settings. Additionally, if the model is not adequately validated, its use could lead to a greater number of incorrect diagnoses, prognoses, or treatment recommendations than expected, which would be adverse for patient health.
Citation: Imrie F, Cebere B, McKinney EF, van der Schaar M (2023) AutoPrognosis 2.0: Democratizing diagnostic and prognostic modeling in healthcare with automated machine learning. PLOS Digit Health 2(6): e0000276. https://doi.org/10.1371/journal.pdig.0000276
Editor: Gilles Guillot, CSL Behring / Swiss Institute for Translational and Entrepreneurial Medicine (SITEM), SWITZERLAND
Received: November 21, 2022; Accepted: May 17, 2023; Published: June 22, 2023
Copyright: © 2023 Imrie et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: This research has been conducted using the UK Biobank resource. Data from UK Biobank is accessible through a request process (https://www.ukbiobank.ac.uk/enable-your-research/register). The authors had no special access or privileges when accessing the data.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have no competing interests to declare.
Source: https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000276#sec030
