Unfractionated heparin (UFH) is a parenteral anticoagulant used for the prevention and treatment of arterial and venous thromboembolic diseases [1,2]. UFH consists of a heterogeneous mixture of polysaccharides with varying molecular lengths and weights; therefore, direct monitoring of serum drug concentrations to guide optimal dosing is not possible [3,4]. Instead, a surrogate of bleeding time, activated partial thromboplastin time (aPTT), is used to monitor the dose-dependent response [5]. The initial bolus and maintenance doses of UFH are estimated by clinicians using weight-based formulas (units of UFH/kg for bolus and units or UFH kg/hour for maintenance), with the aim of achieving a defined therapeutic aPTT range. Future doses are continually adjusted to maintain this therapeutic range (TR) [6-8], which varies depending on the therapeutic indication [1]. An aPTT value below the TR (subtherapeutic) is linked to reduced efficacy (high probability of recurrence or progression of thromboembolic events), whereas values above the TR (supratherapeutic) are linked to the risk of bleeding [9,10]. For patients with life-threatening thromboembolic events, clinicians aim to rapidly achieve a therapeutic aPTT and maintain a TR for the duration of UFH therapy. In the hospital setting, UFH therapy commences with a bolus (loading) dose followed by a maintenance infusion, and an aPPT is quantified within 12 hours [1,6]. This result provides guidance for further dosing, and clinicians often rely on dosing nomograms (Multimedia Appendix 1).

UFH is an extremely complex and difficult drug to accurately dose. The UFH molecules are distributed freely throughout the body; bind to many physiological sites including clotting factors, endothelial cells, and macrophages [4]; and are eliminated from the body via several physiological pathways. This creates marked variation in its pharmacokinetics and dose response between patients, such that there is no standardized one-dose-fits-all strategy [8-11]. Despite the use of nomograms to optimize dosing, it is difficult to achieve and maintain a TR that places patients at risk. For example, excessive dosing may result in up to 5.5% of patients having a bleeding event [12]. Studies evaluating metrics of safety and effectiveness, such as time to TR, time within TR, and percentage of patients within TR, have demonstrated an inability to predict optimal dosing with confidence [13-15]. The time to TR after initiation of UFH can be as long as 60 hours in some studies, and a recent local study of 200 patients showed a median time to TR of 36 hours [16]. In another study, only 29% of the patients had 2 consecutive therapeutic aPTTs [15] throughout the duration of treatment. Even in large clinical trials with strict patient monitoring, the percentage of patients attaining aPTT in TR within 48 hours is less than 50% [17-19]. Clearly, many factors influence bodily responses to UFH, which are independent of body weight and are not accounted for in current dosing strategies [7].

Machine learning (ML) is a subset of artificial intelligence that identifies patterns in large data sets and encodes them into models to predict new data [20,21]. ML has great potential for providing decision support tools in modern health care [20,22-24], which are developed using large volumes of digitized patient data contained within electronic health records (EHRs) [25-27]. To achieve optimal dosing of UFH, ML methods can potentially be used to develop models that make accurate predictions for the target aPTT in response to UFH dosing. However, there have been few studies to date on how to use ML to optimize UFH dosing [28]. A recent systematic review [28] identified 8 studies using ML for UFH. Out of these, 4 studies predicted aPTT [29-32]; 1 study [33] reported out-of-TR surrogates for aPTT, including bleeding and clotting events; and the remaining 3 studies [34-36] evaluated UFH dosing in hemodialysis patients [28]. To date, 5 studies [29,30,32,33,36] have been conducted in the intensive care units (ICUs) of hospitals in the United States using retrospective data and 3 studies in the dialysis setting [34-36].

A variety of modeling approaches were reported. Four studies reported supervised learning methods including random forests, adaptive boosting, extreme gradient boosting, and neural networks [30,32,34,36]. One study used an unsupervised approach to train the model, which was then fine-tuned using a supervised approach [34]. Three studies also used regression analysis [29,30,34], 2 studies used a reinforcement learning approach to develop their models [33,36], and 1 study [31] compared neural networks with nonlinear mixed-effects modeling methods. Studies have reported a wide range of performance metrics including accuracy, precision, recall, area under the receiver operating characteristic curve (AUC), F₁-score (a combination of precision and recall), and coincidence rates. The study by Su et al [32] reported the best model accuracy at 88%. Ghassemi et al [29,30] reported 2 studies on modeling for the prediction of UFH dosing, with 1 study reporting a model developed using multinomial regression to predict subtherapeutic and supratherapeutic aPTT, which had superior performance to ML methods. Their later work explored 4 modeling methods, including reinforcement learning and neural networks, with modest accuracies ranging from 0.56 to 0.6. Overall, the multinomial regression model outperformed the ML methods and was a more appealing model because of its clinical interpretability, which is important in the context of implementation and stakeholder engagement. All studies provided limited reporting of reproducibility, and, except for one study by Smith et al [31], none were validated in a new cohort. Most recently, Li et al [37] reported the development and validation of a multiclass aPTT model and subsequent dose prediction application for use in the ICU setting using a shallow neural network approach. The top 5 features for both data sets included a patient’s baseline aPTT, patient weight, total UFH administered, serum creatinine, and age. The model reported performance metrics similar to those of prior studies, with an F₁-score of 0.887 to 0.925. As with prior studies, the study population was limited to ICU patients and may not be generalizable to other clinical settings. It also does not provide guidance on the exact dose changes that clinicians desire at the point of care.

As evidenced by a recent systematic review [28], no identified models had their impact evaluated within routine clinical practice, and research remains limited and of variable quality. In this body of work, we aimed to develop a model that could be used in hospital general medical and surgical wards, which overcomes the limitations of previous studies with regard to methods, reporting, and external validation.

Figure 1 depicts the data flow and architecture of the project, which is divided into 5 phases. Phase 1 outlines the data collection in which data files were extracted from EHRs from 5 hospitals, of which 4 were used in model development and the fifth was retained for external validation. In phase 2, the 2 data sets underwent the same phase 2 transformations and mapping process (data blending and imputation), except that the clusters built using the training data set were used unchanged in the validation data set to prevent outcome leakage, where parts of the training data used to create a model were not available at the time of prediction. After data blending, we conducted feature engineering in phase 3, which was again applied to both data sets using the same process. The outputs from the feature-engineering workflow were 2 fact tables: the engineered training data, which were input into the H2O Driverless AI tool to build the ML models (phase 4), and the validation data used to validate the model predictions (phase 5). The same pipeline structure, as shown in Figure 1, was applied in developing and validating a regression model to predict the exact aPTT value and a multiclass model to predict the aPTT class (subtherapeutic, therapeutic, and supratherapeutic), with minor differences in data blending to prepare the outcome columns.

This research work was granted a low-risk research protocol approval and a waiver of consent by the Metro South Health (MSH) Human Research Ethics Committee for ethical and scientific review (reference number LNR/2019/QMS/54581). We confirm that the work completed in this project is consistent with ethics approval of the acquired research.

EHR data were collected retrospectively for patients admitted between 2017 and 2020 on consecutive admissions to 5 digital hospitals (one health district) in Queensland, Australia, in which UFH was administered for therapeutic purposes. Model development and external validation were undertaken within the Clinical Informatics Division of MSH. We collected data on UFH that were prescribed using a power plan, which is an EHR decision support tool for specific clinical scenarios that facilitates timely ordering of laboratory tests, medication prescribing, and interdisciplinary communication. Four adult-specific power plans that MSH clinicians use, which were used to identify patients eligible for study inclusion, were acute coronary syndrome, deep vein thrombosis or pulmonary embolism, bridging therapy for oral anticoagulants (warfarin replacement), and low–target-range aPTT anticoagulation for neurosurgical patients. This initial patient cohort was then filtered based on the selection criteria defined by the clinician authors:

A total of 2783 hospital admissions were identified, involving 2470 patients at the 4 hospitals in MSH whose data were used for model development and 236 hospital admissions involving 221 patients at the hospital where data were used for external validation.

Next, we determined the data tables to be collected from the EHRs and generated an initial list of features. Using previous studies in the literature and the content expertise of collaborating clinicians, we identified 15 data tables that were intentionally inclusive at this stage, while recognizing that some would be removed later if found to be irrelevant or if the data were incomplete. Multimedia Appendix 2 lists the tables showing the number of features extracted from each table before and after the feature-engineering phase.

Using the identifier codes of enrolled patients, an aPTT fact table was built by blending UFH bolus dose administration with subsequent aPTT assay results. The rules for inclusion were defined in collaboration between data scientists and clinician researchers to be consistent with the existing literature and to eliminate data noise and ensure data consistency. The following rules were applied:

The generation of the aPTT fact table is illustrated in Figure 2. The data blending process was completed in 6 steps, used patient identifiers, and recorded time stamps to connect and filter the data records. The blending process was performed to satisfy the inclusion or exclusion rules previously defined, resulting in a data set of 2158 records for the model development data set and 236 records for the external validation data set.

During the blending of UFH and aPTT data, several features were identified based on clinician input as listed in Textbox 1. For the other tables (Figure S1 in Multimedia Appendix 2), we excluded all records documented after the time the target aPTT had been performed, as derived from the fact table. Looking at the counts, we excluded 3 tables as they had an insufficient number of examples to incorporate into the model, with each having less than (278/2783, 9.98%) of the total records in the fact table. During the blending process, we first identified the columns of interest in each of the remaining tables. For some features, such as age and sex, data were added to the aPTT fact table with minor or no processing. Other features, to be useful, required data to be aggregated, grouped, or converted in some way. For example, a less granular mapping was applied to 166 distinct order catalogs of medications to categorize them into medication classes. However, during modeling, only the antimicrobial group was used because of uneven distributions across the data set for the other groups. A complete feature list with details of the applied processing is included in Table S2 in Multimedia Appendix 2.

Dealing with missing data was the next step after blending all the identified tables into a single fact table (Table 1; Figure S2 in Multimedia Appendix 2 provides more detail on all features and definitions). In general, we used clinician expertise to decide on the imputation methods for achieving the most accurate representation of missing values. Imputing the missing baseline aPTT (Table 1) assumed a normal physiological aPTT value of 30 seconds on the basis of the literature [40] and the median result derived from our training data set. Missing values of patients height and weight were imputed to the mean value of the cohort after grouping by age (bin interval of 10 years), sex, and marital status.

In this phase, the blended and imputed aPTT fact table was used initially to conduct univariate analysis and data visualization, which aimed to inform decisions about building new features and transforming data. However, this process was not separate from data modeling; rather, it was an iterative process where ML models were built on initial features that changed and evolved, thus serving as new feature inputs to the next cycle of modeling. Multimedia Appendix 3 provides details and visualizations of Pearson correlations between features and outcomes in our aPTT fact table.

Table 2 summarizes the demographic data and important features that are most relevant to the blended (training) data set. The definitions of diagnoses were based on the International Classification of Diseases (ICD)-10 codes; however, we were only able to include categories with large frequencies; that is, ACS and VTE. Other diagnoses were grouped as other. All patients’ recorded codes during their admission were used in the grouping process.

The reported aPTT result showed a distribution heavily skewed to the right (Figure 3) and contained outliers, which negatively impacted the performance of a regression model. Although several statistical methods, such as quadratic mean learning [41], can be used to correct this, we chose, on the basis of clinician expertise, to reduce the negative impact of skewness by introducing a floor and ceiling value to target aPTT of 30 and 150. Values less than 30 seconds reflect normal physiological values. The impact of using floor and ceiling values is visualized in residual graphs in Figure S1 in Multimedia Appendix 3, and more feature analysis information is presented in Figures S3-S5 in Multimedia Appendix 2.

Four calculated features were introduced. The first one was the UFH maintenance dose where, unlike the single bolus dose administration, the cumulative maintenance dose was derived based on the total units in the syringe, the infusion period, and the total infusion time before the target aPTT test was performed, excluding any stoppage periods of the infusion (calculation described as follows):

The standard amount of UFH contained in a syringe was 25,000 units (50 mL syringe, 500 IU/mL), and this, together with the total time for the syringe to be emptied with no interruption, indicated the infusion rate as the number of UFH units infused per minute. The second part of the equation (infusion time—infusion stop) aimed to calculate the exact infusion period in minutes. The 3 other calculated features were body size, UFH bolus time, and UFH bolus time; body size is calculated using the following equations:

Finally, we added a cosine cyclical transformation of aPTT time to build 3 features representing the aPTT day of the week, hour of the day, and month of the year. At the end of this phase, we obtained 93 features in the aPTT fact table. Depending on the data distributions, continuous variables were scaled using the Yeo-Johnson transformation [42] from the SciPy Python library [40] or a min-max transformation. Details about the transformation method applied for every feature are provided in Figure S2 in Multimedia Appendix 2, and equation details are available in Multimedia Appendix 4.

External validation was performed using data obtained from patient records at a fifth hospital (Gold Coast University Hospital), which had an exact schema and table structure as the training set. The final data set comprised 236 records, after applying the same inclusion or exclusion criteria as the training data set. We used the method proposed by Archer et al [55] to calculate the sample size sufficient to validate the proposed model and achieve a preselected target for the CI of R². The equations and calculation details are provided in Multimedia Appendix 4. The equation generated a minimum sample size of at least 235 participants, which was achieved.

The best performing model was an ensemble of 4x LightGBM models that were linearly blended. LightGBM is a gradient-boosting framework that uses tree-based learning algorithms [44]. The model relied on 134 features, some of which are well-established as influencing responses to UFH, such as weight and baseline aPTT, with others first identified in this experiment, of which the most important was the time between when the bolus dose was administered and when the aPTT was measured (Table 3). The bolus dose time, baseline aPTT, age, and bolus dose were also relatively important. Weight, size (weight divided by height), and hematological and biochemical parameters, including serum creatinine level, as a measure of renal function, were also among the top 20 features. Table 4 shows the description and selected list of the LightGBM hyperparameters.

The performance metrics of our ensemble and all the baseline models are listed in Table 5. The H2O Driverless AI ensemble model had best performance with RMSE 31.35 (SD 1.37), residual charts provided in Figure S1 in Multimedia Appendix 3. In addition, this baseline model outperformed all other Python-based linear regression models because the tool tested several algorithms, as previously mentioned, and evolved additional features during the modeling process.

The best performing model was a linear blend ensemble of 4 different models with different weights, 2 XGBoost models and 2 LightGBM models (Table 6).

The performance metrics of the ensemble multiclassification model and baseline models built using the SKlearn library in Python are presented in Table 7, with the ensemble model showing superior performance across all metrics, with an accuracy of 0.599 and macro–F₁-score of 0.613. The simple logistic regression model in Python was the second-best performer, highlighting the efficiency of using auto-ML tools for feature engineering, evolution, and model tuning and blending.

Figure 4 shows the confusion matrix for the ensemble model, demonstrating very good accuracy (0.88) for the subtherapeutic class aPTT<70 seconds, intermediate accuracy (0.512 for the supratherapeutic class aPTT>100 seconds, and poor accuracy (0.098) for the therapeutic class aPTT 70 to 100 seconds. This lower accuracy is most likely a result of class imbalance due to the underrepresentation of the therapeutic class in the data set. Nevertheless, predicting patients at risk of recurrent thromboembolic events from underdosing or at risk of bleeding from overdosing is important for clinicians and patients.

For the multiclassification ensemble model, the validation set returned an accuracy of 0.568 (95% CI 0.538-0.598) and an AUC of 0.724 (95% CI 0.714-0.734), which also compares favorably with the corresponding values for the training set cross-validation of 0.599 and 0.735, respectively. In surveying the confusion matrix (Figure 5), the model demonstrated similar accuracy for each class as the training model: 0.899 for the aPTT class <70 seconds, 0.492 for the aPTT class >100 seconds, and 0.078 for the aPTT class 70 to 100 seconds.

This study reports the development and external validation of an ML model for the prediction of aPTT following bolus and maintenance dosing with UFH. The ML models were developed using EHR data from 4 Australian hospitals with the best performing approach, producing an ensemble with 4x LightGBM models with an RMSE of 31.35. As a multiclassification task, the ensemble model achieved an accuracy of 0.599 and an AUC of 0.735. External validation in a new patient cohort at a fifth hospital showed similar results, with an RMSE of 30.52 for the prediction model and an accuracy of 0.568 and AUC of 0.724 for the multiclassification model.

The final model relied on 93 features, including body weight, baseline aPTT, and bolus dose, with others novel to this study and contemporary clinical knowledge, such as hematological and biochemical features (Multimedia Appendices 2 and 3). The most important clinically informative novel features were the time between administration of the bolus dose and aPTT, age, and baseline aPTT. Baseline aPTT, maintenance UFH dose, and time between bolus administration and aPTT as a grouped feature, which had the highest relative importance (Table 3). UFH is considered a high-risk drug with a narrow therapeutic window, and therefore requires patient-specific dosing to ensure safety and effectiveness [1,7]. Determining the optimal initial bolus and maintenance dosing for UFH therapy is challenging because of the many unknown physiological variables that may contribute to its anticoagulant response. Initial bolus dosing based on body weight is currently preferred [19]; however, other variables must influence the response [56,57]. Nomograms and regular aPTT monitoring, which guide subsequent dose adjustments, increase the proportion of patients achieving a target therapeutic aPTT range [6,8]. Unfortunately, local data derived from 2783 patient episodes suggest that this target is achieved in as few as 23.08% of the patients administered UFH.

As UFH is difficult to administer, new anticoagulants have been introduced in the health setting. Although these new anticoagulants, such as direct-acting oral anticoagulants and low–molecular-weight UFHs, have similar effectiveness to UFH in thromboembolic disease, UFH retains an extensive role in hospital practice because of its several advantages [1,2]. Current dosing is based on nomograms, drug action can be quickly reversed if required, the response can be monitored using aPTT, and its short half-life ensures that the drug is quickly eliminated if urgent surgery is required, or bleeding occurs. As per our data, UFH is still a commonly used drug that requires better dosing and monitoring to ensure patient safety than what is currently achievable. Using ML to derive a predictive model offers a possible approach to more accurately predict individual responses to an initial bolus dose of UFH. This information will assist clinicians in estimating the optimal bolus dose. Developing, testing, and deploying these models is becoming more feasible with the advent of large, digitized data sets such as EHRs [22,26,58]; systems that have been implemented in most large hospitals in Australia. Our study demonstrates that many other features exist beyond the traditional weight-based calculations to determine the best UFH bolus dose. This has the potential to improve the safety profile of UFH. EHR data now afford the opportunity to start using model-based dosing strategies and the ability to develop continuous learning ML models in the future [59].

ML is increasingly being used in early phase drug development [22,26,58] and postmarketing dose design, particularly for other high-risk drugs with narrow therapeutic windows, such as warfarin [56,57,60,61], insulin [62], digoxin [63], immunosuppressants, and chemotherapeutics [64]. Using ML models to guide dosing of UFH in acutely ill, unstable medical and surgical patients to minimize thromboembolic events and bleeding events is an important advancement. In developing such models, as shown in our study, a collaborative approach whereby clinicians and data informaticians work in close consultation is essential. Our study used researchers, data engineers, hematologists, pharmacists, and medical practitioners in its design and conduct. This is essential for developing usable artificial intelligence solutions in hospital settings [65].

In this study, an ensemble approach with supervised learning was used. Five other studies have reported using supervised learning in developing models to assist with UFH dosing [29,30,32,34,36]. To date, although 3 report accuracy [32,33,37] superior to that of our ensemble approach, these models were restricted to ICU data sets from the United States and China and are, therefore, not generalizable to the general medical and surgical wards of hospitals where UFH is most frequently administered. Furthermore, compared with all existing studies of ML in UFH dosing, ours was the only one, apart from one small external validation in a hemodialysis setting [31,66], to evaluate model performance when applied to new unseen data. External validation is considered an essential step before assessing the efficacy in controlled clinical trials and subsequent implementation in routine practice [65,67].

The stage is now set for a feasibility study, the implementation of the model in hospital clinical workflows, and, if successful, further evaluation of clinical utility in a trial comparing current standard practice with model-guided bolus dosing. Implementing the model in routine practice requires an easily accessible decision support platform that can prepopulate most, if not all, the features within the model from the EHR without the need for manual input by clinicians. The model will need to rapidly provide guidance at the exact time of decision-making and will not require end users to undertake extensive training in its use [68,69].

Our model was developed and validated using data from EHRs of 5 hospitals and, therefore, should be tested in other health care systems that use EHRs. The modeling approach only applies to adult inpatients admitted to general medical and surgical specialties. ICU patients were excluded from this study. Furthermore, this modeling approach was intended for the prediction of aPTT after a prespecified bolus and maintenance dose, and as such, further work is required to allow dose calculation and adjustment.

Some features (such as activities of daily living assessments), which were included in the 93 influential features, may not always be available at the time of dosing UFH, and appropriate surrogates should be considered in future iterations of the model. It is also important to consider the level of data standardization within the EHR data sets, which may limit the applicability and usefulness of ML-derived models [70]. For example, differences in how features are measured (eg, the weight and height using different scales), differences in aPTT assays, or different locations of data in EHR may affect model performance and generalizability.

Finally, similar to many other dosing regimens for intravenous drugs, a perfect algorithm for UFH is probably not achievable, as UFH interacts with a myriad of hematological and physiological factors that may affect its anticoagulant effect. Many of these cannot be measured or remain unknown. The goal of our study was to produce and validate a predictive model for UFH dosing that is significantly more accurate than the current weight-based nomograms that have been in use for many years.

This study reports the development and validation of an auto–ML-built ensemble modeling approach for predicting aPTT results and determining their therapeutic classification within 12 hours of administration of a de novo UFH bolus accompanied by a UFH maintenance infusion. ML models were developed using retrospective data from the EHRs of the 4 hospitals. These models were shown to have a consistent performance when applied to an external data set from a fifth hospital. To our knowledge, this is the first study of ML regression and multiclassification models applied to UFH dosing that has used auto-ML tools in model development and conducted external validation. Future work should include the optimization of model performance and its redesign and incorporation into a dose calculation software tool that can be easily used by clinicians at the point of care.