Inaccuracies of venous thromboembolism risk assessment and prevention practices among medically ill patients Open Access

Venous thromboembolism (VTE), composed of deep vein thrombosis and/or pulmonary embolism (PE), is a common condition among hospitalized patients.¹ When used appropriately, VTE prophylaxis has been shown to significantly reduce the incidence of deep vein thrombosis and PE, including fatal PE among hospitalized medically ill patients.² Comprehensive, evidence-based VTE risk assessment models exist to help clinicians calculate VTE risk and determine what, if any, VTE prophylaxis is appropriate for each patient.^3-6 

The Padua VTE risk assessment model was developed and validated for medically ill patients.⁴ The Johns Hopkins Hospital implemented Epic as its electronic health record (EHR) system and, expanding upon previous work,^7-9 included a Padua risk assessment model, coupled with computerized clinical decision support, to guide providers in prescribing risk-appropriate VTE prophylaxis for medically ill patients during admission. For this risk assessment tool to be valid, it is crucial that risk assessments are completed accurately and consistently for every hospitalized patient.

The purpose of this study is to evaluate the accuracy of VTE risk assessments completed for medically ill patients, the appropriateness of VTE prophylaxis prescription based on each patient’s risk profile, and the factors associated with VTE development. We hypothesize that some individual risk factors included in the Padua risk assessment model are not appropriately identified by providers, leading to suboptimal VTE prophylaxis prescription and, ultimately, potentially preventable harm from VTE.

This was a retrospective study of consecutively hospitalized medically ill patients at Johns Hopkins Hospital, a quaternary care medical center in urban Baltimore, Maryland. Patients were identified using the general medicine VTE risk assessment tool from 1 January through 30 April 2019. This study was approved by the Johns Hopkins Medicine Institutional Review Board.

When completing a mandatory VTE risk assessment order set, a checklist of VTE risk factors and contraindications to pharmacologic VTE prophylaxis are selected by the admitting provider in the EHR at the time of admission and stored as discrete data elements. Patient demographics, clinical characteristics, VTE risk factors, initial prophylaxis prescribed, and administration of prescribed VTE prophylaxis were retrieved from the EHR. VTE outcomes were identified using International Classificiation of Diseases, tenth revision (ICD-10) codes, and incidence of clinically relevant bleeding were manually validated via chart review, independent of the VTE risk assessment data. A hematologist (M.B.S.) subject matter expert (SME) retrospectively reviewed the chart of all patients and completed an independent risk assessment applying the criteria used by the Padua risk assessment model.⁴ Each patient’s recorded age and body mass index at the time of admission were used. The admission history and physical examination were reviewed for applicable VTE risk factors for each patient.

One of the VTE risk factors in the Padua risk assessment model is reduced mobility. To standardize and quantify mobility, we used the Johns Hopkins highest level of mobility (JH-HLM) scale, which is a reliable and validated 8-point scoring system to describe patient mobility, ranging from bed rest (JH-HLM score 1) to ambulating ≥250 feet (JH-HLM score 8).¹⁰ This score is required to be documented at least once during each nursing shift.¹¹ To align with the Padua risk assessment model’s definition of reduced mobility, immobility was defined as ≥3 consecutive days with a documented JH-HLM score of ≤3, indicating only bed-level mobility.¹² To complete the Padua risk assessment in real time, providers used their clinical judgment about the patient’s expected level of mobility and did not have access to JH-HLM scores at admission; however, when the SME retrospectively completed the Padua risk assessment, the JH-HLM score was used in determining patient mobility.

To avoid missing patients who would benefit from VTE prophylaxis, an institutional multidisciplinary VTE prevention workgroup previously agreed to implement a Padua score cutoff of 3 points, rather than the standard cutoff of 4 points, to identify patients at high risk for VTE. As such, in this study, patients were identified as low risk (Padua score <3) or high risk (Padua score ≥3) for developing VTE and were assessed for contraindications to pharmacologic VTE prophylaxis, renal function, and weight. Compliance with appropriate prophylaxis was determined using a predefined algorithm (Figure 1), which is built into the institutional clinical decision support tool. The EHR system requires documentation of all administered and nonadministered prescribed prophylaxis doses, as well as the reason when not administered.

Descriptive statistics were used for demographic and clinical characteristics. Agreement on individual VTE risk factors between the admitting provider and the SME was described using Cohen kappa. Categorical data were reported as counts and proportions and were analyzed using χ² tests. Continuous data were reported as medians and interquartile ranges and were analyzed using Wilcoxon rank-sum tests. A receiver operator curve and c-statistic were generated to determine the discrimination between the Padua risk scores for developing VTE assigned by the admitting provider and by the SME. A c-statistic value <0.7 was considered weak, a value between 0.7 and 0.8 was considered good, and a value >0.8 was considered excellent.¹³ All analyses were performed using STATA Statistical Software, version 18 (Stata Corp LP, College Station, TX).

A total of 4021 patients were included (Table 1). Admitting providers identified 1058 (26.3%) as high risk for VTE and 2963 (73.7%) as low risk. Upon SME review, 1974 (49.1%) were identified as high risk for VTE, and 2047 (50.9%) were identified as low risk. The distribution of the Padua scores between the SME and admitting providers is shown in the supplemental Figure 1 and supplemental Table 1.

Agreement was achieved between admitting providers and the SME that 818 patients (20.3%) were high risk and 1807 (44.9%) were low risk (agreement 65.3%; κ = 0.2996; P < .001). Upon review by the SME, 1396 patients (34.7%) were found to have been misclassified, which ultimately affected the appropriate VTE prophylaxis regimen recommended. A total of 1156 patients (28.7%) were identified by admitting providers as low risk for VTE, but upon review, they were identified as high risk by the SME. Additionally, 240 patients (6.0%) who were identified as high risk for VTE were determined to be at low risk by the SME (Table 2).

Based on the initial risk assessment completed by the admitting provider, 1894 patients (47.1%) were prescribed the appropriate VTE prophylaxis regimen. Overall, 2141 patients (53.2%) were prescribed risk-appropriate prophylaxis as determined by the SME. Among 2625 patients who were accurately risk assessed, 50.7% were prescribed appropriate VTE prophylaxis for their risk level. For patients at high risk for VTE for whom there was agreement between admitting providers and the SME, 81.5% were prescribed appropriate VTE prophylaxis, and this group also had the highest proportion of contraindications to pharmacologic VTE prophylaxis identified (44.6%). Among patients for whom there was agreement about low VTE risk, only 36.7% were prescribed risk-appropriate prophylaxis due to inappropriate prescription of pharmacologic VTE prophylaxis (Table 2).

Among 1396 patients who were inaccurately risk assessed, 58.0% were prescribed appropriate VTE prophylaxis for the risk level determined by the SME. For patients who were ultimately determined to be at high risk, 67.2% were prescribed appropriate prophylaxis; for patients who were determined to be at low risk, 13.8% were compliant with appropriate VTE prevention (Table 2).

Among individual VTE risk factors, the strongest agreement was observed for age ≥70 years (κ = 0.9842). However, there was significant disagreement between the admitting providers and the SME on respiratory or heart failure (κ = 0.4054), reduced mobility (κ = 0.0419), and acute infection or rheumatologic disorder (κ = 0.1398; Table 3).

A total of 36 patients (1.5%) developed in-hospital VTE. Patients with VTE had a significantly longer median length of stay and a significantly higher median Padua score as determined by the SME (5 vs 2; P < .001); however, the median Padua score determined by the admitting providers was not different between patients with and without VTE (1 vs 1; P = .61; Table 4). The Padua score determined by the admitting providers showed weak discrimination for in-hospital VTE (c-statistic = 0.52), whereas the Padua score determined by the SME showed good discrimination (c-statistic = 0.74).

A total of 13 patients (0.3%) were prescribed pharmacologic VTE prophylaxis on admission and experienced clinically relevant bleeding during hospitalization. The SME determined 6 patients as high risk and 7 patients as low risk for VTE on admission.

Significantly more patients with hospital-acquired VTE were prescribed prophylaxis (86.1% vs 57.5%; P < .001); however, a significantly higher proportion of prescribed VTE prophylaxis doses were not administered among patients who developed VTE (19.6% vs 15.2%; P = .007). Significantly more doses of prescribed VTE prophylaxis were documented as having been refused among patients with VTE (16.9% vs 11.9%; P = .001; Table 4).

We found that 34.7% of medically ill patients were inaccurately assessed for their VTE risk on admission hospital, resulting in inappropriate VTE prophylaxis regimen recommendation. Only half of patients (53.2%) were prescribed risk-appropriate VTE prophylaxis as determined by the SME. This discrepancy was due to inadequate VTE prophylaxis prescribing for high-risk patients and unnecessary VTE prophylaxis prescribing for low-risk patients. Major VTE risk factors most commonly missed by providers on admission were acute infection and reduced mobility. Among patients who developed VTE, the majority were prescribed prophylaxis but missed significantly more prescribed doses than those without VTE.

We found that providers frequently misjudged immobility on admission, leading to significant errors in determining overall VTE risk. This was largely due to the fact that immobility is one of the most heavily weighted factors, contributing 3 points in the Padua risk assessment model. Immobility alone would categorize a patient as being at high risk for VTE at our institution. This underscores the importance of reassessment of VTE risk throughout hospitalization, giving consideration not only to how the patient presents on admission but how their clinical condition improves or worsens. If patients are categorized as low risk but are subsequently immobile for 3 days, their VTE risk score should be increased and prophylaxis initiated. Future efforts to improve VTE risk assessment should leverage existing documentation of known risk factors and measures, such as the JH-HLM mobility score. Additionally, clinical decision support tools should incorporate dynamic assessments of VTE risk factors to proactively prompt reassessment when clinically meaningful changes in VTE risk assessment scoring occur. To improve the accuracy of risk assessment, several VTE risk factors that are already discrete fields in the EHR, including age and obesity, should be autopopulated in the risk assessment tool when applicable. Other risk factors including history of VTE, active cancer, heart failure, and infection were identified in the admission history and physical examination by the SME but were frequently missed by the admitting provider. Efforts should be made to leverage EHR data regarding past medical history and new diagnoses to guide risk assessment and minimize the likelihood of risk factors being overlooked during busy admission process.

To our knowledge, this is the first study to explore the accuracy of VTE risk assessment by providers and its impact on patient outcomes among medically ill patients. A previous study demonstrated that admitting providers were largely unable to predict immobility when admitting patients to the hospital.¹² Similarly, in the neurosurgical population, inaccuracies in VTE risk assessments occurred, leading to incorrect prophylaxis recommendations and inappropriate prescriptions.¹⁴ These inaccuracies undermine efforts to provide decision support at the time decisions are being made to prescribe or not prescribe VTE prophylaxis.

A systematic review of the accuracy of risk assessment models for predicting VTE among hospitalized patients in real-world settings reported low accuracy.¹⁵ Our findings support the notion that VTE risk may not be entirely quantifiable on admission but may require ongoing monitoring during hospitalization. The Padua score calculated based on the risk assessment completed by providers on admission showed weak discrimination for in-hospital VTE, whereas the Padua score calculated based on the retrospective review by the SME showed good discrimination. Although the second review was done by an SME with expertise in the field of VTE prevention and treatment, which may have contributed to greater accuracy, they also had the benefit of retrospective data to objectively quantitatively assess immobility during the initial days of hospitalization, which were not available to providers completing VTE risk assessment at the time of hospital admission. Although some may characterize this as bias with the benefit of hindsight, our study highlights the difficulty providers have in identifying patients who will or will not have poor mobility. To date, we are unaware of any validated prediction tool that can be used to identify, at the time of admission, patients who will have ongoing immobility that puts them at risk for VTE.

A previous study showed significant differences in risk-appropriate VTE prophylaxis prescription among surgical residents for hospitalized surgical patients.¹⁶ These results led to the provision of individualized feedback to admitting providers, resulting in significant improvement in appropriate VTE prophylaxis prescription^17-20; however, these efforts assumed that the VTE risk assessment was performed accurately. These findings suggest that a more robust assessment of both prescription compliance and risk assessment accuracy is warranted to ensure optimal VTE prevention practice.

Alarmingly, significantly more doses of prescribed VTE prophylaxis were missed among patients who developed VTE. Previous studies have identified nonadministration of prescribed VTE prophylaxis as a common occurrence in a variety of hospital settings.^21-23 In particular, nonadministration of VTE prophylaxis seems to be most common among medically ill patients, and consistently, the leading documented reason for nonadministration is patient refusal.^24-26 Previous work has suggested that nurses may underappreciate the harms of VTE and benefits of prophylaxis, inaccurately communicating to patients that ambulation may be sufficient for VTE prevention.^27,28 However, there is no evidence to demonstrate the comparative effectiveness of ambulation vs pharmacologic prophylaxis to prevent VTE among hospitalized patients.²⁹ Furthermore, a growing body of evidence suggests that missed doses of prescribed VTE prophylaxis may be associated with developing VTE in surgical populations.^30-32 This study now shows the association of missed doses of prophylaxis is also associated with in-hospital VTE events in medically ill patients.

Several approaches have been tested and demonstrated to be effective for significantly reducing missed doses of prescribed VTE prophylaxis, with varying levels of resources required corresponding with varying magnitudes of effectiveness. Providing broad education to nurses using learner-centric, scenario-based education to ensure that all nurses have a common understanding of the harms of VTE and benefits of VTE prophylaxis required a limited amount of time and was associated with a 17% reduction in missed doses of prescribed VTE prophylaxis.³³ Providing monthly individualized feedback in the form of a scorecard to nurse managers to share with nurses on their floor required slightly more effort and was associated with a 28% reduction in missed doses.³⁴ Finally, leveraging transactional data from the EHR to notify a health care provider when a dose of VTE prophylaxis is missed, providing just-in-time patient-centered education, has been tested in numerous settings and has been repeatedly associated with a reduction in missed doses by >40%.^34-36 

Our study had several limitations. First, the SME assessment for immobility was based on data documented over the course of hospitalization that were not available to the provider at the time of admission. When the Padua risk assessment model was developed and validated, their method for determining immobility ≥3 days was based on retrospective review of patient medical records as well. The same limitation is true of other VTE risk factors such as active infection. Second, our VTE risk assessment tool at Johns Hopkins uses a Padua score cutoff of 3 to categorize patients as high risk for VTE. This institution-specific practice was implemented due to provider concerns at the time of VTE risk assessment tool development that too few patients would receive prophylaxis if the original threshold of 4 points was used. Future efforts should focus on improving the prediction of immobility and leveraging EHR data for dynamic reassessment throughout hospitalization. Third, our approach required manual review of all VTE risk factors for all hospitalized, medically ill patients by an individual SME to assess accuracy. This is impractical for future quality improvement efforts to assess risk assessment accuracy. Finally, the overall number of VTE events was low, and the study was underpowered to determine what processes are associated with developing VTE. Contributing to our reported low number is that we were only able to reliably identify in-hospital events because many VTE events likely occur after discharge, diagnosed in outpatient settings or outside hospitals.

Among consecutively hospitalized medically ill patients, we found that inaccurate VTE risk assessment and VTE prophylaxis nonadministration were common care defects. These data highlight the importance of accurate continual VTE risk assessment throughout hospitalization. Immobility is a major risk factor for developing VTE, and future efforts should focus on identifying predictors of immobility to better guide VTE risk assessment on admission. Until improved predictors of immobility are identified, routine measurement of mobility and documentation in the EHR should be required and incorporated into VTE risk assessment clinical decision support such that real-time notifications are generated when mobility changes significantly during the hospital stay. In addition, the impact of the duration and severity of immobility on the risk of VTE remain unclear. Future research efforts to develop quantitative measures of mobility and their association with VTE risk are warranted. Incorporation of validated quantitative assessments of mobility in future VTE risk assessment models may help target VTE prophylaxis to patients at greatest risk.

The authors express their gratitude to Sam Sokolinsky for his assistance in obtaining data from the electronic health record system.

B.D.L., D.L.S., M.B.S., and E.R.H. were/are supported by contracts from the Patient-Centered Outcomes Research Institute (PCORI) (CE-12-11-4489 and DI-1603-34596). B.D.L., M.B.S., and E.R.H. were supported by a grant from the Agency for Healthcare Research and Quality (1R01HS024547) and a grant from the National Institutes of Health/National Heart, Lung, and Blood Institute (R21HL129028). E.R.H. is supported by a contract from PCORI (PCS-1511-32745). B.D.L. was supported by the Institute for Excellence in Education Berkheimer Faculty Education Scholar Grant and a contract (AD-1306-03980) from PCORI. E.H. and B.D.L. received research grant support from the Department of Defense/Army Medical Research Acquisition Activity.

Contribution: B.D.L., J.C.Y., E.H.H., E.R.H., and M.B.S. made substantial contributions to the conception or design of the work; B.D.L., A.B., J.C.Y., R.N., K.E.D., J.L., D.L.S., P.S.K., E.H.H., E.R.H., and M.B.S. participated in acquisition, analysis, or interpretation of data; B.D.L. drafted the manuscript; A.B., J.C.Y., R.N., K.E.D., J.L., D.L.S., P.S.K., E.H.H., E.R.H., and M.B.S. reviewed the manuscript critically for important intellectual content; and all authors gave final approval for the manuscript version to be published.

Conflict-of-interest disclosure: M.B.S. has consulted for Attralus, Bristol Myers Squibb, CSL Behring, Janssen, and Pfizer; and has given expert witness testimony in various medical malpractice cases. The remaining authors declare no competing financial interests.

Correspondence: Michael B. Streiff, Johns Hopkins School of Medicine, 1830 E Monument St, Suite 7300, Baltimore, MD 21287; email: mstreif@jhmi.edu.