Association Between Milestone Ratings and Certification Exam Scores of Medical Residents and Patient Outcomes

Background

In the United States, the education and competency assessment of internal medicine residents are primarily conducted through two methods: the Milestone assessment system and the American Board of Internal Medicine (ABIM) certification exam. However, there is limited research on how these assessments and exam scores relate to inpatient outcomes. Therefore, this study aims to explore the association between physicians’ Milestone ratings and certification exam scores with inpatient outcomes.

Source of Study

This study was conducted by Bradley M. Gray, Jonathan L. Vandergrift, Jennifer P. Stevens, Rebecca S. Lipner, Furman S. McDonald, and Bruce E. Landon. These authors are affiliated with the American Board of Internal Medicine (ABIM), Beth Israel Deaconess Medical Center, Harvard Medical School, among other institutions. The research was published in the JAMA journal on May 6, 2024.

Study Design and Methods

This study adopted a retrospective cohort analysis method, utilizing data from 6898 internal medicine residents who completed training from 2016-2018, and their care records for Medicare service beneficiaries hospitalized between 2017-2019. The primary outcome indicators included 7-day mortality and readmission rates, as well as 30-day mortality, readmission rates, length of stay, and specialist consultation frequency.

Workflow

1. Sample Selection: The study sample included internal medicine residents who completed training from 2016-2018, had valid Milestone ratings, and whose national provider ID could be determined by ABIM. These physicians provided care to Medicare beneficiaries aged 65 and older hospitalized in the U.S. between 2017-2019.

2. Exclusion Criteria: Elective admissions and patients receiving hospice care at the time of admission were excluded to avoid the possibility of physicians being randomly assigned to non-random admissions. The study also excluded patients with uncommon acute short-term hospitalization diagnostic related groups.

3. Assignment and Matching: Physicians were linked to Medicare inpatient claims data using their national provider ID. Physicians were assigned to inpatient cases if they had the most inpatient evaluation and management encounters during the admission period.

4. Outcome Indicators: The primary focus was on 7-day and 30-day mortality and readmission rates, length of stay, and specialist consultation frequency. Analyses included hospital fixed effects and adjustments for patient characteristics, physician years of experience, and year.

Data Processing and Analysis

1. Milestone Ratings: Based on 22 sub-competencies, the ratings were converted to numerical scores, and overall core competency ratings and medical knowledge core competency ratings were constructed. Overall core competency ratings were categorized into low, medium, and high based on score distribution.

2. Certification Exam Scores: Scores from the internal medicine certification exam were categorized into quartiles, using the first attempt scores.

3. Statistical Analysis: Multiple linear regression analysis was conducted, including hospital fixed effects, adjusting for the impact of various variables within the sample. Regression-adjusted differences between categories were calculated, and sensitivity analyses were performed.

Study Results

Among 455,120 inpatient cases, the median age of patients was 79 years, with 56.5% being female. The 7-day mortality and readmission rates were 3.5% and 5.6%, respectively, while the 30-day mortality and readmission rates were 8.8% and 16.6%, respectively.

Major Findings

• Milestone Ratings: No significant association was found between high and low overall core competency ratings or medical knowledge ratings. For example, a high overall core competency rating was not significantly associated with a 2.7% increase in 7-day mortality.

• Certification Exam Scores: Compared to the lowest quartile, the highest quartile had an 8.0% reduction in 7-day mortality and a 9.3% reduction in readmission rates. For 30-day mortality, this association was -3.5%. Higher scores were also associated with more specialist consultations (2.4%).

Conclusion and Significance

The study indicates that certification exam scores of newly trained residents are significantly associated with outcomes of Medicare beneficiaries, while Milestone ratings are not. This suggests that incorporating standardized exam results into resident competency assessments could enhance their effectiveness.

Study Highlights

• Problem Solving: The study is the first to systematically explore the relationship between resident Milestone ratings and certification exam scores with patient outcomes.

• Data Analysis Method: Strict methods were used for data matching and regression analysis to ensure the reliability of the study results.

• Importance of Results: The significant role of certification exam scores indicates that this metric can be an effective tool for assessing a physician’s capability for independent practice.

This study provides important insights into the field of medical education, suggesting that future training and assessments could incorporate more effective evaluation tools to improve the quality of clinical practice among physicians.