A Validity Study of COMLEX-USA Level 2-CE and COMAT Clinical Subjects: Concurrent and Predictive Evidence (2024)

Abstract

Introduction

Students enrolled in osteopathic medical schools are required to pass a series of licensing examinations, the Comprehensive Medical Licensing Examination (COMLEX-USA), in order to practice osteopathic medicine in unsupervised clinical settings. The Comprehensive Osteopathic Medical Achievement Test (COMAT) is typically administered to osteopathic students after clinical rotations or clerkships to assess knowledge with respect to core disciplines. Although these 2 assessment programs have different purposes, they both measure similar knowledge (ie, clinical competencies). Empirical evidence of the relationship between performance on both examinations supports the validity of these assessment programs and is a critical component in educational measurement.¹ Now that students attending osteopathic and allopathic medical schools can match to the same residency program under the Accreditation Council for Graduate Medical Education's single accreditation system, it is imperative for residency program directors and graduate medical educators to have sufficient information about COMLEX-USA and COMAT in order to make informed decisions about applicants graduating from osteopathic medical schools. Performance on COMAT and its correlation with performance on COMLEX-USA may become increasingly important in discussions regarding moving licensure examinations to pass/fail.

The 4-part COMLEX-USA series includes Level 1, Level 2–Cognitive Evaluation (CE), Level 2–Performance Evaluation (PE), and Level 3. Prior to 2018, all levels were designed to measure proficiency across the same 9 patient presentations and 6 physician tasks. A new test blueprint was implemented in 2018 for Level 3, and in 2019 for Levels 1, 2-CE, and 2-PE. Levels 1, 2-CE, and 3 are computer-based examinations, while Level 2-PE is performance-based. Level 2-CE covers clinical concepts required for medical problem-solving with a concentration on collecting a thorough patient history, analyzing physical examination findings, and making appropriate medical diagnoses.² COMAT is a series of 8 computer-based examinations, each focusing on a specific discipline.³ Some osteopathic medical schools use COMAT scores as an evaluation of students' clerkship or clinical rotation.⁴

Despite having different purposes, COMAT and Level 2-CE have several similarities. First, physicians write items and review content employing the same item writing and test development principles. Secondly, each COMAT represents a unique medical discipline that is also represented on Level 2-CE. Moreover, both examination programs are administered in a time-measured environment and consist of multiple-choice questions. Lastly, osteopathic students typically take COMAT and Level 2-CE during their third or fourth year, although no widespread rule is applied to the sequence.

Several research studies have investigated the concurrent and predictive validity of assessments that purport to measure similar constructs. Prior research has supported the relationship between students' outcomes on clinical rotations and medical licensure examinations.^4,5 After investigating the correlation between osteopathic school performance and Level 2-CE, researchers found a stronger relationship between second-year osteopathic school performance and Level 2-CE than first-year performance (r = 0.70–0.59; N = 171, N = 86, respectively).⁵ Two years following the launch of COMAT, Li and colleagues found moderate correlations between performance on each of the 7 COMAT clinical subjects and Level 2-CE. For most clinical subjects, the correlations were stronger for the second year of administration compared to the first year of administration (r = 0.45–0.65, r = 0.38–0.64, respectively).⁴ These findings were based on a relatively small dataset shortly after the launch of COMAT.

Similar studies have been conducted on the National Board of Medical Examiners' clinical subject “shelf” examinations.^6–9 Zahn and colleagues found performance on shelf examinations accounted for 60% of the variance in Step 2–Clinical Knowledge (CK), where primary care specialties were weighted more heavily (r = 0.64, N = 507).

The current study aimed to answer the following research questions: (1) What is the correlation between performance on Level 2-CE and performance on COMAT clinical subjects? (2) What is the correlation between Level 2-CE discipline subscores and COMAT clinical subject scores? (3) Does performance on each COMAT clinical subject predict performance on Level 2-CE?

Methods

Results

Discussion

When COMAT is administered in part to evaluate students' clinical rotations, scores on COMAT explain 68% of the variance in Level 2-CE scores. In addition, there are significant correlations among scores on COMAT and Level 2-CE discipline subscores. This research expands on prior studies that used smaller sample sizes and did not include COMAT EM.⁴

The evidence presented suggests a moderate, statistically significant relationship between scores on COMAT and Level 2-CE. Similar to findings from prior research that compared NBME “shelf” examinations and Step 2-CK,⁶ the strongest relationships were evidenced between COMAT IM and Level 2-CE scores, and COMAT EM and Level 2-CE scores (r = 0.68, P < .0001 for both examinations). The relationship was further exemplified by the moderate, statistically significant correlations between COMAT scores and their corresponding Level 2-CE discipline subscores (r = 0.31–0.61, P < .0001). Overall, there were stronger relationships between primary care subscores (family medicine and IM), likely due to the emphasis on Level 2-CE as a generalist examination. These results are consistent with prior literature indicating stronger correlations between primary care specialties.⁶ These findings support the concurrent validity between COMAT and Level 2-CE.

The results from the multivariate linear regression provide strong evidence that performance on COMAT predicts performance on Level 2-CE, supporting the predictive validity of examination programs that are designed to measure similar constructs. High performance on COMAT IM and EM is the strongest predictor of high performance on Level 2-CE because of the emphasis of IM and EM on Level 2-CE. Another factor that may contribute to the greater weight of EM scores than other subjects may be that COMAT EM is typically the last COMAT subject taken by students.

The ongoing move to joint accreditation makes this study an important component of the broader graduate medical education context. Now that students attending osteopathic and allopathic medical schools participate in the Match, the findings of this research can inform residency program directors when making decisions about osteopathic applicants.

Practical considerations should be taken into account when interpreting the results of this research. First, we included only students with complete data (eg, graduation year) and those who were enrolled in osteopathic medical schools that use COMAT for evaluative purposes. Therefore, these results may not hold for the entire population. Second, the multivariate regression model was based on a smaller sample size consisting of students who took all 8 COMAT due to the relative lower volume of EM administrations. Additionally, this research did not explore any other factors that may be related to Level 2-CE scores (eg, Level 1 scores, clerkship grades, sequencing of COMAT administrations). Furthermore, although COMAT and COMLEX-USA are designed to measure similar topics (eg, disciplines) and have similar computer-based formats, the purposes of the assessments differ. While COMAT is designed to measure discipline-specific knowledge in order for osteopathic medical schools to assess student competencies, Level 2-CE emphasizes primary care specialties as a generalist examination; therefore, discipline subscores on Level 2-CE are less reliable than total scores.

Future research can address these areas by including additional variables (eg, clerkship grades, scores on Levels 1 and 2-PE) or by examining the timing of COMAT and Level 2-CE as other factors that may influence performance on either examination.

Conclusions

The findings from this study indicate that there is a strong positive relationship among performance on COMAT and Level 2-CE, with up to 68% of variance explained. This study is applicable to osteopathic medical schools that do not administer COMAT for evaluative purposes as it indicates a moderate significant relationship between 2 assessments administered to osteopathic students. These findings should be taken into consideration when evaluating osteopathic applicants during the residency application process.

References