![[Visit Client Website]](/img/acr/banner_2009.gif)
Search
Print Page
Browse by Day: 
Monday, October 19, 2009: 5:00 PM
104 A (Pennsylvania Convention Center)
Presentation Number: 1222
Purpose: We have reported that the ROSCE is an effective tool for assessing competencies such as professionalism among rheumatology fellows. Our prior ROSCEs have excluded immediate feedback to participants due to concerns that station scores might be skewed by intra-exam effects on fellow performance. However, fellows have voiced a desire for immediate feedback, stating that it would improve the ROSCE's educational value. This year we instituted immediate feedback, and assessed its impact on fellows' performance/ratings as they progressed through the ROSCE stations.
Methods: The 2009 NY ROSCE included four patient-centered stations focusing on rheumatic disease. Rheumatology trainees (n=22) and faculty MD-evaluators (n=24) participated from 6 NY rheumatology training programs. To accommodate feedback time, the 2009 NY ROSCE included fewer stations than prior ROSCEs (reduction of 2-3 stations). Professional actors were trained to role-play patients. At each station, fellows were evaluated by patient- and MD-evaluators (9-point Likert scales for patient care, interpersonal skills and professionalism). After each station fellows received immediate feedback from both evaluators, who were instructed to emphasize constructive criticism. Fellows also rated their own overall performance (immediately after each station but prior to receiving feedback). Following the exam, the average scores obtained as fellows moved from their initial to final stations were determined, and compared to station scores from the 2008 NY ROSCE, in which no immediate feedback was given.
Results: During the 2009 ROSCE, physician global assessments of fellows (n=22) increased progressively from first (mean 6.727) to last (mean 7.149) stations (p=0.05). Patient assessments of fellows demonstrated a similar increase (from 6.015 to 6.419), though this trend did not achieve statistical significance (p=0.14). In contrast, the 2008 ROSCE showed no progressive score improvements across stations. Fellows' self-assessments for individual stations (2009) showed progressive improvement for only 7/22 fellows. In a post-exam questionnaire, fellows reported satisfaction with the immediate feedback process.
Conclusion: A potential strength of the ROSCE is the opportunity for immediate feedback. However, our data indicate that giving immediate feedback improves trainee performance as the ROSCE progresses. Educators considering incorporating feedback into the ROSCE should be aware of both the positive benefits of feedback as a teaching tool, and the possibility that feedback may alter the performance of the ROSCE as an assessment instrument.
Methods: The 2009 NY ROSCE included four patient-centered stations focusing on rheumatic disease. Rheumatology trainees (n=22) and faculty MD-evaluators (n=24) participated from 6 NY rheumatology training programs. To accommodate feedback time, the 2009 NY ROSCE included fewer stations than prior ROSCEs (reduction of 2-3 stations). Professional actors were trained to role-play patients. At each station, fellows were evaluated by patient- and MD-evaluators (9-point Likert scales for patient care, interpersonal skills and professionalism). After each station fellows received immediate feedback from both evaluators, who were instructed to emphasize constructive criticism. Fellows also rated their own overall performance (immediately after each station but prior to receiving feedback). Following the exam, the average scores obtained as fellows moved from their initial to final stations were determined, and compared to station scores from the 2008 NY ROSCE, in which no immediate feedback was given.
Results: During the 2009 ROSCE, physician global assessments of fellows (n=22) increased progressively from first (mean 6.727) to last (mean 7.149) stations (p=0.05). Patient assessments of fellows demonstrated a similar increase (from 6.015 to 6.419), though this trend did not achieve statistical significance (p=0.14). In contrast, the 2008 ROSCE showed no progressive score improvements across stations. Fellows' self-assessments for individual stations (2009) showed progressive improvement for only 7/22 fellows. In a post-exam questionnaire, fellows reported satisfaction with the immediate feedback process.
Conclusion: A potential strength of the ROSCE is the opportunity for immediate feedback. However, our data indicate that giving immediate feedback improves trainee performance as the ROSCE progresses. Educators considering incorporating feedback into the ROSCE should be aware of both the positive benefits of feedback as a teaching tool, and the possibility that feedback may alter the performance of the ROSCE as an assessment instrument.
Keywords: education, medical
Disclosure: J. Berman, None; T. Fields, Takeda Pharmaceuticals, 8 ; A. R. Bass, None; D. M. Lazaro, None; S. Krasnokutsky, None; E. S. Weinstein, None; E. Dwyer, None; H. Do, None; S. Paget, None; M. H. Pillinger, None.