The capabilities delivered by the day-one consultant mark the culmination of surgical training. They involve mastery of a complex set of skills and knowledge that must be integrated and applied to broad clinical activities in order to manage patients effectively. 

Traditionally, trainee assessment has focused on individual competencies, but the 2021 surgical curriculum defines the end point of training by the attainment of high-level ‘outcomes’, which emphasise a more holistic approach to assessment. These outcomes are known as Generic Professional Capabilities (GPCs)¹ and Capabilities in Practice (CiPs)². The GPCs focus on professional behaviours while the CiPs incorporate the associated clinical skills and knowledge. 

Poor performance in any area of the GPCs and CiPs could significantly impact a surgeon’s ability to deliver safe, effective and high-quality care. These professional and clinical capabilities are assessed through the new Multiple Consultant Report (MCR), which compares trainee performance with the endpoint of training during training and aims development towards that end. As part of an ongoing review process and with sufficient data available, we underook a study to evaluate the effectiveness of the MCR in the first year. 

Questions we asked

The MCR enables trainers to rate trainees on each of the nine GPC domains as either ‘Appropriate for Phase’ or as having  ‘Areas for Development’, and use the provided descriptors to support the ratings. Each CiP is rated according to how much supervision trainees need, charting their ability from novice to expert. The end of training is marked by the ability to act at the level of a day-one consultant, level IV, while level V captures exceptional performance. Trainees perform a Self-Assessment (S-A) at the same time, using the same tool. 

The study looked at how closely the assessment of trainees through the MCR and the trainees’ own ratings as expressed through the S-A corresponded with each other. The MCR and S-A assessment data used in the study were extracted from the database of the Intercollegiate Surgical Curriculum Programme³. The data set for analysis comprised the trainee and trainer ratings on each GPC domain and CiP taken from the S-As and MCRs in the first academic year of curriculum introduction (August 2021 to July 2022). We looked at how trainees’ S-A ratings compared with their trainers’ MCR ratings. 

Although the study made no judgement about which ratings were more ‘accurate’, a good degree of agreement might indicate that perspectives were aligned, that the assessment was working well and that there was good communication between trainees and trainers. The comparison included determining whether there were any differences in the degree of agreement for male and female trainee cohorts. 

We also considered whether there was greater agreement between trainees and trainers in the final assessment if a midpoint assessment had taken place, providing insight into whether the midpoint assessment played a role in improving communication between trainees and trainers. 

The study focused on the results of the final S-As and MCRs since these were used to inform the assigned educational supervisor’s end-of-placement report for the Annual Review of Competence Progression, and therefore the population used had both final S-As and MCRs. GPCs and CiPs were considered separately and only the five generic CiPs were included as they represented a good sample size. We used Microsoft Excel™ 2016 for comparative analysis.

Insights from the data

The database revealed 6,344 of 7,129 trainees (89%) had records of completion of both final S-As and MCRs in the first year of the new curriculum. The GPCs showed a very high degree of agreement between trainee and trainer perspectives (see Table 1). Use of the Appropriate for Phase rating outweighed the Areas for Development rating by 83% in S-As and 82% in MCRs. There was 68% trainee/trainer agreement of the Appropriate for Phase rating across all domains. 

The level of agreement between trainees and trainers on GPCs might reflect a recognition of the nature of professional standards consistent with working daily with others in the multi-professional team where good behaviours are predominantly fostered. 

Chart 1 shows the domains against which the rating of Areas for Development were used and rating frequency. In order to address development needs, it is intended that trainee feedback sessions would be held and that insights gained from these sessions would be incorporated into subsequent performance reviews with the aim of improving performance.

The CiPs showed a more moderate agreement between trainee and trainer views (see Table 2). In the final S-As and MCRs, 21% of trainee/trainer ratings matched across all CiPs. This may suggest that clinical skills are more difficult to self-assess and that, because many clinical interactions take place with trainers rather than peers, the reference point for self-assessment is different. The precise supervision scale given to the clinical elements of each CiP also offers greater scope for judging the practical capabilities of surgery. Trainee ratings demonstrated a generally even positive/negative variance away from those of trainers with no real male/female differences. There was a small (9%) improvement in agreement between final S-As and MCRs when a midpoint S-A/MCR had taken place. 

In perspective

The study showed that there was a good measure of agreement between the assessment of trainees through the MCR and the trainees’ own ratings as expressed through the S-A, with no real difference between males and females. 

It was also shown that conducting a midpoint assessment could have led to a small improvement in trainee/trainer agreement. In contrast with GPCs, CiPs showed a greater variance of trainee/trainer agreement. 

Overall, the study demonstrated that there was a strong alignment of perspectives between trainees and trainers, suggesting that the assessment process held validity and that effective communication between trainees and trainers existed. However, the study did not examine how far the free text and descriptor features were used to support ratings. 

To gain a more comprehensive understanding of how the new measures impact on achieving curriculum outcomes, further quantitative and qualitative evaluation will continue.

Click here to view additional graphs and tables: