Competency Based Training in Colorectal Surgery

Sandra de Montbrun

Competency based education is moving toward an evidence- base and the early data is encouraging. Sandra de Montbrun reports: “We currently evaluate knowledge and judgment using oral and written examinations. Helen MacRae and I have been working with the American Society of Colon and Rectal Surgeons (ASCRS) to develop an assessment of technical competence in three pilot studies. The first pilot took place at the surgical skills lab at Mount Sinai Hospital at the University of Toronto using cadavers and benchtop models. With 8 stations, we compared fellows in colorectal surgery to fifth year general surgery residents. The fellows outperformed the residents. The study was small (n=20), but it provided initial evidence of validity for the exam – the Colorectal Objective Structured Assessment of Technical Skill (COSATS). We then did a second study using volunteers. (There is a potential bias based on volunteering.) This led to our 3rd study just out in the Annals of Surgery1. There were 8 stations. The study included 70 participants who were concurrently taking their American Colorectal Board exam. It was carried out at Northwestern University in Chicago as a mandatory component of the certification process, although it did not impact certification. These results were compared to oral board scores. When we compared the pass/ fail status on the COSATS and the oral Board exam, we found that the small number of individuals that failed the COSATS exam had actually passed the oral Board exam, suggesting that the COSATS is identifying technical deficiencies in individuals who would otherwise go on to be Board certified by the present certification process. The COSATS seems to be adding validity to the Board exam as a whole. The question that arises is - What are we to do with these data? These participants had been in practice for one year already before they took the exam. This timing question is also pertinent for the oral examination, which is administered after one or more years in practice. The public could ask - Why were these surgeons allowed to practice before their knowledge and skills were fully evaluated and certified?

“Some oppose this kind of examination, in part because it’s a change and, like the new math, further validity evidence is required. We never say we have accomplished that mission. It is a high stakes test with potentially troublesome consequences for those who do not succeed.”

Q: Are you personally convinced of the validity of the COSATS competency test?

A: We all accept the written and oral and we believe in the validity of those assessments. I believe that the COSATS should also be included. The evidence is promising and that’s the basis for my belief. I think skeptics are valuable in that the stakes are high, and we should be challenged to prove that the COSATS is a reliable and valid assessment. However, validity evidence is accrued over time and further study will help push this type of assessment forward. I am a believer in the data so far, but more study is needed. Certainty is rare in medicine and surgery.

“I am working on an exam for general surgery, a GOSATS, under a grant from the Royal College of Physicians and Surgeons of Canada. We are looking at when to do the exam, and asking: Can we develop a reliable exam for senior residents? The General Surgery Objective Structured Assessment of Technical Skill (GOSATS) has been developed and the first pilot study comparing the performance of PGY3 to PGY5 general surgery residents has taken place at the Surgical Skills laboratory September 2016. This data is now being analyzed and the results pending.

Q: Does the COSATS identify talented residents who far exceed their peers in dexterity and may qualify for accelerated training?

A: We want to define competency in a binary fashion, either a resident is competent or they are not. This requires establishing a passing score, a threshold beyond which someone is deemed competent. Residents who are naturally talented would in this binary system be deemed competent.

A recent study we have conducted and published in British Journal of Surgery (2), used a decade of OSATS data to apply standard setting methodologies to this performance based exam. That was the first time in surgery that passing scores have been set for this type of assessment. These methodologies will help guide the setting of passing scores for other examinations in surgery and allow for the differentiation between a candidate that is competent and one that is not. This will also allow potentially for the early identification of residents at risk, and for early remediation if the exam is implemented in the early years of residency.

Sandra and her husband Landy have 2 girls - “Devon aged 9 and Reese aged 5 who walk to our local public school.”

MM

REFERENCES

de Montbrun S, Roberts PL, Satterthwaite L, MacRae H. Implementing and Evaluating a National Certification Technical Skills Examination: The Colorectal Objective Structured Assessment of Technical Skill. Ann Surg. 2016 Jul; 264(1):1-6.

de Montbrun S, Satterthwaite L, Grantcharov TP. Setting Pass Scores for Assessment Of Technical Performance By Surgical Trainees. Br J Surg. 2016 Feb;103(3):300-6.