Evaluating web videos found feasible for RARP peer review

Ann Arbor, MI-Evaluation of online videos is a feasible method for peer review of robot-assisted radical prostatectomy (RARP) surgical skills, according to a pilot project undertaken by the Michigan Urological Surgery Improvement Collaborative (MUSIC). 

Also see: Are you seeing a decline in PSA screening?

The research, which was presented at the 2015 European Association of Urology annual congress in Madrid, Spain and has been published online in European Urology (Jan. 3, 2016), also found that compared to practicing urologists, layperson reviewers were similar in their identification of lower-performing surgeons.

“This pilot project is the first time that practicing surgeons’ skill in performing robotic surgery has been prospectively peer-reviewed and also the first time that crowdsource methodology has been used to assess real surgery,” said first author Khurshid R. Ghani, MD, assistant professor of urology at the University of Michigan, Ann Arbor and co-director of MUSIC.

He explained that the overall aim of the project is to improve the technical quality of RARP throughout Michigan.

“Establishing methodology to assess performance is the first step toward that goal. According to our findings, peer review of online videos is a viable approach and there may be a role for crowdsourcing,” Dr. Ghani said.

The idea of using video review germinated from knowledge of work by the Michigan Bariatric Surgery Collaborative that showed greater technical skill in bariatric surgery, as rated through surgical video review, correlated with better postoperative outcomes (N Engl J Med 2013; 369:1434-42). To help develop the initiative, the MUSIC Coordinating Center turned to James O. Peabody, MD, a robotic prostatectomy pioneer and staff surgeon at Vattikuti Urology Institute, Detroit, to serve as study lead. In addition, they enlisted Thomas S. Lendvay, MD, of the University of Washington, Seattle, who had done work with crowdsourced review.

Next: Exciting potential ramifications

Dr. Peabody told Urology Times that the research has exciting potential ramifications.

“RARP has become the leading surgical treatment for prostate cancer in the United States, and emerging evidence indicates that individual surgeon skill affects important outcomes of potency, continence, and cancer control,” he said.

“Therefore, we believe assessment of surgeons’ technical skill represents a path toward quality improvement, but it is an area that has been understudied.”

The pilot project included RARP case videos voluntarily submitted by 12 surgeons from the MUSIC collaborative. Each video was edited to a running length ≤10 minutes, containing only segments on bladder neck division, nerve sparing, apical dissection, and urethrovesical anastomosis.

Read: Number of pre-op prostate biopsies not linked to outcomes

The videos were reviewed by 25 surgical experts from MUSIC and crowdsourced reviewers from the Amazon Mechanical Turk platform using the validated Global Evaluation Assessment of Robotic Skills (GEARS) instrument. In addition, unedited portions of the video pertaining to the anastomosis were assessed using the Robotic Anastomosis and Competency Evaluation (RACE). Surgeon identity was unknown to the reviewers.

The crowd completed 2,531 ratings over a period of just 21 hours. The expert reviewers were given a 2-week turnaround time for their evaluations. After 15 days and with several reminders, the 12 MUSIC experts rated 318 videos.

Next: Wider range of scores seen with experts

Wider range of scores seen with experts

The analyses showed that the expert rating scores had a much wider range than the crowd.

“In contrast to the crowd workers, who tend not to give very low or very high scores, the experts are more confident in their ability to rate extreme performance levels,” Dr. Ghani explained.

Nevertheless, there was good correlation between the expert and crowd scores on both the GEARS and RACE instruments (r=0.78 and 0.74, respectively), and the urologists and crowd consistently agreed in their rankings of the lower scoring surgeons.

“That is an important finding because in a quality improvement endeavor, we want to identify the surgeons who would benefit from coaching strategies that improve performance,” Dr. Ghani said.

A larger, more comprehensive study will be undertaken that will also examine correlations between surgeon skill ratings and prospectively collected outcomes data. First, however, the group is developing validated rating tools that, unlike GEARS, will be specific to this type of RARP evaluation.

Dr. Ghani and Dr. Peabody suggested that if the larger study confirms that the crowd and experts match in their ratings on the lower performing surgeons, the crowd assessment might serve as a first-pass mechanism in the evaluation process.

“This initiative is not about picking winners and losers,” Dr. Peabody said. “We want to raise the bar of RARP for everyone. We think that is an exciting possibility, and we are fortunate that urologists in Michigan have been forward thinking and supported this project.”

With the focus being on quality improvement, there is also a plan to implement coaching and improvement strategies for all surgeons. In the pilot, all urologists who submitted a video were given customized feedback reports. A survey of the peer reviewers also showed the process of reviewing their colleagues’ videos was considered educational.

MUSIC is funded by Blue Cross Blue Shield of Michigan and the study authors said they are grateful to the Value Partnership Program for supporting this collaborative.

More on Prostate Cancer:

Study: mpMRI predicts lymph node involvement

Study: High surgeon volume linked to post-RP outcomes

AUA, others fight measure penalizing docs who order PSA

Subscribe to Urology Times to get monthly news from the leading news source for urologists.