ABU exams: A look inside the question-writing process

Mr. Lifton is a health care management consultant and adjunct faculty member at the University of Illinois Chicago. He is also a public director of the American Board of Medical Specialties. Urology Times blogs present opinions, advice, and news from urologists and other urology professionals. Opinions expressed by bloggers are their own, and do not necessarily reflect the views of Urology Times or its parent company, UBM Medica. Please let your voice be heard by joining the conversation in the comments section of each post.

Serving as a public (non-physician) director of the American Board of Medical Specialties has brought me into contact with physicians involved in physician education, training, and certification. Gerry Jordan, MD, executive secretary of the American Board of Urology, is one of my ABMS colleagues. As a result of our relationship, I was invited to observe the recent ABU Examination Committee meeting, where questions for the Qualifying Examination are written and evaluated. At the meeting, I had an opportunity to see firsthand how the examination is developed, how it is administered, and the role it plays in the overall certification process for urologists.

I joined 25 physicians-four assigned to each of four task forces, along with consultants who facilitated the working sessions, committee leadership, and members of the AUA Core Curriculum Committee-supported by ABU and AUA staff. As I came to appreciate during the meeting, developing the certification examinations is a team sport, and an impressive team with a strong bench had been assembled. Tony Caldamone, MD, chaired the committee, successfully keeping the meeting focused and on schedule.

Also see - Guideline-based medicine: A take-away from the AUA oral board exam

The Qualifying Examination is the first step in board certification for urologists. (Step two, the Certifying Examination, is the subject of Dr. Henry Rosevear’s blog post “Taking the boards: Not a bad rite of passage after all.”) The Qualifying Examination consists of 300 multiple-choice questions, 200 of which are used to make the certification decision and 100 that are being field tested.

Friday evening was devoted to introductions, with orientation on Saturday morning, and then on to the task at hand. There were working sessions Saturday afternoon, all day Sunday and Monday, and Tuesday morning. Over these 4 days, I witnessed dedicated professionals hard at work on behalf of their colleagues, the medical profession, and the public.

Each of the four task forces addressed a different area of urology:

• physiology, immunology, molecular biology, hypertension, transplantation

• sexual dysfunction, voiding dysfunction

• infections, inflammatory diseases, endocrinopathy, calculus disease, trauma, obstructive uropathy, fertility

• neoplasms, urinary diversion, fistulae

• pediatric urology, anatomy, fluid and electrolyte disorders.

Task force members are all volunteers appointed to a 2-year term with the potential for a second term. They are responsible for drafting questions in advance of the meeting. Each question must be accompanied by a discussion of the concept being tested and supported by citations from “Campbell-Walsh Urology” and other sources, including peer-reviewed publications and AUA guidelines.

Next: "Questions are distributed by subject area to reflect urology training and what urologists can expect in practice."

Questions are distributed by subject area to reflect urology training and what urologists can expect in practice. As he said in his blog post, Dr. Rosevear found that the Certifying Examination tested “treatment options and their complications related to mostly everyday urology.” That’s true of the Qualifying Examination as well, and from my observation that happens by design, not by chance.

On Saturday afternoon, questions that had been field tested previously were evaluated to make sure that they were suitable for use in making the certification decision. Field-tested questions were reviewed for clarity and level of difficulty (eg, not too easy, where almost all candidates answer the question correctly). Questions that had been successfully field tested became part of the bank of questions to be used in the 200-question component of subsequent Qualifying Examinations.

The remainder of the meeting was devoted to reviewing new questions in advance of field testing. The process was collegial, but serious. A draft question went up on the screen, and the facilitator started the discussion: “Is the question clinically relevant? Remember, we want to test what matters.” Questions that got over this hurdle faced further scrutiny. “Is the question clear? Are all of answers clear? We’re not out to confuse the candidates.” And, finally: “Are all of the ‘wrong’ answers really ‘wrong’? We’re not looking to trick people.” Questions on arcane topics, or that related to conditions for which recent studies have identified potentially better treatment options, didn’t meet the criteria and were rejected.

Read: How would MedPAC replace MIPS?

After discussion and revision, questions are ready for field test. Some questions aren’t used because they duplicate existing questions. They might end up being used on the In-Service Examination given annually to urology residents or be made available for the AUA’s Self-Assessment Study Program, for candidates preparing for the Qualifying Examination. Task force members who have developed questions that don’t meet the criteria for relevance and clarity have the option of reworking them for consideration the following year.

Following the meeting, committee leadership reviews all questions under consideration, subjects them to a final editing, and selects those questions to be included in the Qualifying Examination, either for use in the certification decision or for field-testing.

Training, certification, and Lifelong Learning (ABU’s continuing certification program) require commitment and effort on the part of urologists. But they seem to me to be a reasonable part of the physician self-regulation bargain, which benefits physicians as well as the public. Urologists already in practice, as well as physicians relying on them for consultation or referral, get some assurance that their new colleagues are prepared to practice independently.

And what does the public get? We receive assurance that if and when we need urologic care, the board-certified physicians we turn to have demonstrated the knowledge, skills, and judgment necessary to provide the kind of care we want and need.

More from Urology Times:

What one thing about MOC would you change if you could?

ABU: Life-long learning will be relevant, less burdensome

Why your practice should consider team-based care

To get weekly news from the leading news source for urologists, subscribe to the Urology Times eNews.