Study assesses quality of ChatGPT-generated summaries of incontinence research

In this interview, Mubashir S. Billah, MD, highlights the study, “Comparing AI-Generated vs. Cochrane Review Summaries in Incontinence Research: A Comprehensive Analysis,” which was presented at the 2024 American Urological Association (AUA) Annual Meeting in San Antonio, Texas. Billah is a urologist and an assistant professor at Hackensack University Medical Center in New Jersey.

Mubashir S. Billah, MD

Could you describe the background and rationale for this study?

Artificial intelligence is one of the newest technologies available, and finding ways to integrate it into the health care system is a burgeoning field. We wanted to be at the forefront of this unique opportunity. From a research standpoint, we are trying to see how AI can be used to help urologists better appreciate and digest information quickly. That's where artificial intelligence excels. We wanted to narrow the focus of this first research project–we have other research projects going on–and we chose incontinence as a topic. The purpose of this was for us to compare ChatGPT's ability to summarize articles compared to human written summaries. That was the purpose of the study: can artificial intelligence adequately summarize research articles as well as humans summarize research articles?

What were the key findings from this study?

We used a metric looking at readability scores and other standardized metrics to evaluate the summaries generated by humans vs ChatGPT. We found that they were very similar in terms of readability and content. It was very impressive to find that ChatGPT was, on its own, able to generate a summary that was similar using a standardized metric compared to the human generated summaries.

What are some of the implications of these findings?

One day we'll be able to more quickly absorb information. On the plus side, you can read a dozen articles. You can throw in a dozen articles into ChatGPT and read the summary in 20 seconds. The downside is nobody's reading the original article anymore. People are just going to be focusing on reading the summaries, which in some ways is unfortunate.

Are there any other areas that you're looking to expand this research to?

We did publish artificial intelligence research about the ability of ChatGPT to take urology exams, and we found some interesting findings there. We're looking to expand our topics and breadth of research when it comes to artificial intelligence. This is just the beginning. We think artificial intelligence has a wide variety of applications that will benefit urologists both in and out of training.

Are there any next steps planned with this research?

This was a podium presentation at the AUA, which are extremely difficult to get. Podium presentations are the pinnacle of AUA research projects. That means you're presenting in front of a large audience. The AUA is the premier national conference for urology; it is the most competitive, the most difficult to get into. So, for us to be able to present this research, not only at the most prestigious conference, but in the most prestigious form at the AUA really speaks to the volume of the project and its uniqueness in the field. This was just the beginning; we've done the data, we've written the abstract, and the next step is to actually publish the paper. We're actually working on the draft and moving forward towards the publication.

What's also exciting is our upcoming conference. Hackensack [will be] involved in an international conference that we'll be attending soon. Dr. Michael Stifelman, Dr. Mutahar Ahmed, and myself will be traveling to South Korea to present a lot of our research. We have more than 20 abstracts being presented at the World Congress of Endourology in Seoul, South Korea this year. We're very excited to have the opportunity to present at international and nationally renowned conferences within urology. I'm also teaching a course in Korea. I have the privilege of being invited to be a course director and teacher faculty for single-port robotic surgery. So, I'll be teaching single-port robotic surgery in South Korea to both locals and international participants at this conference.

Is there anything else you'd like to add?

It's always a privilege and an honor to mentor and work with the medical students and residents at Hackensack University Medical Center and Hackensack School of Medicine. I continue to enjoy this opportunity, and I thrive from seeing the success of our students and the success of our residents. It's such a privilege to participate in all the research projects that I'm personally involved in and the ones that I work with the students on.