Study shows limitations of US News ranking system for urologic care

In this interview, Timothy D. Lyon, MD, FACS, highlights the recent Journal of Urology publication “Missing the Mark? US News & World Report Urology Specialty Rankings Do Not Assess the Majority of Urologic Care,”¹ for which he served as the lead author. Lyon is an associate professor of urology at Mayo Clinic in Jacksonville, Florida.

Timothy D. Lyon, MD, FACS

Could you describe the background and rationale for this publication?

Like many urology departments, we follow our US News & World Report specialty ranking and periodically evaluate the data that go into these rankings to look for opportunities for improvement in the clinical care we provide. In 2023, the US News & World Report urology specialty ranking changed with the inclusion of new outpatient complications metrics. This led us to do a deeper dive into our own data to evaluate how that change may [affect] our practice's quality ranking. In the process of doing that, we were extremely surprised to learn how few of the patients that we care for were actually providing data to US News to generate this ranking. That led us to develop this manuscript, both to disseminate this information as well as to provide some constructive feedback to the US News folks about ways we think they may be able to improve their ranking system to better reflect the reality of contemporary urologic care.

How is the US News ranking system currently structured?

As you may know, the US News & World Report urology specialty ranking includes 4 different domains. The largest domain is patient outcomes, which I think makes sense. These are things like 30-day mortality and likelihood of discharge to home after urologic procedure. These are the things that we usually focus on as under our control with patient outcomes. Domain number 2 is structure. These are the things related to the practice in terms of size, number of hospital beds, surgical volume for different procedures, the availability of [intensive care units] or nurse magnet recognition, things of that nature. The third domain is process, which is reputation, or responses on peer surveys from other specialists on how they regard your care. The fourth is patient experience or satisfaction through [Hospital Consumer Assessment of Healthcare Providers & Systems] (HCAHPS) surveys. We focus for this analysis only on the outcomes domain, so patient outcomes after surgery.

The way that US News & World Report identifies patients to evaluate their outcomes is they look at Medicare fee-for-service inpatients, and they identify them using MS-DRGs, or Medicare Severity-Diagnosis Related Groups, which are coded at the time of billing of a hospitalization. [There are] several important points there. Number 1, they have to be inpatients, meaning they have to spend at least 2 midnights in the hospital. Patients that have ambulatory surgery or only stay 1 night in the hospital are not eligible. They have to have traditional Medicare fee-for-service. So, if you have a commercial insurance plan or if you have a Medicare Advantage Plan, which is becoming an increasing proportion of Medicare patients, the data from those patients are not captured by US News for the outcomes metric. I think those are important things to keep in mind when evaluating their methodology.

What were the notable findings from this work?

In this analysis, we started with the year 2022, and we looked at all urologic procedural encounters that happened in our main operating room in the year 2022. We applied the published US News & World Report urology specialty ranking methodology to our own patients to see the overall number that would have counted or provided data to that quality ranking. We were very surprised to see that only 4.6% of our patients that had a urologic surgical procedure in 2022 contributed data to the US News & World Report patient outcomes ranking. We then specifically looked at several procedures that are bread and butter urological practice, things like robotic prostatectomy, [holmium laser enucleation of the prostate] for prostate outlet obstruction, robotic partial nephrectomy, and ureteroscopy with laser lithotripsy for stone. [Of] all those cases, less than 5% of our annual case volume met the criteria for evaluation by the US News patient outcomes metric. Very surprising.

We did look at their new component of potentially preventable outpatient complications. We found that when applying the definition for that metric to our patient population, about half of our patients would have met criteria for evaluation. However, that still comprised a very small component of their ranking system, only 5% of the overall score. Although the new metric did capture more patients, it's not contributing much to the overall ranking score. In addition, they have not yet published specifics about those potentially preventable outpatient complications. So, it is currently unknown how they identify those patients and what complications they are and are not counting. We think more scrutiny is needed of that particular subset to better understand its relevance to our practice.

Lastly, we looked at mortality. This was the most surprising finding to us. 30-day mortality after urologic procedure is the largest single component to the patient outcomes score, so we wanted to look at that specifically. We looked at mortalities over a 4-year period, 2019 to 2022, just because it was a smaller sample size. We specifically examined the details of each of these mortalities to assess appropriate attribution to us as a urology department. Overall, we found 26 mortalities over that period. Seven of these, or 27%, we felt were appropriately attributed to our department, meaning they had had a urologic procedure and died within 30 days of that procedure due to a complication. However, that means that 73% of the mortalities attributed to our department were not admitted to a urology service. These were patients admitted to a medical service with a terminal underlying disease. All of them had either stage 4 widely metastatic cancer, advanced heart failure, or advanced liver failure, and they were admitted with an MS-DRG, or an admitting diagnosis, of either urinary tract infection or urinary obstruction from malignant obstruction from the tumor. All of these deaths occurred after the inevitable progression of their otherwise terminal disease. Some of them did have palliative urinary drainage procedures, nephrostomy tubes or stents, although many of them did not. About 1/3 of them were not only not on the urology service, they were not even seen by a urologist in the 30 days preceding their death. We feel that it is inaccurate to attribute deaths in these type of situations to the quality of care provided by a urology department, since our department wasn't the primary caregivers for these patients. I think that highlights an opportunity for US News to refine the way that they look at their mortality metric to make sure they're measuring what they're actually trying to measure.

What are the take-home messages from this analysis?

I think there are 2 major take-home points from this study. Number 1, I think we've provided some constructive criticism to the US News group as to how to better structure their ranking system for the urology specialty. Contemporary urologic care is increasingly including outpatient and ambulatory procedures, and many things that historically were done only on an inpatient basis are now being increasingly offered on an outpatient basis. Radical prostatectomy comes to mind as a major one. I think this needs to be better reflected in the rankings. Inclusion of an outpatient complications metric is a very good first step in that direction, although we need to better understand how they're measuring and calculating those data. We have also shown that reliance on MS-DRGs lacks appropriate specificity to evaluate urology departments. I think that's really well highlighted in the mortality data we presented, where basically 3/4 of the patients that were attributed to death from a urology department were never admitted to a urology service and had non-urologic problems that led to the cause of their death. Changing the way they identify patients, potentially by using things such as the specialty of the admitting provider or the specialty of a surgical provider, may be a better way to appropriately capture patients.

The second major takeaway is, as for the urology community or urology departments, I think this highlights some of the inherent limitations, not only of the US News & World Report ranking, but also quality metrics in general. Every quality measure is going to have some advantages and some drawbacks. To develop the best comprehensive picture of the quality of a department requires multivariable inputs. We would advocate for looking at multiple different quality measures, patient safety, patient experience, or experience of care, as well as other quality measures in order to develop the most comprehensive picture of the quality of care we provide. At Mayo Clinic, we use the Composite Hospital Quality Index as a measure for this, which takes input from the [Centers for Medicare & Medicaid Services] star rating, from the Leapfrog Safety Group, from HCAHPS surveys, in addition to US News, and combines it all into 1 single score. We use multivariable inputs to assess our quality performance. The future of quality measurement is multiple different inputs to give us the best picture possible of the care we provide.

Is there anything else that you’d like to add?

We are currently in the process of better evaluating the potentially preventable outpatient complications network. That's a software that's been developed by 3M, but they haven't published how they identify cases or how they identify outcomes. So, we're currently in the process of scrutinizing and better evaluating that process, and we will report in the future our findings and ways that it does well and ways in which that process could improve.

Reference

1. Lyon TD, Ugwuowo UC, Pollock BD. Missing the mark? US News & World Report urology specialty rankings do not assess the majority of urologic care. J Urol. Published online November 30, 2023. Accessed January 4, 2024. doi:10.1097/JU.0000000000003795