Impact factor may not be best for assessing journals

San Francisco-A urology journal's impact factor may not always represent the best way to capture a publication's value and influence, say researchers from Washington University in St. Louis.

In a comparison with three other commonly used bibliometric indices, the impact factor identified the top journals in urology across multiple other indices, but overall had poor correlation with the other indices.

Three of the four indices, including impact factor, identified European Urology as the top peer-reviewed journal in urology. They differed considerably with respect to the rest of the top five journals.

Impact factor is the most widely used metric for evaluating the impact of peer-reviewed journals in urology. The metric is defined as the ratio of citations in 1 calendar year to the number of citable items published in the previous 2 years.

Though widely used, impact factor has its critics, who have pointed to the metric's potential to be manipulated by editorial policies and reporting of citations as one example of its limitations.

Within the past 5 years, several other bibliometric indices have emerged, and supporters say the newer metrics offer a more objective assessment of a journal's impact than does the impact factor. To examine those claims, Dr. Benway and colleagues chose three newer bibliometric indices for evaluating the quality of journals in the field of urology and compared them with the impact factor.

The three newer indices are defined as follows:

Using a bibliometric utility that queries the Google Scholar database, Dr. Benway and colleagues queried bibliometric indices for all English-language, peer-reviewed urology journals for calendar year 2007.

Consistency seen with newer metrics

The impact factor, g-index, and h-index all rated European Urology as the top urology journal. By the e-index, the journal came in second, behind the Journal of Urology. In addition to European Urology and the Journal of Urology, two other journals ranked in the top five by all four indices: Journal of Sexual Medicine and Prostate.

Urology ranked in the top five as determined by the three newer indices but not the impact factor. Impact factor was the only bibliometric index that rated Nature Clinical Practice Urology in the top five.

"I think urologists can use this type of information to see which journals people find interesting and useful," said Dr. Benway, who presented the team's findings at the AUA annual meeting in San Francisco. "Possibly, it gives an indication where the hot topics are being published and who's publishing hot topics. Bibliometric indices can be used as a general guide, because we all have a limited amount of time available for reading material to keep abreast of the specialty.

"This study is not intended, by any means, to place a valuation on any of the journals," he added. "Every journal in urology serves a very specific purpose, and they all have a good readership and publish high-quality articles."