Individual factors affect gains from speech recognition

Jun 16, 2009

While speech recognition (SR) can improve department-wide radiology report turnaround time, gains for individuals appear to be influenced largely by their performance prior to adoption and by nontechnology factors such as individual work habits, according to research from the University of North Carolina (UNC) Hospitals in Chapel Hill.

"[Speech] recognition showed in our study a decrease in report turnaround time for the department, for each section, and the majority of individuals included in the study," said Dr. Arun Krishnaraj. "However, what's interesting is that while the percentage of improvement in report turnaround time after [speech] recognition varied among members of the department, the rank order of the faculty didn't vary. Those who did well before continued to do well after [speech] recognition. This suggests that personality and work ethic has probably more to do with it than the technology itself."

The UNC researchers noted that improvement in report turnaround time (RTT) was not uniform across their radiology department, and sought to analyze the cause for this variation. Krishnaraj presented two UNC studies evaluating SR technology during the American Roentgen Ray Society (ARRS) annual meeting in Boston in April.

In the first study, the team collected data from a radiology department with 30 faculty members serving a 700-bed tertiary academic hospital. Data were collected both nine months before and after implementation of the speech recognition system. A six-month "wash-in" period was excluded from the study, Krishnaraj said.

The researchers ranked faculty members by their average RTT, which was defined as the interval between when a study is available for interpretation and the time the study was verified by an attending physician. They then calculated the percentage of reduction in RTT after implementation of SR, and observed faculty work habits to analyze causes for variation.

Prior to SR, the department had an overall RTT of 28 hours, which dropped to 12.7 hours following implementation, according to Krishnaraj. For the 30 faculty members, RTT ranged from 4.6 hours to 65.9 hours before SR. After adoption of SR, RTT ranged from 1.2 hours to 47.3 hours.

In addition, 28 of the 30 faculty members showed improvement following SR. Of the 30, seven had a greater than 75% improvement, 10 had improvement ranging from 51% to 75%, nine had improvement ranging from 26% to 50%, and four had less than 25% improvement (including the two who did not improve).

Of the 13 faculty members who showed less than 50% improvement in RTT following SR implementation, five were in the top RTT group before and after SR and, thus, showed less room for improvement, he said. The other eight faculty members were more heterogeneous and had variable reasons for their modest improvement, including work habits as well as nonmodifiable factors such as waiting for prior studies for comparison, he said.

To determine the effects of variations in workflow habits, attending physicians were stratified into three groups:

Type 1: Attending physicians who reviewed, revised, and finalized trainee reports at the time of image review.
Type 2: Attending physicians who reviewed images, asked trainees to revise reports, and signed off on reports in batches throughout the day.
Type 3: Attending physicians used the same legacy system as before, having trainees make corrections to reports, and signing off on them once a day or every couple of days.

Type 1 showed a 66% improvement in RTT after SR, while Type 2 showed a 60% gain. Type 3 had only a 24% improvement, he said.

Some common abilities among users who showed the greatest RTT improvement included the desire to embrace and learn about the full feature set of the available technology, the presence of alpha users in their sections, and more frequent checking of report queues, Krishnaraj said.

Those who showed more modest improvement did not invest time in working with the SR vocabulary editor, didn't use SR to finalize reports, or didn't work with IT support to learn more about the technology, he said.

"Recognition of the impact of individual work habits on the effect of productivity-enhancing technology may facilitate the design and implementation of similar technologies in the future and allow for more targeted solutions," he said. "It's [speech] recognition today, it was PACS yesterday, and it will be something else tomorrow."

In a second part of their research, the UNC researchers sought to assess the relationship between case load and RTT for the same radiologists before and after SR implementation.

RTT for the faculty members with the highest number of verified reports per month (1,812 before SR and 2,100 after) were 18.1 hours and 1.2 hours, respectively. The RTT for the faculty members with the lowest number of verified reports (53 before SR and 33 after) were 30.9 hours and 22.5 hours, respectively, he said.

Before SR, each faculty member in the section with the shortest RTT (13.3 hours) verified 575 reports, while the section with the longest RTT (50 hours) verified 526 reports per month, Krishnaraj said. After SR, each faculty member in the section with the shortest RTT (1.8 hours) verified 1,864 reports, while the section with the longest RTT (23.2 hours) verified 144 reports per month.

In three of the eight sections, members within the same section had a threefold difference in average RTT before or after SR, according to Krishnaraj.

"The percentage improvement for [RTT] after [SR] doesn't seem to be affected by our volume; whether before or after [SR], it doesn't seem to affect how quick you can get those reports out," he said. "Factors other than case load may be more important in determining report turnaround time, specifically in academic institutions which have smaller margins."

By Erik L. Ridley
AuntMinnie.com staff writer
June 17, 2009

Radiologists underestimate overlooked SR report errors, January 23, 2009

Speech recognition vendors tackle dialect challenges, December 17, 2008

SR technology is no panacea for reporting work habit change, December 15, 2008

ED report turnaround times drop with speech recognition, November 19, 2008