Radiologists underestimate overlooked SR report errors

The accuracy of speech recognition (SR) software has improved dramatically since the first products were introduced for radiology reporting, but dictation errors still occur. A recent study at one institution found that radiologists underestimated the number of errors occurring in report generation via speech recognition.

For radiologists who self-edit final reports, the process of finding and fixing errors generated by speech recognition technology can be difficult and time consuming. Many radiologists admit that in spite of their best efforts, a percentage of signed reports do contain errors. Do they know the percentage?

If radiologists at the University of Michigan Health System in Ann Arbor are representative, the answer is "no." Some 22% of radiology reports utilized in thoracic oncology conferences at the institution contained significant and potentially confusing errors; meanwhile, a majority of radiologists who responded to a survey believed that the overall error rate for their department did not exceed 10%, according to a study published in the December 2008 issue of the Journal of the American College of Radiology (Vol. 5:12, pp. 1196-1199).

Reducing SR errors

AuntMinnie.com asked several speech recognition system vendors the advice they give to customers who inquire about ways to reduce significant errors.

"With PowerScribe for Radiology, Nuance customers can achieve up to 99% accuracy. At the 95% recognition threshold, we recommend that users continue with speech recognition as their prime tool to capture dictation, creating the final report with speech-recognized dictation, self-editing, and without transcription," said Robert Fleming, senior product manager at Nuance Healthcare.

"However, for users who dictate at recognition rates lower than 95%, for whatever reason, we recommend that they leverage speech-supported transcription. They still dictate ... but rather than editing it, they incorporate medical transcriptionists to review it and make corrections as needed."

David Talton, marketing manager for reporting at Agfa HealthCare in Greenville, SC, advised the following: "Agfa recommends that customers make their own decisions pertaining to report editors. Individual accuracy rates vary dramatically. We offer a variety of workflows within our solutions so that customers may choose to have reports proofread by a so-called correctionist if desired. This workflow can be selected on a report-by-report basis. Often sending a report to a correctionist is the best method to ensure patient care continues uninterrupted."

Although the study is narrow in scope, evaluating only 256 radiology reports, it provides rarely published documentation of uncorrected mistakes found in reports prepared using a speech recognition system. The findings of similar but unpublished analyses are often used by radiology practices and hospital radiology departments to justify the need to hire independent report editors.

Over a 12-week period, lead author Dr. Leslie Quint, a professor of radiology at the University of Michigan Medical School, analyzed radiology reports and related imaging studies that were discussed in weekly multidisciplinary thoracic oncology group conferences. The 256 reports interpreted CT scans of the head, neck, chest, abdomen, and pelvis performed between April 2006 and February 2008. Most of the cases consisted of patients with known or suspected lung or esophageal cancer.

Fifty-nine, or 22%, of the reports contained a total of 96 significant errors. A significant error was defined as an error that could potentially lead to the conveyance of incorrect or confusing information. The errors were categorized as containing an incorrect word or a nonsense phrase, not including an important word, right-left substitution, having an incorrect or nonsense date, using an incorrect image number or measurement number, using an incorrect measurement scale, or having a template-related error.

The radiologists whose reports contained errors all were experienced at using the speech recognition system (PowerScribe, Nuance Communications, Burlington, MA), and the group consisted of both native and non-native English speakers who were faculty, fellows, and residents. There was no significant difference in results between native and non-native language speakers, Quint reported. Residents and fellows whose reports were also signed by faculty radiologists had a lower percentage of significant errors (18%) than those signed by faculty alone (24%).

The majority of errors represented substitution of a wrong word (31 times), followed by nonsense phrases (19 times) and missing words (16 times). Sixty-three of the errors were in the "findings" section of the report, and 18 were in the "impressions" section. The remaining errors were located in the report headers and the "procedure" sections.

Error estimate

While the analysis was underway, the researchers sent an e-mail questionnaire to all 106 faculty radiologists at the institution. The questionnaire asked radiologists to estimate what percentage of finalized radiology reports produced by the entire department contained transcription errors that could cause confusion or misinterpretation, as well as what percentage the respondent individually generated. For each question, an answer of 0-5%, 5%-10%, 10%-20%, 20%-30%, 30%-40%, 40%-50%, and more than 50% could be selected.

The questionnaire generated an 83% response rate from 88 radiologists. Only eight radiologists selected a department error rate of 20%-30%. Thirty-five estimated that errors occurred in 5% or fewer reports, 20 estimated 5%-10%, and 20 estimated 10%-20%. If the reports analyzed by Quint and colleagues were representative, most of the faculty radiologists underestimated the overall percentage of errors. The authors noted that because the reports were lengthy and complex, if shorter reports had been analyzed, the significant error rate percentage may have been different.

With respect to individual error rates, 56% of the radiologists indicated error rates of less than 5%, and 23% selected less than 10%. The small sample size precluded meaningful comparisons with the reports analyzed for the study.

Quint suggested that "lack of awareness of [speech recognition errors] probably contributes to the high error rate." In response to an e-mail inquiry by AuntMinnie.com, she stated that the authors informally alerted colleagues to the fact that the errors were ubiquitous and quite common, but the study results have not yet been formally presented in a faculty meeting. One colleague was motivated to independently review 100 randomly selected reports, and determined that his rate for significant errors was 25%.

Quint also told AuntMinnie.com that she had received e-mail correspondence and telephone calls from radiologists at other medical centers advising that they had found similar high error rates in their own departments.

By Cynthia E. Keen
AuntMinnie.com staff writer
January 23, 2009

Related Reading

White noise may improve accuracy of SR system, November 27, 2007

Keys to a successful voice recognition implementation, July 12, 2007

Speech recognition technology shows double-digit error rate, November 6, 2006

Speech recognition hits mark, thanks to editors, June 8, 2004

Copyright © 2009 AuntMinnie.com

Page 1 of 603
Next Page