Interest—and concern—about artificial intelligence ran so high at the 2017 RSNA meeting that fully 20 minutes prior to the start of “Deep Learning in Radiology: How I Do It,” all seats were occupied and the room was ringed by a standing-room only crowd.

Luciano M. Prevedello, MD, MPH, Curtis Langlotz, MD, PhD, and Bradley Erickson, MD, PhD, repaid their eager audience with a rare gift in these times of high AI anxiety—in describing how they are approaching deep learning in their respective departments, they demystified a powerful new tool.

Each shared their respective programs’ goals, team composition, tools, and details on a project or two.

Prevedello, a neuroradiologist who is building the deep learning effort in radiology at Ohio State University, began with a brief history and some clear-cut definitions of terms often erroneously used interchangeably:

Artificial intelligence is a broad term that applies to any technique allowing computers to mimic a human brain.

Machine learning is what happens when you attempt to teach a computer to make an inference about new data, typically task-related.

Deep learning is a form of machine learning that uses multiple hidden layers in a neural network.

To illustrate the progress of the science, Prevedello referenced ImageNet, a large-scale image recognition challenge in which the ability of computer algorithms to categorize images is evaluated. He noted that the error rate of 27% in 2010 dropped precipitously after convolution neural network algorithms were introduced, and the ability of computers to classify images managed to best the human error rate of 5% in 2015.

Biologically inspired, convolution neural networks were able to mimic what the human retinue does, which reduces a hundred million photo-receptors into about a million ganglion cells, reducing the number of cells that deal with the incoming information.

The OSU Laboratory for Augmented Intelligence team is composed of six members—two physicians (including Prevedello and department chair Richard White, MD), three engineers, and one medical physicist—with plans to add two data scientists, one programmer, and a database curator.

Their “Hal” is three water-cooled super computers with 15,000 cores each that operate with the assistance of four GPUs. For comparison purposes, Prevedello noted that home computers have an average of eight cores. Each computer is capable of 44 teraflops.

For the most part, the lab used open source tools, including Linux, Caffe, TensorFlow, DL4J, and Python. Prevedello’s team created a pipeline that siphons off imaging and clinical data to repurpose for enabling discoveries related to deep learning.

Solving Problems

At OSU, 40% of inpatient studies are stat, not uncommon in the academic setting, so initial projects addressed the problem of prioritization of imaging studies. Prevedello and team used machine learning to train a deep learning convolutional neural network algorithm to distinguish normal CT head studies from those showing hydrocephalus, mass effect, and hemorrhage by feeding the network a dataset of about 250 head CTs, half of which were normal.

The network had a  90% sensitivity and 85% specificity in recognizing hydrocephalus, and for stroke, which requires a more refined observation, the network achieved 62% sensitivity and 96% specificity.

OSU’s objective is to make scanners more intelligent so that radiologists can be notified to read critical cases sooner, or the worklist can be reshuffled to put the highest priority cases at the top of the queue, Prevedello said. Results were published in Radiology.

The OSU team also is doing a lot of work on text analysis based on the concept of grouping related words, which Google pioneered and does very well. Prevedello offered an example from neuroradiology: grouping sclerosis with multiple, tuberous, amyotrophic, and meningioma.

Using a dimensionality algorithm to group like words into a vector space enables greater clarity, Prevedello said.  In the upcoming year, the lab will use the  algorithm not just for image interpretation, but for creating study protocols, natural language understanding of radiology reports, and text summarization projects.

Lessons Learned

He also shared important lessons learned since the lab was established a year ago:

Finance is important. Time and money are interchangeable here, and support is necessary to make deep learning happen.

It’s all about the team. There is no way you can do this correctly with one single perspective. You need people with different views of the same problem and that’s why we have physicians, engineers, physicists, data scientists, data curators all working on the same problem.

Clinical data is messy and not created for research. Determining how much curation is required is challenging and requires focusing on the issue you are addressing. “If the noise within data is just noise, then big data can figure it out,” Prevedello said.  If there is bias in the data, it will be reflected in your results, he added.

Data curation is the bottleneck. “I feel like we are spending 90% of our time on data curation as opposed to working on the deep learning algorithms,” he shared.

Metrics are important. Prevedello recommends against using accuracy as a metric, because it will give the research a false sense that you are doing better than you are. Use area under the ROC curve and manifold validation

In conclusion, Prevedello recommended that radiologists jump in and get started. “It doesn’t matter how or what you do, this is a ride that we are all getting into, so we might as well do it with excitement,” he said.

Check back in the weeks ahead for coverage of the Mayo and Stanford deep learning programs.

—Cheryl Proval