Natural language processing captures new insights on patient safety

Safety monitoring at Boston Children’s Hospital is getting even more thorough, thanks to natural language processing systems. Aided by machine-learning algorithms, these systems are capturing problems that clinicians didn’t explicitly flag through a safety reporting system, but documented in their notes. The net result is further opportunities to improve care.

Natural language processing, or NLP, uses linguistics and artificial intelligence (AI) to draw meaning from unstructured “free text” — in this case, the notes that clinical providers insert in patients’ medical records.

“The hospital places an emphasis on patient safety and collects information on safety events, but no system can capture all the events that occur,” says Al Ozonoff PhD, CPPS, a scientist in Boston Children’s Division of Infectious Diseases whose research focuses on health surveillance methods. “There are many reasons why an event may go uncaptured. Providers may not have had time to stop and report during their shift, or the event could be relatively minor, or they may have caught the event before harm occurred.”

In the past, additional safety events have been identified through manual review of patient records — but that’s not feasible on a large scale. Ozonoff, then part of the Program for Patient Safety and Quality at Boston Children’s, wanted a more automated, scalable way to gather safety data from patients’ electronic health records.

Jonathan Finkelstein, MD, MPH, Boston Children’s chief safety and quality officer, introduced Ozonoff to Amir Kimia, MD, in the Division of Emergency Medicine. Kimia had been working extensively on applying NLP to clinical queries. He and Ozonoff joined forces to conduct a one-year pilot study, together with Patricia Pratt, BSN, MA, RN, CPN, NE-BC, vice president and associate chief nurse for Pediatric Medical Services.

Infiltrating clinical notes

The pilot, which has now led to larger projects, focused on so-called “infiltrate events.” These are safety events in which a child had an IV placed, but due to breakage or other mishap, the fluid infiltrated into the surrounding tissue rather than the vein.

“This is not the most serious safety event, and it’s often caught very early, but it can be important in kids whose limbs are small and whose IVs are hard to place,” says Ozonoff. “There are many of these events per year across the hospital, the vast majority of which cause minimal harm. We were interested in how many are missed and why they’re not fully captured by our existing systems. This was a chance to try out NLP, see if it’s feasible, and see what we could learn.”

Ozonoff and Kimia extracted 21,362 clinical notes from the records of 3,497 patients over a six-month period. They “trained” the NLP system by showing it a subset of 2,342 notes that they had manually classified as “yes” or “no” for infiltrate events. They then ran all the clinical notes through the NLP system. While the system flagged many false positives, it reliably ruled out infiltrate events and identified 44 new cases that hadn’t been captured previously.

Deconstructing NLP

So what is NLP? Kimia explains:

“NLP is a field within artificial intelligence and linguistics that helps computers ‘understand’ and process everyday language. One of its common tasks is to classify documents — those that contain a particular event versus those that do not. A search engine like Google is one example of a classifier: it provides us with webpages possibly containing our query, while excluding billions of others that are not relevant.

“Clinical experts provide the system with key phrases they think are predictive of the event,” Kimia elaborates. “These data form an initial ‘training set.’ Our experts can provide feedback to refine the algorithm. We feed that back into the machine, and over time it will ‘learn’ to recognize the event.”

NLP algorithms make no assumptions; they are completely agnostic, using word patterns or statistical correlations with the event of interest. Kimia applies two main NLP methods. One, called “bag of words,” initially strips each clinical note down to a list of words that the system uses to identify relevant notes, in this case those involving an infiltrate. The other method is based on the proximity of words to each other; it figures out what words tend to occur together and how that proximity correlates with actual infiltrate events.

Most importantly, Kimia created a clinician interface that distills expertise related to the question at hand into a robust training set.

“I am not developing NLP methods,” he says. “I am of one of very few who make the methods accessible to medical professionals.”

Sometimes there are false positives, but with human curation and machine-learning feedback, the models’ accuracy improves over time. “What we’re doing is shifting a lot of the grunt work over to the computer but still providing human intelligence where it’s needed,” says Ozonoff.

Monitoring patient safety in bedside procedures

Ozonoff admits that he struggled with the concepts at first. It’s sometimes mysterious how the NLP system differentiates between safety events and non-events, purely based on language patterns. But he came to embrace NLP enthusiastically, and is now leading a five-year, $1.98 million study, funded by the Agency for Healthcare Research and Quality. It expands the pilot to other safety events such as patient falls, pressure injuries, and venous thromboembolisms. Kimia will lead efforts to validate the NLP approach in a community hospital setting, working with Boston Children’s partner South Shore Hospital.

Patient safety is not a well funded area of research, which is one of the reasons we’re very proud of these projects. It’s about keeping children safe.

Meanwhile, Kimia is leading a two-year, $200,000 study funded by the Controlled Risk Insurance Company (CRICO), which insures all of the Harvard medical institutions and their affiliates. Titled “Bedside Procedure Attempts: ‘If at First We Don’t Succeed…,’” it will focus on safety events during ten kinds of interventions at the patient’s bedside, such as spinal taps. While successful attempts leave a long trail of evidence, failed attempts may only be documented in clinical notes or procedure logs, making success rates difficult to monitor.

“We’re trying to enhance the data available and apply it to deliver safe care,” says Ozonoff. “Patient safety is not a well funded area of research, which is one of the reasons we’re very proud of these projects. It’s about keeping children safe.”

Learn more about the Program for Patient Safety and Quality at Boston Children’s.