In the recent webinar “Artificial Intelligence (AI) Hype in Healthcare: Pros and Cons of Rules-Based vs. Machine Learning Decision Support Tools,” we explored the differences in these two types of engines and how they are used in consumer patient symptom checker and care direction tools. In our opinion, the single most important takeaway from the session is to do your due diligence.
Consumer patient symptom checker and care direction tools are designed to help a patient understand their symptoms and to find an appropriate level of care. Given the importance of what the applications are designed to do, you should look for several key items:
- A high level of clinical accuracy
- Coverage of a broad range of conditions
- A number of independent, medically peer-reviewed studies and articles
To accomplish this level of scrutiny certainly isn’t easy but should be the minimum for tools helping patients with healthcare decisions. We expect this of physician level tools. Why would consumer tools be any different?
In our opinion, the single most important takeaway from the session is to do your due diligence.
In our opinion, the single most important takeaway from the session is to do your due diligence.
In our reviews, we have found that a major determinant is whether the system is rules-based or machine learning based. To help understand the type, use this quick test:
Does the system allow entry of free text symptoms with no translation or a fixed symptom from a pre-defined list?
With most rules-based systems, the symptoms are entered from a fixed list provided by the tool. A few rules-based systems put Natural Language Processing (NLP) on the front end to recognize symptoms, but still must attempt mapping them to only the fixed symptoms they cover. Either way it is limiting; there are a finite set of symptoms recognized and those are also then mapped to a limited number of potential conditions. This technology is very complex to maintain and scale which inherently limits the breadth of coverage — ultimately not doing the full job the tool is meant to do.
Other indicators to look for include:
- The system asks the user to pick a Chief Complaint (sometimes stated as the “symptom bothering you the most”)
- The system specifically asks the patient if they feel they need to go to the emergency room
- The system forces the patient to select a condition (basically self-diagnose) that helps determine the correct level of care to seek (treat at home, virtual visit, retail clinic, primary care doctor, urgent care or emergency room) from the generated list.
Don’t be fooled by clever user interfaces like chatbots, voice, etc. The Clinical Engine is the most important component.
With a true machine learning/NLP engine, symptoms can be entered via natural language and all of the symptoms, including those leveraging full free-text, are used to drive the list of possible conditions and the level of care direction. These tools are easy to scale, the condition coverage is broad and the resources do the job they are intended to do.
You must investigate and test these systems for accuracy and efficacy. We should expect the same independent medical validation that tools designed for providers receive. First, ask the vendor how the system has been independently validated (not internal studies). Second, run sample cases to see if they come up with the correct suggestions for both conditions and care direction recommendations using cases with three to four symptoms at a minimum. Getting to this level of detail is critical to understand true accuracy and efficacy. While not easy, it is imperative to choosing the correct tools for your patients.
In summary, don’t just accept the marketing hype. Thorough due diligence is key to fully understand how each system works, as this also determines how well they perform. To take a deeper dive into this topic, click the link below to view our recent webinar, “Artificial Intelligence (AI) Hype in Healthcare: Pros and Cons of Rules-Based vs. Machine Learning Decision Support Tools.”