Full Text Article

Background

Diagnostic errors lead to permanent damage or death for as many as 160,000 patients each year, according to researchers at Johns Hopkins University [1]. Not only are diagnostic problems more common than other medical mistakes—and more likely to harm patients—but they’re also the leading cause of malpractice claims, accounting for 35% of nearly $39 billion in payouts in the U.S. from 1986 to 2010, measured in 2011 dollars, according to Johns Hopkins [1].

Misdiagnoses of some diseases range from 71% to 97% (RSD, electrical injuries, and fibromyalgia) [2-5]. This high rate of misdiagnosis is costly to insurance companies and other payers, as well as to employers of chronic pain patients, where 13% of the workforce lose productive time, estimated to cost industry $61 billion a year [6] . Furthermore, misdiagnosis creates protracted treatment and psychological problems for the patients themselves. Of all the misdiagnosed disorders, the most prevalent problem is chronic pain, which, according to the Academy of Pain Medicine, accounts for 100,000,000 patients in the United States alone [7]. The annual cost of health care for pain ranges from $560 billion to $635 billion (in 2010 dollars) in the United States, which includes the medical costs of pain care and the economic costs, related to disability, lost wages and productivity [7].

Insurance companies and physicians could improve patient care if they had a mechanism which would address the two leading cause of misdiagnosis: faulty history taking and ordering the wrong medical tests [1]. Therefore, a valuable tool for any health care system would be a questionnaire which could provide accurate diagnoses, and, based on the accurate diagnoses, predict the outcome of an expensive medical laboratory tests, to allow physicians to determine, using “evidence based medicine,” which tests would be diagnostic and which test would be of no value. This concept is best exemplified by the Ottawa Ankle Rules, and Ottawa Knee Rules, developed in Canadian emergency rooms. They developed a questionnaire, using “predictive analytic techniques,” which could predict which patient would or would not have abnormal ankle or knee X-rays. When the use of the Ottawa Ankle and Knee Rules was applied in emergency rooms, for the selection or denial of patients for ankle or knee X-rays, it decreased ankle and knee radiography up to 26 percent, with cost savings of up to $50,000,000 per year [8-11]. This significant savings was just in the city of Ottawa, and just for ankle and knee pain. If these techniques were applied to other cities and other conditions, the extrapolated savings would be billions of dollars a year.

Later research by the group from Ottawa demonstrated that “expert system” evaluations, based on predictive analytic research, were more accurate in predicting the results of cervical spine X-rays, and CT, than unstructured physician judgment [12-14]. “Expert systems” are best defined as computer software which attempts to mimic the reasoning of a human specialist, and improve the decision making process. This commonly employs a “self-improving system,” where the results of a decision are evaluated for accuracy and then fed back to the system, to improve the decision-making process. Theoretically, this leads to more accurate decisions.

These same “predictive analytic” techniques allowed physicians from Mensana Clinic and Johns Hopkins Hospital to develop a Pain Validity Test, an “expert system” which could predict with 95% accuracy who would have moderate or severe abnormalities on medical testing, and predict, with 85% to 100% accuracy, who would have no abnormalities, or only mild ones [15-18]. This “expert system” can be used to determine if a patient is faking, malingering or drug seeking, or if the patient has a valid complaint of pain [15-18, 51,52] The Pain Validity Test has always been admitted as evidence in legal cases in nine states. (see Appendix A for a partial list of these cases).

Past research reports indicate that 40% to 67% of chronic pain patients involved in litigation are misdiagnosed [19,20]. When evaluating complex regional pain syndrome, (CRPS), formerly called reflex sympathetic dystrophy (RSD), Hendler found that 71% of the patients, and Dellon found that 80% of the patients, who were told they had only CRPS I, actually had nerve entrapment syndromes [2,3]. These errors in diagnoses are costly to the patient and the insurance industry alike, since they prolong or result in inappropriate treatment.

Another complicating factor in medical evaluations is the specificity and sensitivity of medical testing. If a test is too sensitive, it can lead to “false positive” results being reported, i.e. claiming there is a pathological condition when one does not exist. Conversely, if a test is very specific, so that the only pathology it detects will be just a certain diagnosis, it will lead to “false negative” reports, i.e. overlooking a medical condition which does exist. So there is an inverse relationship between sensitivity and specificity of medical testing. This concept is best summarized by Table 1

A practical example of sensitivity and specificity is the MRI, a test used by most physicians to detect anatomical abnormalities, such as disc damage in the spine. The medical literature shows that the MRI has a 28% false positive rate, and a 78% false negative rate for detecting spinal disc damage, compared to a provocative discogram. [21,22], but most clinical practices do not employ provocative discograms to diagnose vertebral disc damage. Moreover, difference in the technique of performing provocative discograms, with both a provocation and anesthetic component, [23], or if a discogram was merely anatomical versus physiological produce different results [24]. Similar problems arise when physicians use merely a CT, which misses pathology 56% of the time compared to a 3D-CT, in unoperated patients, and 76% of the time in patients who have had previous surgery [25].

One of the factors leading to overlooked or missed diagnoses is the history taking techniques of physicians. Traditionally, physicians would take a careful history, derive a diagnosis and differential diagnosis, and use medical testing to confirm or reject the various diagnoses. However, a shift in the medical evaluation paradigm has occurred. Recent research documents that after a physician entered the room, patients were able to speak an average of 12 seconds, before being interrupted by the physician. The time with patients averaged 11 minutes, with the patient speaking for about 4 minutes of the 11 minutes [26]. Computer use during the office visit accounted for more interruptions than beepers. Another study confirmed the truncated time physicians spend with patients. The average face-to-face patient care time measured by direct observation in this recent study was 10.7 minutes. When researcher evaluated the time spent on “visit-specific” work outside the examination room and combined it with face-to-face time, the average time per patient visit was 13.3 minutes. [27]. Therefore, this increasingly prevalent process follows the format of first getting a list of the symptoms, then getting medical testing pertinent to the symptoms to establish or eliminate diagnoses, and finally reaching a diagnosis, i.e. using medical testing to make a diagnosis. Unfortunately, with an inadequate history, the chance of selecting incorrect medical testing increases, leading to an erroneous diagnosis.

An article by Donlin Long, MD, PhD, published at the time he was the chairman of neurosurgery at Johns Hopkins Hospital, summarizes the compellation of these errors. Dr. Long and his colleagues published an article reporting their evaluation of 70 patients, who had their symptoms for longer than 3 months, who were referred to Johns Hopkins Hospital with normal MRI, X-ray and CT studies [28]. In the absence of abnormal medical testing, no clear diagnosis had been established by the referring doctors, other than cervical sprain or strain. By definition, all 70 of these patients had been misdiagnosed, since sprains and strains are defined as a self-limited injury, which lasts no more than one to six weeks (29,30). Cervical disc disease, facet syndrome, and anteriolysthesis were the most commonly overlooked diagnoses. When properly diagnosed and tested, Dr. Long found that 95% of the patients needed interventional testing, such as facet blocks, root blocks, and provocative discograms, to confirm diagnoses. After the diagnostic testing was performed, 63% of the patients were found to be candidates for anterior or posterior cervical fusions, and 93% of those had good or excellent results post-operatively [28].

All of the above medical facts impact upon the use of artificial intelligence for the diagnosing medical problems. While there are many methodologies employed in the creation of an artificial intelligence application, this paper will limit discussions to the development of “expert systems,” which use computer scoring of questionnaires to duplicate diagnoses made by physicians.

Some authors feel only limited progress has been made in expert systems [31]. Engelbrecht feels that the quality of knowledge used to create the system, and the availability of patient data are the two main problems confronting any developer of an expert system, and advocates an electronic medical record system to correct one component of the problem [32]. Babic concurs with the value of the longitudinal collection of clinical data, and data mining to develop expert systems [33]. Those expert systems that seem to have the best results are the ones that focus on a narrow and highly specialized area of medicine. One questionnaire, to cover 32 rheumatologic diseases, consists of 60 questions, was tested on 358 patients [34]. The diagnosis correlation rate between questionnaire and clinical diagnosis was 74.4%, and an error rate of 25.6%, with 44% of the errors attributed to “information deficits of the computer using standardized questions.” [34]. However, a later version called “RHEUMA” was used prospectively in 51 outpatients, and achieved a 90% correlation with clinical experts [35]. Several groups have approached the diagnosis of jaundice. ICTERUS produced a 70% accuracy rate while ‘Jaundice’ also had a 70% overall accuracy rate [36, 37]. An expert system for vertigo was reported, and it generated and accuracy rate of 65%, [38]. This later was reported as OtoNeurological Expert (ONE), which generated the exact same results reported in the earlier article [39]. There was a 76% agreement for diagnosis of depression, between an expert system and a clinician [40]. When a Computer Assisted Diagnostic Interview (CADI) was used to diagnosis a broad range of psychiatric disorders, there was an 85.7% agreement level with three clinicians [41]. Former Johns Hopkins Hospital doctors have developed “expert systems” which address patients with the highest level of misdiagnosis. For headaches, where 35%-70% of patients were mistakenly told they have migraine headaches, the former Johns Hopkins Hospital doctors developed an “expert system” questionnaire which scores answers using Bayesian analytic methods, which gives diagnoses with a 94% correlation of diagnoses of former Johns Hopkins Hospital doctors [42]. For patients with chronic pain problems, where the misdiagnosis rate has been reported to be 40%-80%, even as high as 97% [2-5, 28], the former Johns Hopkins Hospital doctors have an “expert system” questionnaire which gives diagnoses with a 96% correlation with diagnosis of Johns Hopkins Hospital doctors [43]. Finally, in a combined research project with University of Rome, the Diagnostic Paradigm for Chronic Pain was able to predict with 100% accuracy what a surgeon would find intra-operatively, based on the pre-operative diagnosis [44].

The expert systems were designed to be self-improving, using results to re-examine the data, and suggest different type of questions, with improved diagnosis and outcomes. An accurate diagnosis is the ultimate predictive analytic tool. It allows a physician to know the origin of the problem, and select the proper treatment to address the cause of the pathology. However, while the value of improving on the accuracy of diagnosis is obvious, from a cost containment perspective [1,6], the real gold standard is outcome results. This can take the form of elimination of unnecessary testing or surgery, reduced mortality for a given diagnosis, reduced number of doctor visits, reduced use of medication, the selection of the proper type of medication, and a host of quantifiable and objective outcome measures. The major thrust of artificial intelligence programs in medicine is the creation of methodology which improves the accuracy of diagnosis, and ultimately improved outcomes [45]. But the major problem with this approach is the collection of accurate data. Almost all medical artificial intelligence systems utilize the concept of “data mining” which is analyzing massive amounts of data, in the hopes that the “law of big numbers” (the guiding concept in actuarial analysis used by insurance companies) [46] will make their results more accurate. Therein lies the problem with the use of artificial intelligence for improving medical care and containing costs. As reported above, the medical diagnoses are erroneous 35%-80% of the time or more, and the medical testing has false negative rates ranging from 56% to 78%. These errors are multiplicative if taken in tandem. No matter how elegant the analysis of the data, if the data are incorrect, then the garbage in will yield nothing more than garbage out producing a classic case of GIGO.

One way to reduce the errors inherent in reviews of electronic medical records is the application of patient-generated health data (PGHD), where the patient themselves reports symptoms and outcomes, rather than rely on physician notes and interpretations [47]. This methodology would at least afford the opportunity of a more comprehensive and accurate history. This is the methodology used by the “expert systems” designed by former Johns Hopkins Hospital doctors [42, 43]. Another process for optimizing accuracy of artificial intelligence results is to utilize reported outcome studies, and retrospectively determine what factors led to the best results. Unfortunately, this methodology is also prone to multiple sources of errors. In a classic meta-analysis of the efficacy of treatments for reflex sympathetic dystrophy (RSD), Payne found physicians’ reported improvement of 12% to 97% of their treated cases [48]. The sources of error ranged from accuracy of diagnosis, to methodology used to treated RSD, to definition of criteria needed to be considered a treatment success. Of course, outcome results of treatment reported by a physician are subject to bias. The more objective the criteria to measure outcome, the easier it is to quantify data. As an example, research reports which measure “pain relief” are using a very subjective criteria, compared to quantification of narcotic use before and after treatment. The ultimate outcome measures would be validated by third parties, not use physician self-reporting.

The outputs of AI are clearly skewed if the data being used are not consistent with reality. Anyone who works with artificial intelligence knows that the quality of the data goes a long way toward determine the quality of the result. That is the problem to be addressed, and therein lies the opportunity. Mitigating the effects of garbage data becomes a moral imperative when an algorithm is being used to make to invest billions of dollars in a medical treatment program. The problem gets even worse when the garbage is derived from malfeasance or bias. To use machine learning effectively, a researcher should embrace the potential for garbage data and anticipate it. That approach means that these data need to be challenged at the time they become data --- before they are used to create invalid, possibly dangerous outcomes. Mikey Shulman is the head of machine learning for Kensho Technologies, acquired by Standard and Poor (S&P Global) last year. He insists that his teams develop a deep understanding of the datasets they use. “A lot of this comes down to the machine-learning practitioner getting to know the data and getting to know the ways in which it’s flawed,” he says. [49]. Accurate data will allow machine learning to lower costs and get better outcomes. Therefore there is a need to build tools and methodologies which will help validate the data going into an AI system. If done correctly, everyone benefits.

One preliminary step toward devising a viable AI processing system was reported by Hendler and Joshi (50). They describe the Large Analytic Bayesian System (LABS) which uses Bayesian analytics to list, on a symptom by symptom basis, the likelihood of having abnormal medical testing from a list of possible medical tests associated with a symptom, or cluster of symptoms. LABS produces a rank ordered list of medical tests, comparing the symptom(s) with the frequency and severity of abnormal medical tests. Seventy-eight medical charts with evaluations including all pertinent medical tests, and a completed Diagnostic Paradigm with 2008 possible symptoms were reviewed. On a symptom by symptom basis, medical test results in the chart were compiled using the Large Analytic Bayesian System (LABS) and created a rank ordered list of abnormal medical tests from a list of possible 107 medical tests. This resulted in a 2008 by 107 matrix, which was analyzed by the use of a program called the Diagnostic Test Manager. The results clearly demonstrated that physiological testing, such as root blocks, facet blocks and provocative discograms had nearly double the frequency of abnormalities compared to anatomical tests, such as X-ray, CT scans and MRIs for the same symptoms in the same patient. This evidence-based approach will help physicians reduce the use of unnecessary tests, improve patient care, and reduces medical costs [49,50].

In summary, the potential of AI to enhance medical practice is enormous, ranging from increased accuracy of test interpretation, to more accurate diagnosis with the improved patient benefit. This is clearly demonstrated by a number of the studies above, which achieved accuracy well over 90% [15,16,17,18,28,35,42,43,44,51,52] However, these high accuracy studies all shared a common feature. There were granular in nature, evaluating small number of patients (51-251) with great attention to the individual clinical aspects of each case. Perhaps there is a lesson to be learned from reviewing examples from the basic biological literature, where the variables are less complex, and methodology can be better tested [53-58]. The methodology applied from this research can be applied to clinical evaluations, i.e. the utilization of feedback to enhance machine learning, which is essential for a self-improving system. This is the very essence of artificial intelligence.

On the other hand, data mining of large populations, where the accuracy of individual cases is not determined, nor verified, is susceptible to the GIGO phenomena. No collection of data should be assumed to be accurate, without controlling for all the variable outlined above. Only in this fashion will the full benefit of artificial intelligence in medicine be realized.