AI is getting 80% of medical diagnoses wrong - report

A new study has raised new concerns about the growing use of artificial intelligence for self diagnosis and warned AI systems still struggle to correctly identify diseases during the early stages of illness.

The research done by Harvard Medical School found that artificial intelligence chatbots and large language models recorded an 80% failure rate when attempting to diagnose patients using incomplete medical information.

The team tested 21 advanced AI systems, including GPT, Gemini and Claude, to assess how accurately they could identify diseases based on symptoms and signs presented by patients.

Researchers found that the systems performed poorly during “differential diagnosis,” the stage where doctors analyse symptoms to narrow down possible illnesses. 

The error rate dropped to 40% after additional test results and medical information were added.

The researchers concluded that while AI performs better when complete medical data is available, it remains unreliable during the early stages of patient care where uncertainty is common.

“AI excels at identifying final diagnoses when data is complete but struggles in early stages with limited information. It has not yet reached a level where it can be entrusted with decision making for patients without medical professionals’ intervention,” the research team warned.

The findings come at a time when many people increasingly turn to AI chatbots for health advice before visiting hospitals or clinics.

Health experts have repeatedly warned against relying entirely on online symptom checkers and self diagnosis tools because symptoms for different diseases often overlap.

Ugandan health authorities have also encouraged careful use of AI in healthcare. While government officials and researchers support the use of AI to improve health services, experts warn that the technology should not replace trained medical workers.

Uganda has in recent years increased investment in AI powered healthcare research through institutions such as the Makerere University AI Health Lab. 

Ugandan researcher Rose Nakasi has led projects using artificial intelligence to support diagnosis of diseases such as malaria, tuberculosis and cervical cancer. 

AI is also being explored in mental healthcare in Uganda. A recent project involving researchers from Makerere University and Butabika Hospital is developing AI systems that can identify signs of depression and suicide risk in local languages such as Luganda and Swahili. 

However, experts involved in these projects have stressed that AI should support healthcare workers rather than replace them.

Medical experts warn that AI systems can misunderstand symptoms, miss critical details or provide misleading advice when users enter incomplete or inaccurate information.

Global studies on self diagnosis have also shown that online health tools may increase the risk of misdiagnosis, delayed treatment and unnecessary panic among users. 

Uganda’s health sector already faces challenges such as shortages of medical workers, limited access to healthcare facilities and high patient numbers, making accurate diagnosis even more important. 

Researchers say AI still holds strong potential in healthcare, especially in improving access to medical support and speeding up analysis of medical data. However, they insist that human medical professionals must remain central in patient care and diagnosis.