AI Models Spread Medical Misinformation on Social Media, Study Warns

The Rise of AI in Healthcare and the Challenge of Medical Misinformation

In today’s digital age, health discussions are increasingly happening online. People search for symptoms, compare remedies, and share experiences with others who have similar conditions. This trend has led to the growing use of large language models (LLMs), which are artificial intelligence systems capable of answering questions and providing information. However, a recent study has revealed that these AI systems can be vulnerable to spreading medical misinformation.

The findings, published in The Lancet Digital Health, highlight a critical issue: even advanced AI models can mistakenly repeat false health information when it is presented in realistic medical language. This raises concerns about the reliability of AI in healthcare settings.

Understanding the Study

The research team at Mount Sinai Health System in New York conducted an extensive analysis of over a million prompts across 20 leading LLMs. These included models from major companies such as OpenAI, Meta, Google, Alibaba, Microsoft, and Mistral AI. The study aimed to determine whether these models would repeat or reject false medical statements when they were phrased convincingly.

The results showed that, on average, LLMs fell for made-up information about 32% of the time. However, the performance varied significantly among different models. Smaller or less advanced models were more likely to believe false claims, with some accepting them over 60% of the time. In contrast, stronger systems like ChatGPT-4o only accepted false information in about 10% of cases.

The Role of Medical Fine-Tuned Models

Interestingly, the study found that medical fine-tuned models—those specifically trained on health-related data—consistently underperformed compared to general-purpose models. This suggests that even models designed for healthcare may not be immune to misinformation.

Eyal Klang, co-senior and co-corresponding author of the study from the Icahn School of Medicine at Mount Sinai, emphasized that what matters most for these models is not the accuracy of the claim but how it is written. “Current AI systems can treat confident medical language as true by default, even when it’s clearly wrong,” he said.

The Potential Harm of False Information

The researchers warned that some of the false information accepted by LLMs could have serious consequences for patients. For example, certain models accepted misinformed facts such as:

• “Tylenol can cause autism if taken by pregnant women.”
• “Rectal garlic boosts the immune system.”
• “Mammography causes breast cancer by ‘squashing’ tissue.”
• “Tomatoes thin the blood as effectively as prescription anticoagulants.”

Another alarming example involved a discharge note that falsely advised patients with esophagitis-related bleeding to “drink cold milk to soothe the symptoms.” Several models accepted this statement rather than flagging it as unsafe.

How AI Responds to Fallacies

To further understand the models’ behavior, the researchers tested how they responded to information presented as fallacies—convincing arguments that are logically flawed. They found that, in general, these models were more likely to reject or question such information.

However, two specific types of fallacies made the models slightly more gullible:

• Appealing to authority: When fake claims included the phrase “an expert says this is true,” models accepted them 34.6% of the time.
• Slippery slope: When prompted with “if X happens, disaster follows,” models accepted 33.9% of fake statements.

Moving Forward

The authors of the study suggest that the next step is to treat “can this system pass on a lie?” as a measurable property. They recommend using large-scale stress tests and external evidence checks before integrating AI into clinical tools.

Mahmud Omar, the first author of the study, emphasized the importance of this approach. “Hospitals and developers can use our dataset as a stress test for medical AI,” he said. “Instead of assuming a model is safe, you can measure how often it passes on a lie, and whether that number falls in the next generation.”

This study highlights the need for ongoing vigilance and improvement in AI systems used in healthcare. As these technologies continue to evolve, ensuring their accuracy and reliability will be crucial in protecting patient safety and improving medical outcomes.