Should You Trust ChatGPT for Medical Advice? New Study Warns
The Growing Use of AI in Medical Advice
More than 230 million people a week turn to ChatGPT for medical advice, ranging from checking whether food is safe to eat, managing allergies, or finding remedies for a cold. This widespread usage highlights the increasing reliance on artificial intelligence for health-related queries. However, a recent study published in Nature has raised concerns about the effectiveness of AI tools like ChatGPT Health in handling serious medical emergencies.
Limitations in Handling Emergencies
While ChatGPT Health performs well for textbook cases, such as identifying symptoms of a stroke or severe allergic reactions, it failed to advise emergency care in more than half of the cases that required urgent medical attention. Ashwin Ramaswamy, lead author of the study at Mount Sinai in New York, emphasized the critical question: “If someone is experiencing a real medical emergency and turns to ChatGPT Health for help, will it clearly tell them to go to the emergency room?”
The research team created 60 structured clinical scenarios across 21 medical specialties, including both minor conditions and true medical emergencies. Three independent physicians determined the correct level of urgency for each case using guidelines from 56 medical societies.
Challenges in Recognizing Hidden Dangers
ChatGPT Health struggled in situations where the danger was not immediately obvious. For instance, in an asthma scenario, the system identified early warning signs of respiratory failure but still advised waiting rather than seeking emergency treatment. This inconsistency raises concerns about the reliability of AI in critical health situations.
Integration with Personal Health Data
ChatGPT Health was launched by OpenAI in January 2026, allowing users to connect their health information—such as medical records and data from wellness apps like MyFitnessPal—to receive more personalized and contextual responses. However, the integration of such data does not guarantee accurate medical advice.
Inconsistent Responses to Self-Harm
The study also examined how the model responded to users reporting self-harm intentions. ChatGPT Health is supposed to be programmed to encourage users to seek help and call a public health number when they mention self-harm or suicidal thoughts. However, during the study, the banner “Help is available,” linking to the suicide and crisis lifeline, appeared inconsistently.
The authors noted that the guardrail answered more reliably for patients who had not identified a means of self-harm than for those who had. The pattern was not merely inconsistent but paradoxically inverted relative to clinical severity.
Balancing Trust and Caution
Despite these findings, the researchers did not suggest consumers should abandon AI health tools altogether. Alvira Tyagi, second author of the study, emphasized that as a medical student training in an era where AI health tools are already in use, she sees them as technologies that must be integrated thoughtfully into care rather than substitutes for clinical judgment.
The study authors advised that people experiencing worsening or concerning symptoms, including chest pain, shortness of breath, severe allergic reactions, or changes in mental status, should seek medical care directly rather than relying solely on chatbot guidance.
Ongoing Evolution of AI Models
The study also noted that AI language models are constantly evolving and frequently updated, meaning performance can change over time. Tyagi highlighted that starting medical training alongside tools that are evolving in real time makes it clear that today’s results are not set in stone.
She added that the rapidly changing reality calls for ongoing review to ensure that technology improvements translate into safer care. As AI continues to advance, it is essential to balance its potential benefits with the need for careful oversight and human expertise.
