Why AI Struggles in Medical Environments
The Challenges of Clinical AI: Beyond Generating Text
Heidi Health’s Chief Technology Officer and Co-Founder, Yu Liu, highlights that the primary challenge in clinical artificial intelligence (AI) lies not in generating fluent text but in designing systems that can function effectively within the complex constraints of healthcare environments. These constraints include accuracy, auditability, privacy, and workflow variability, all of which directly impact patient care.
In a conversation with Frontier Enterprise, Liu shared insights into why broad AI models struggle with clinical documentation, why enterprise readiness cannot be added later, and what surprised the company most when integrating Heidi into existing clinical workflows across different healthcare settings.
Where Do AI Models Fall Short in Clinical Documentation?
General-purpose AI models often fail in areas where precision, traceability, and governance are essential. While they may produce readable text, they typically lack features critical for medical documentation, such as audit trails, authoritative sourcing, and controls that allow clinicians to determine how data is used. Their outputs may appear accurate, but they can still deviate from local clinical standards, hospital policies, or regulatory requirements.
AI tools should never replace clinical judgment. They must serve strictly as decision-support tools, with the final responsibility resting with the practitioner. Heidi was designed with task-specific accuracy, auditability, and source transparency in mind to support safety and accountability in clinical practice.
What Problem Emerged When Deploying AI Models into Real Clinical Use?
One of the biggest challenges was enterprise readiness. Supporting enterprise requirements involves more than just adding functionality. Enterprise readiness cannot be retrofitted because moving quickly on capability without the underlying infrastructure in place can lead to re-engineering cycles that cost more in both time and trust than building it properly from the start.
Today, the focus is on building fundamental engineering setups so new features can scale from a single clinician to a small clinic, and eventually to hundreds of thousands or millions of users on the same platform.
Which Matters Most in Healthcare AI Systems: Latency, Privacy, or Accuracy?
All three—latency, privacy, and accuracy—are critical in healthcare AI, but accuracy has had the most significant impact on system design. In clinical settings, an AI system that is fast but unreliable can introduce patient risk, so models are fine-tuned for specific clinical tasks, whether it involves speech-to-text transcription or clinical note generation.
That said, latency and privacy also shape the architecture. Real-time tasks such as transcription require computation to happen close to the clinic, which is why Heidi is building on-premises and on-device deployments alongside cloud infrastructure to reduce delays and improve resilience. At the same time, patient data is never used to train models, but is processed in regionally appropriate environments and governed according to HIPAA, GDPR, and local privacy laws.
Designing Heidi required balancing these constraints: clinical accuracy, response times for real-time use cases, and protection of sensitive data through privacy controls and onshore storage practices. The goal is to build a platform clinicians can trust for reliability and regulatory compliance.
How Does Heidi Handle Uncertain or Incorrect AI Outputs?
Transparency is at the core of Heidi’s design. Within Heidi’s Evidence product, medical facts and claims are supported by citations so clinicians can review the underlying sources directly. The system can also detect some forms of ambiguity, such as conflicting guideline recommendations or gaps in the literature, although this is not guaranteed in all cases and outputs should still be reviewed critically by the clinician.
AI is viewed as a decision-support tool rather than a decision-maker, with final clinical responsibility remaining with the practitioner. The aim is to support safety, accountability, and trust in day-to-day clinical use.
What Was Most Unexpected About Integrating AI into Clinical Workflows?
The biggest surprise was the diversity of workflows, even within a single hospital. Departments, teams, and individual clinicians often operate differently, so one-size-fits-all approaches rarely work. Early on, the assumption was that standard integration paths would be sufficient, but in practice, successful deployment required integrating Heidi more closely into each team’s daily routines.
This involved iterative adjustments to interface design, task flows, and role-specific access to support different users without disrupting care. Clinicians can interact with the platform while maintaining their existing workflows and using AI support where appropriate. This approach has been crucial for adoption, operational efficiency, and patient safety across deployments in different regions.
